Repository navigation
Conversation
…y builds Move CONFIG_PIPELINE_2_0, CONFIG_ZEPHYR_DP_SCHEDULER, CONFIG_ZEPHYR_DP_SCHEDULER_MIN_STACK_SIZE, and CONFIG_CROSS_CORE_STREAM from zephyr/Kconfig to src/schedule/Kconfig so they can be enabled on both Zephyr ACE firmware targets and host library (CONFIG_LIBRARY) testbench builds. Also default CONFIG_MP_MAX_NUM_CPUS to 8 when CONFIG_LIBRARY is enabled and expose the non-Zephyr scheduler_dp_ll_tick() prototype. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved concurrency, memory-safety, allocation-accounting, and host-test linkage issues block approval.
11 open issues
Legacy stream paths bypass synchronization and race · New Host tests fail to link without telemetry hook mocks · New Cached output state becomes invalid after topology changes · New Reset clears pipeline statistics before they are reported · New Predictable temporary paths enable symlink file overwrites · New Copier sinks incorrectly reuse output pin 0 audio format · New EOF check accesses file state after releasing topology lock · New Tracked allocations are released without updating accounting · New Free accounting charges releases to the wrong memory pools · New Per-core telemetry updates race across concurrent DSP modules · New CTest commands run stale or missing binaries · New
What changed in this PR
This PR expands the SOF host library and testbench to support multi-pipeline audio graphs, multi-core LL/DP scheduling, platform simulation, and interactive remote control while retaining single-threaded Xtensa simulator support.
Changes:
- Adds graph-aware pipeline setup, multi-pin bindings, and expanded audio-format selection.
- Introduces host pthread schedulers, runtime platform configuration, allocation accounting, and telemetry.
- Integrates remote-control interfaces, CLI options, and tiered test-suite registration.
| File | Description |
|---|---|
zephyr/Kconfig |
Moves shared scheduling options out of Zephyr-only configuration. |
tools/tplg_parser/tokens.c |
Parses string bindings and processing domains. |
tools/tplg_parser/pipeline.c |
Parses IPC4 pipeline priority. |
tools/tplg_parser/include/tplg_parser/topology.h |
Adds graph, pin, and scheduling metadata. |
tools/tplg_parser/include/sound/asound.h |
Adds a minimal ALSA header stub. |
tools/tplg_parser/include/linux/types.h |
Adds UAPI compatibility definitions. |
tools/tplg_parser/CMakeLists.txt |
Adds header fallback and Xtensa compiler options. |
tools/tplg_parser/audio_formats.c |
Parses multi-pin binding names. |
tools/testbench/utils.c |
Integrates remote tick handling and control commands. |
tools/testbench/utils_ipc4.c |
Supports multi-PCM graph setup and cleanup. |
tools/testbench/topology_ipc4.c |
Adds core assignment, queue binding, and format selection. |
tools/testbench/testbench.c |
Adds server/platform options and telemetry reporting. |
tools/testbench/README.md |
Documents architecture, usage, and test tiers. |
tools/testbench/include/testbench/utils.h |
Extends runtime state and helper declarations. |
tools/testbench/include/testbench/topology_ipc4.h |
Defines the copier module identifier. |
tools/testbench/include/testbench/sof_remote_proto.h |
Defines remote protocol messages and operations. |
tools/testbench/include/testbench/remote_server.h |
Declares remote APIs and simulator stubs. |
tools/testbench/include/testbench/file.h |
Adds tone-generation phase state. |
tools/testbench/include/sound/asound.h |
Adds a minimal ALSA header stub. |
tools/testbench/include/linux/types.h |
Adds UAPI compatibility definitions. |
tools/testbench/file.c |
Adds tone generation, fan-out, capture hooks, and restart handling. |
tools/testbench/CMakeLists.txt |
Integrates server/client builds and CTest tiers. |
src/schedule/Kconfig |
Enables shared pipeline and DP options for host builds. |
src/platform/library/schedule/ll_schedule.c |
Adds cooperative multi-core context and telemetry. |
src/platform/library/schedule/ll_schedule_pthread.c |
Adds per-core LL worker threads. |
src/platform/library/schedule/host_schedule.h |
Declares host thread-affinity support. |
src/platform/library/schedule/dp_schedule.c |
Adds cooperative simulator DP scheduling. |
src/platform/library/schedule/dp_schedule_pthread.c |
Adds per-module DP worker threads. |
src/platform/library/schedule/CMakeLists.txt |
Selects pthread or cooperative schedulers. |
src/platform/library/lib/memory.c |
Uses runtime memory sizes and fixes secondary-core heaps. |
src/platform/library/lib/CMakeLists.txt |
Includes scheduler telemetry implementation. |
src/platform/library/lib/clk.c |
Converts time using configured core frequencies. |
src/platform/library/lib/alloc.c |
Adds tracked allocation and capacity enforcement hooks. |
src/platform/library/include/platform/lib/telemetry_sched.h |
Defines telemetry and platform-simulation APIs. |
src/platform/library/include/platform/lib/memory.h |
Sizes system heaps by configured core count. |
src/platform/Kconfig |
Defaults host builds to eight CPUs. |
src/ipc/ipc4/helper.c |
Validates runtime core IDs and enables host cross-core setup. |
src/include/sof/schedule/dp_schedule.h |
Exposes the DP tick hook outside Zephyr. |
src/include/kernel/tokens.h |
Adds processing-domain and resource tokens. |
src/audio/pipeline/pipeline-stream.c |
Instruments pipeline execution. |
src/audio/pipeline/pipeline-schedule.c |
Instruments DP execution and adjusts userspace flags. |
src/audio/google/Kconfig |
Permits Google CTC in host library builds. |
src/audio/google/google_ctc_audio_processing.c |
Corrects blob-size type and formatting. |
src/audio/google/google_ctc_audio_processing_ipc4.c |
Corrects IPC4 configuration-size handling. |
src/audio/copier/dai_copier.h |
Provides host-only DAI stubs. |
src/audio/copier/CMakeLists.txt |
Excludes hardware DAI copier code from host builds. |
src/audio/component.c |
Instruments module copy execution. |
src/audio/CMakeLists.txt |
Excludes legacy hardware DAI code from host builds. |
src/audio/buffers/ring_buffer.c |
Adds atomic host ring-buffer offset access. |
src/audio/buffers/comp_buffer.c |
Adds locking to shared-buffer callbacks. |
src/arch/host/include/arch/lib/cpu.h |
Adds runtime core hooks and external core-ID access. |
src/arch/host/configs/library_defconfig |
Enables eight-core, cross-core, and DP features. |
src/arch/host/CMakeLists.txt |
Adds Xtensa compiler compatibility options. |
posix/include/sof/lib/cpu.h |
Allows host IPC handling without core forwarding. |
Review effort: Balanced
| shared_buf_lock(buffer); | ||
| buffer_stream_writeback(buffer, commit_size); | ||
| comp_update_buffer_produce(buffer, commit_size); | ||
| shared_buf_unlock(buffer); |
There was a problem hiding this comment.
| #if CONFIG_LIBRARY | ||
| uint64_t telem_start = host_telem_mod_entry(dev, false); |
There was a problem hiding this comment.
Fixed in 7a9715a: guarded telemetry hooks in component.c, pipeline-stream.c, and pipeline-schedule.c with #if CONFIG_LIBRARY && !defined(UNIT_TEST) and added inline no-op stubs in telemetry_sched.h when UNIT_TEST is defined.
| if (next_control_ns && out_stat) { | ||
| time_ns = (int64_t)(samples_to_ns * out_stat->n); |
There was a problem hiding this comment.
Fixed in bb95a2a: removed cached out_stat across ticks in pipline_test() and handle_remote_run_test(), and instead compute tp->last_output_time_ns and tp->has_output_file from tp->fw[0].state while holding g_tb_lock inside tb_schedule_pipeline_check_state().
| tb_remote_server_stop(); | ||
|
|
||
| if (tp->tplg_file) { | ||
| err = tb_set_reset_state(tp); |
There was a problem hiding this comment.
Fixed in bb95a2a: moved test_pipeline_stats() (in testbench.c) and format_remote_test_summary() (in remote_server.c) before tb_set_reset_state(tp) so end-of-run sample, copy, and cycle counts are reported before file_reset() clears them.
| tp->frame_fmt = tplg_find_format(tp->bits_in); | ||
| } | ||
| if (!tp->input_file_num) { | ||
| FILE *sfp = fopen("/tmp/sof_tb_srv_in.raw", "wb"); |
There was a problem hiding this comment.
Fixed in bb95a2a: server mode now creates a private per-instance directory via mkdtemp("/tmp/sof-tb-XXXXXX") (tp->server_tmp_dir) for default input/output/LLEXT files and cleans up the files and directory on exit.
| tb_remote_tick_end(tp, cycles1 - cycles0, delta_ns); | ||
|
|
||
| /* Check if all file components are running */ | ||
| return tb_is_file_component_at_eof(tp); | ||
| return !tb_remote_should_continue(tp, tb_is_file_component_at_eof(tp)); |
There was a problem hiding this comment.
Fixed in bb95a2a: tb_schedule_pipeline_check_state() now evaluates tb_is_file_component_at_eof(tp) before tb_remote_tick_end() releases g_tb_lock.
| if (host_platform_account_alloc && !host_platform_account_alloc(flags, actual)) { | ||
| free(ptr); | ||
| return NULL; | ||
| } |
There was a problem hiding this comment.
Fixed in bb95a2a: replaced plain free() calls on rzalloc()-allocated ccd, cd, and dd structures in tools/testbench/file.c with rfree().
| if (host_platform_account_free) | ||
| host_platform_account_free(malloc_usable_size(ptr)); |
There was a problem hiding this comment.
Fixed in 7a9715a: host_platform_account_alloc() and host_platform_account_free() now record each tracked allocation's pointer, byte size, and charged pool (HPSRAM, LPSRAM, or DRAM) so freeing an allocation decrements the exact pool that was charged.
| host_telem_core_dp_run(task->core, core_dp_cnt, | ||
| (t1 > t0) ? (t1 - t0) : 1); | ||
| pthread_mutex_lock(&dp_sch->lock); |
There was a problem hiding this comment.
Fixed in 7a9715a: protected host_telem_core_dp_run(), host_telem_core_ll_tick(), host_telem_pipe_begin(), host_telem_pipe_end(), and _sim_dp_run_cycles / _sim_ll_tick_cycles updates in host_telem_mod_exit() with TELEM_LOCK() / TELEM_UNLOCK().
| add_test(NAME testbench_tier5_controls_remote | ||
| COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/run_test_suite.sh --tier 5) |
There was a problem hiding this comment.
Fixed in bb95a2a: updated all add_test() entries in tools/testbench/CMakeLists.txt to pass TB_BIN=$<TARGET_FILE:${testbench}> and TB_CTL_BIN=$<TARGET_FILE:sof-tb-ctl> via ${CMAKE_COMMAND} -E env.
98c98d9 to
bb69d57
Compare
…and Google CTC on host Update audio buffers and modules for multi-core and multi-threaded host library execution: - Add atomic synchronization for shared cross-core comp_buffer and ring_buffer operations when running on host pthreads (CONFIG_LIBRARY). - Provide inline no-op stubs in dai_copier.h and exclude hardware dai-legacy.c / copier_dai.c on CONFIG_LIBRARY builds where file copiers handle endpoint I/O. - Allow CONFIG_COMP_GOOGLE_CTC_AUDIO_PROCESSING on CONFIG_LIBRARY builds and fix comp_get_data_blob() size_t argument types. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…st library Support cross-core pipeline/module creation and runtime-configurable DSP core counts on host library (CONFIG_LIBRARY) builds: - Add weak host_cpu_* hooks in arch/host/include/arch/lib/cpu.h so host testbench builds can configure core count and track enabled core masks at runtime while standalone unit tests fall back to CONFIG_CORE_COUNT. - Validate pipeline and module core_id against arch_cpu_num_cores() and enable target cores in ipc4_comp_new_config() and ipc_pipeline_new() under CONFIG_LIBRARY. - Allow primary-core IPC thread to allocate shared cross-core buffers and provide host library ll_block/ll_unblock definitions. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…, and platform simulation Add multi-core LL and preemptive DP pthread schedulers, per-core/pipe/module scheduling and buffer telemetry, CPC workload execution time simulation, and hardware platform preset / memory / clock accounting to the host library platform: - Implement per-core LL worker pthreads (ll_schedule_pthread.c) and per-module preemptive DP pthreads (dp_schedule_pthread.c), alongside single-threaded Xtensa simulator fallbacks (ll_schedule.c, dp_schedule.c). - Implement scheduling and entry/exit pin buffer telemetry, CPC/SRAM/DRAM workload simulation, and Intel DSP platform presets (tgl, tgl-h, mtl/arl, lnl, ptl, wcl, nvl, nvl-s, default) in telemetry_sched.c. - Track live/peak heap allocations and enforce HPSRAM/LPSRAM/DRAM capacity limits in alloc.c and memory.c, and compute clock_*_to_ticks() from the active core frequency in clk.c. - Enable 8-core, PIPELINE_2_0, ZEPHYR_DP_SCHEDULER, and CROSS_CORE_STREAM in library_defconfig. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…multi-format arrays Enhance libsof_tplg_parser to parse IPC4 multi-pin bindings, Data Processing (DP) scheduling domain/stack/heap tokens, pipeline priority, and multi-format audio arrays: - Define SOF_TKN_COMP_NO_WNAME_IN_KCONTROL_NAME, SOF_TKN_COMP_SCHED_DOMAIN, SOF_TKN_COMP_DOMAIN_ID, SOF_TKN_COMP_STACK_BYTES, and SOF_TKN_COMP_HEAP_BYTES in src/include/kernel/tokens.h. - Parse input/output pin binding arrays and multiple input/output audio formats in tools/tplg_parser/audio_formats.c. - Parse SOF_TKN_SCHED_PRIORITY in tools/tplg_parser/pipeline.c and add string and processing domain token helpers in tools/tplg_parser/tokens.c. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…d remote server Enhance sof-testbench4 with multi-pipeline DAG execution, cross-core and DP pipeline setup, hardware platform/workload simulation flags, and an interactive SOFR v2 TCP network server: - Support arbitrary multi-pipeline DAGs, multi-pin module bindings, DP module initialization, and pure IPC4 pipeline construction without -t. - Add CLI flags for platform presets (-A <platform|list>), core count (-N <cores>), memory capacities (-m <hpsram[,lpsram[,dram]]>), fractional MHz/kHz/Hz core frequencies (-F), workload simulation (-w, -M), and interactive server mode (-I, -S <port>, -W). - Implement the SOFR v2 TCP server (remote_server.c) exposing IPC4 TX/RX, large config TLVs, emulated SRAM windows, winstream shell, composite telemetry snapshots, ALSA/DAPM/Compress/Kcontrol introspection, probe extraction, and live topology/pipeline/platform control. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…opology suite Add the standalone sof-tb-ctl TCP client utility supporting all interactive testbench remote server commands (topology load, core count, memory limits, clock frequencies, platform presets, PCM file bindings, batch run, single-step ticks, kcontrol get/set, pipeline graph queries, and per-core telemetry). Also add the 5-tier 70-test verification suite (run_test_suite.sh) covering single-pipeline modules, complex DAGs, LL+DP scheduling, multi-core affinity, and persistent single-session sof-tb-ctl remote verification. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
bb69d57 to
f71e750
Compare
|
The x86 build works well. But I get with xt-clang build of testbench (for MTL) this: |


Summary
This series upgrades
sof-testbench4and the SOF host library (CONFIG_LIBRARY) to support arbitrary multi-pipeline IPC4 DAG topologies, multi-core Low-Latency (LL) and Data Processing (DP) pthread scheduling, hardware platform/workload simulation, an interactiveSOFRv2 TCP network server, and thesof-tb-ctlCLI utility, while preserving single-threaded Xtensa simulator (-DTESTBENCH_XTENSA_SIM=ON) compatibility.Patch Series Overview
schedule: Kconfig: allow PIPELINE_2_0 and DP scheduler on host library buildsCONFIG_PIPELINE_2_0,CONFIG_ZEPHYR_DP_SCHEDULER,CONFIG_ZEPHYR_DP_SCHEDULER_MIN_STACK_SIZE, andCONFIG_CROSS_CORE_STREAMfromzephyr/Kconfigtosrc/schedule/Kconfigso they can be enabled on both Zephyr ACE firmware targets and host library (CONFIG_LIBRARY) testbench builds.CONFIG_MP_MAX_NUM_CPUSto8whenCONFIG_LIBRARYis enabled.audio: library: support shared cross-core buffers, DAI copier stubs, and Google CTC on hostcomp_bufferandring_bufferoperations when running on host pthreads (CONFIG_LIBRARY).dai_copier.hand excludes hardwaredai-legacy.c/copier_dai.conCONFIG_LIBRARYbuilds where file copiers handle endpoint I/O.CONFIG_COMP_GOOGLE_CTC_AUDIO_PROCESSINGonCONFIG_LIBRARYbuilds.ipc4: helper: support cross-core streams and runtime core count on host libraryhost_cpu_*hooks inarch/host/include/arch/lib/cpu.hso host testbench builds can configure core count and track enabled core masks at runtime while standalone unit tests fall back toCONFIG_CORE_COUNT.core_idagainstarch_cpu_num_cores()and enables target cores inipc4_comp_new_config()andipc_pipeline_new()underCONFIG_LIBRARY.platform: library: add multi-core pthread LL/DP schedulers, telemetry, and platform simulationll_schedule_pthread.c) and per-module pthread DP (dp_schedule_pthread.c) schedulers alongside single-threadedll_schedule.c/dp_schedule.cfor Xtensa simulator builds (-DTESTBENCH_XTENSA_SIM=ON).telemetry_sched.c), CPC-based workload execution time simulation (SRAM vs. DRAM fetch models), and runtime hardware platform presets (tgl,tgl-h,mtl/arl,lnl,ptl,wcl,nvl,nvl-s) enforcing core count, HPSRAM/LPSRAM/DRAM capacity limits (alloc.c,memory.c), and DSP core clock frequencies (clk.c).tools: tplg_parser: support IPC4 pin bindings, DP domain tokens, and multi-format arrayslibsof_tplg_parserto parse IPC4 multi-pin input/output bindings, DP scheduling domain/stack/heap tokens, pipeline priority, and multi-format audio arrays.<sound/asound.h>UAPI stub for non-host Xtensa cross-compiler sysroots.tools: testbench: add multi-pipeline DAG, DP, platform simulation, and remote serversof-testbench4with multi-pipeline DAG execution, cross-core and DP module setup, pure IPC4 pipeline construction without-t, and CLI flags for platform presets (-A), core count (-N), memory sizes (-m), core frequencies (-F), workload simulation (-w,-M), and interactive server mode (-I,-S,-W).SOFRv2 TCP server (remote_server.c) exposing IPC4 TX/RX, large config TLVs, emulated SRAM windows, winstream shell, composite telemetry snapshots, ALSA/DAPM/Compress/Kcontrol introspection, probe extraction, and live topology/pipeline/platform control.tools: testbench: add sof-tb-ctl CLI tool and 70-test multi-core/DP topology suitesof-tb-ctlC CLI client (tb_ctl.c) supporting all interactive testbench remote server operations, batch script execution (-f), and interactive REPL mode.run_test_suite.sh) covering all 26 built-in audio modules, complex multi-pipeline DAGs, LL+DP combinations, multi-core execution across DSP Cores 0–2, and persistent single-session remote control.Verification
sof-testbench4+sof-tb-ctl):-Wall -Werror(0 warnings, 0 errors).tools/testbench/run_test_suite.sh(single persistentsof-testbench4 -Isession, 0 restarts): 70 / 70 passed (Tier 1: 46/46, Tier 2: 7/7, Tier 3: 6/6, Tier 4: 6/6, Tier 5: 5/5).-DTESTBENCH_XTENSA_SIM=ON):-DTESTBENCH_XTENSA_SIM=ONand verified cross-compilation withxtensa-intel_tgl_adsp_zephyr-elf-gcc(-Wall -Werror, 0 warnings, 0 errors).tools/testbench/run_test_suite.sh --per-processacross Tiers 1–4: 65 / 65 passed.