Asymmetric Multiprocessing (AMP): Managing Inter-Processor Communication Without Latency Spikes
11 September 2026 · Lance Harvie
Modern System-on-Chip (SoC) architectures increasingly rely on Asymmetric Multiprocessing (AMP) to meet the dual demands of high-level application processing and ultra-low-latency real-time control. Devices like the STMicroelectronics STM32MP1, NXP i.MX8M, and TI Sitara series pair high-performance 64-bit application processors (such as ARM Cortex-A series running Linux) with deterministic microcontroller cores (such as ARM Cortex-M series running FreeRTOS or bare-metal code).
This heterogeneous approach offers the best of both worlds on paper: Linux handles complex networking, graphical user interfaces, and cloud connectivity, while the RTOS manages time-critical sensor sampling, motor control loops, and safety-critical actuation. However, the key engineering challenge lies in the boundary between these two isolated processing worlds: Inter-Processor Communication (IPC).
When an application core and a real-time core must exchange telemetry, control commands, or high-throughput sensor streams, naive IPC implementations frequently introduce unpredictable latency spikes. A single unexpected 50-microsecond delay in message delivery can cause missed deadline constraints, dropped audio frames, or catastrophic control loop instability in motor drives. Achieving sub-microsecond, zero-jitter IPC performance across an AMP boundary requires a deep, hardware-aware understanding of shared memory structures, cache coherence protocols, interrupt handling overhead, and software framework optimizations.
Root Causes of Latency Spikes in Heterogeneous IPC
To eliminate latency spikes in AMP systems, firmware and system architects must first identify the physical and software mechanisms that introduce non-deterministic delays during inter-core data transfers.
1. Cache Maintenance Operations and Memory Coherency
In most heterogeneous SoCs, the Cortex-A application cores feature hardware-managed L1/L2 cache systems governed by Memory Management Units (MMUs), while the Cortex-M cores operate with either no cache, a simpler software-managed L1 cache, or Memory Protection Units (MPUs). Because there is rarely hardware cache coherency spanning across heterogeneous core clusters (e.g., between ARM v8-A ACE domains and ARM v7-M non-coherent buses), data written by one core to shared DRAM or SRAM is not automatically visible to the other.
Software-managed cache coherence relies on explicit cache line cleaning (flushing dirty lines to main memory) and cache line invalidation (forcing fresh reads from main memory). Latency spikes occur when:
Large Buffer Clean/Invalidate Operations: Flushing large, continuous memory blocks causes the CPU to stall during write-backs, blocking critical execution threads.
False Sharing: Shared variables and private data share the same cache line (typically 32 or 64 bytes). Flushing or invalidating the line on Core A unintentionally invalidates unrelated data accessed by Core B, triggering expensive cache misses.
Bus Contention: Concurrent cache eviction operations saturate the internal SoC interconnect (e.g., AXI/AHB crossbars), throttling memory bandwidth for time-critical DMA transfers.
2. Hardware Mailbox Interrupt Jitter and Overhead
Heterogeneous cores signal each other via Hardware Mailbox IP blocks (also known as Inter-Processor Communication Controllers or IPCCs). Sending a message involves writing a trigger register in the mailbox module, which asserts an interrupt line on the receiving processor.
Interrupt-driven IPC suffers from several sources of latency variation:
Interrupt Service Routine (ISR) Latency: On the Linux side, kernel preemptibility, high interrupt loads, and lock contention can delay the execution of the hardware mailbox ISR by hundreds of microseconds.
Context Switching Costs: Wake-up delays incurred when transitioning a user-space daemon or kernel thread from sleeping to executing upon receiving a hardware interrupt.
Interrupt Storms: High-frequency messaging (e.g., issuing an interrupt for every 64-byte payload at 50 kHz) overwhelms the real-time core’s interrupt controller, consuming massive CPU cycle budgets purely in ISR entry/exit overhead.
3. Mutex and Spinlock Lock Contention
When multiple cores attempt to read or write to shared data structures simultaneously, software lock primitives are required to ensure data integrity. Standard spinlocks utilizing atomic bus transactions (such as ARM LDREX/STREX instructions) can stall the executing core indefinitely if the opposing core holds the resource.
Cross-core lock contention is exacerbated when an OS running preemption or variable-frequency scaling (DVFS) holds a shared spinlock and is subsequently interrupted or context-switched, trapping the real-time RTOS core in an unbounded polling loop.
Deconstructing Standard IPC Frameworks: RPMsg, VirtIO, and OpenAMP
To simplify development, embedded platforms standardise on open frameworks like OpenAMP, which builds upon the Linux kernel’s remoteproc and RPMsg (Remote Processor Messaging) subsystems.
The VirtIO and RPMsg Stack
OpenAMP utilizes VirtIO as its virtual device abstraction layer. VirtIO organizes shared memory into unidirectional lock-free ring buffers called vrings:
TX vring: Transmits messages from the Master (typically Linux) to the Remote processor (RTOS).
RX vring: Transmits messages from Remote to Master.
Each vring consists of three components in shared SRAM or DDR:
Descriptor Table: Holds pointers to physical memory buffers alongside buffer lengths and flags.
Available Ring: Stores indices into the descriptor table indicating buffers filled by the sender and ready for processing.
Used Ring: Stores indices into the descriptor table indicating buffers processed by the receiver and freed back to the sender.
On top of VirtIO sits RPMsg, a datagram-based messaging bus that assigns logical source and destination endpoints (similar to UDP ports).
The Overhead of Generic Frameworks
While frameworks like RPMsg provide excellent portability across SoCs, their default configurations are rarely optimized out-of-the-box for low-jitter control systems:
Buffer Copy Operations: By default, RPMsg copies data from user space into a kernel buffer, then copies that buffer into a pre-allocated VirtIO shared memory buffer, and finally the remote core copies it into its local application buffer. These multiple memcpy operations consume valuable memory bandwidth and clock cycles.
Fixed Dynamic Allocation Schemes: Dynamically acquiring and returning VirtIO descriptors adds algorithmic overhead execution time, but with memory access overhead that varies depending on bus saturation).
Technical Strategies for Eliminating IPC Latency Spikes
Achieving deterministic, sub-microsecond IPC performance requires re-architecting the memory layout, data structures, and signaling primitives. Below are proven, production-grade strategies used in high-reliability AMP systems.
Strategy 1: Lock-Free, Single-Producer Single-Consumer (SPSC) Ring Buffers
To eliminate spinlock contention across core boundaries, replace shared locks with lock-free Single-Producer Single-Consumer (SPSC) ring buffers.
In an SPSC model:
Core A (Producer) is the only entity allowed to modify the Write_Pointer.
Core B (Consumer) is the only entity allowed to modify the Read_Pointer.
Memory barriers (__DSB() or Data Synchronization Barriers) ensure that data writes to shared RAM are physically committed before updated index pointers become visible to the opposing core.
By decoupling the read and write indexes, neither core ever stalls waiting for a mutual exclusion lock.
Strategy 2: Zero-Copy Shared Memory Architecture
To bypass the latency penalties of memcpy inside interrupt routines, implement zero-copy ring buffers using static memory pools located in non-cacheable On-Chip SRAM (AXI SRAM or TCM).
Instead of transferring payload data through message queues, cores exchange fixed-size buffer pointers or indices:
Core A acquires a free buffer index from a shared static descriptor pool.
Core A writes sensor data directly into the SRAM buffer using DMA hardware.
Core A pushes only the buffer index into the SPSC queue and triggers the mailbox IRQ.
Core B receives the IRQ, reads the index, processes the data directly in SRAM, and returns the index to the free pool.
Zero-copy designs guarantee deterministic time complexity, independent of payload size.
Strategy 3: Cache-Aware Alignment and Non-Cacheable Partitioning
Improper cache management destroys deterministic execution. Modern AMP memory maps should be divided into distinct functional zones:
Cache Line Alignment Rules
When shared buffers must reside in cacheable DDR memory for bandwidth reasons, strict memory alignment rules must be enforced:
Align to Cache Lines: Every IPC descriptor structure and buffer address MUST be aligned to the maximum system cache line size (e.g., 64 bytes for ARM Cortex-A53).
Pad Structures: Pad message structures so that individual fields modified by different cores do not occupy the same cache line.
Batch Invalidation Operations: Perform invalidation once per transaction burst, using start and end addresses rounded to cache line boundaries:
// Example: Invalidate Cache for a Shared Buffer on Cortex-M7
void prepare_buffer_for_read(uint32_t addr, uint32_t size) {
uint32_t start_addr = addr & ~(ARM_CACHE_LINE_SIZE — 1);
uint32_t end_addr = (addr + size + ARM_CACHE_LINE_SIZE — 1) & ~(ARM_CACHE_LINE_SIZE — 1);
SCB_InvalidateDCache_by_Addr((uint32_t *)start_addr, end_addr — start_addr);
}
Strategy 4: Hybrid Signaling — Interrupt Moderation and Polling Modes
Relying strictly on hardware interrupts for high-frequency IPC creates unsustainable interrupt context-switching overhead. To maintain zero jitter, employ a hybrid signaling strategy:
Low-Frequency / Event-Driven Traffic (Control Commands): Use standard Hardware Mailbox Interrupts. The receiving core sleeps in a low-power state until an explicit event arrives.
High-Frequency Traffic (Sensor Telemetry Streams at >10 kHz):
Interrupt Moderation (Coalescing): The producing core accumulates $N$ packets in the SPSC ring buffer before firing a single hardware mailbox IRQ, drastically reducing interrupt firing rates.
Deterministic Polling: Rather than generating IRQs, the RTOS core reads incoming control pointers directly within a dedicated, high-priority periodic task loop (e.g., executing at a fixed 10 kHz tick rate). This completely eliminates ISR entry/exit overhead and context-switching jitter on the real-time core.
Practical Implementation: High-Throughput Motor Control Telemetry
Consider a practical example: An industrial motor drive where an ARM Cortex-M4 core runs a 20 kHz Field-Oriented Control (FOC) loop and streams high-frequency phase current data to a Cortex-A53 running Linux for predictive maintenance AI inference.
Hardware Setup
Processor: NXP i.MX8M Dual (1x Cortex-A53 @ 1.5 GHz, 1x Cortex-M4 @ 400 MHz).
Shared Memory: 64 KB On-Chip OCRAM, configured via MPU/MMU as Non-Cacheable, Shared, Device Memory.
Signaling: Messaging Unit (MU) peripheral.
The Optimization Pipeline
Memory Reservation: The linker scripts on both Linux and FreeRTOS reserve an exact 64 KB region in OCRAM (0x20200000 to 0x2021FFFF).
Descriptor Allocation:
0x20200000: SPSC Ring Buffer Control Structure (128 bytes, aligned).
0x20200080: Array of 64 fixed-size buffers (1 KB each).
3. Execution Sequence:
M4 Core (Producer):
Reads ADC inputs at 20 kHz inside a hardware timer ISR.
Writes raw phase data into Buffer Index K in OCRAM via DMA (0% CPU load).
Executes spsc_push(&telemetry_ring, K).
If the ring contains 10 unread items, the M4 writes to the MU register to trigger an interrupt on the Cortex-A53.
A53 Core (Consumer):
Linux MU driver handles the ISR, clearing the MU flag.
Passes the buffer indices to a dedicated real-time user-space thread (SCHED_FIFO).
The application reads the OCRAM buffers directly via mmap() on /dev/mem or a custom UIO (Userspace I/O) driver, bypassing kernel memcpy overhead entirely.
Releases indices back to the free ring buffer.
Performance Benchmarks: Unoptimized vs. Optimized IPC
Press enter or click to view image in full size
Architectural Checklist for Zero-Jitter AMP Systems
Before shipping a heterogeneous AMP production system, ensure your firmware team has validated the following items:
[ ] Memory Protection Mappings: Is shared SRAM explicitly marked as Non-Cacheable or Write-Through across both core MMU/MPU tables?
[ ] Lock Audit: Have all cross-core spinlocks been eliminated in favor of lock-free Single-Producer Single-Consumer (SPSC) ring buffers?
[ ] Cache Line Alignment: Are all shared ring buffer descriptors and payload structures aligned to the hardware cache line boundary (e.g., 32 or 64 bytes)?
[ ] Zero-Copy Pipelines: Are large sensor payloads transferred via pointer exchange rather than value copying?
[ ] Interrupt Rate Limiting: Is high-frequency telemetry using interrupt moderation or fixed-rate task polling to prevent interrupt storms on real-time cores?
[ ] Memory Barriers: Are appropriate bus synchronization barriers (__DSB(), __DMB()) placed around ring buffer index updates to ensure write order integrity across dissimilar pipeline architectures?
By systematically addressing cache behavior, locking mechanisms, and interrupt handling, embedded engineers can harness the full processing power of heterogeneous SoCs while retaining absolute deterministic real-time control.
Need Elite Embedded Talent for Your Next Heterogeneous Architecture?
Designing deterministic AMP systems requires deep expertise in low-level firmware, kernel driver design, and RTOS internals. RunTime Recruitment connects engineering organizations with world-class embedded software and hardware specialists who understand how to optimize high-performance SoCs from bare-metal to Linux.
Connect with RunTime Recruitment today to secure the specialist talent needed to elevate your engineering team.