Why AI Accelerators Stall on Data Movement, Not Math

The datasheet promises so many trillion operations per second. The first real model runs at a fraction of that, and nothing is broken: no assertion fires, no data is wrong. The multipliers are simply waiting. The math is usually the easy part; moving operands in and results out on time is where the cycles go.

This article covers how to reason about that gap before RTL is frozen, the structures that close it, and how to measure where cycles go instead of guessing.

Utilization is a pipeline property

Peak TOPS assumes every MAC does useful work every cycle. Utilization is the fraction of cycles where that actually happens. In any given cycle, a compute engine is in one of four states:

• Busy: operands present, output space available, doing useful work.
• Stalled on input: it needs operands that have not arrived.
• Stalled on output: it has results but nowhere to put them.
• Idle: nothing scheduled (between layers, waiting for configuration).

Only the first state counts toward delivered TOPS. The useful question is not "how fast is the array?" but "which of the other three states eats the cycles, and why?"

Roofline thinking, in plain terms

The roofline model needs two numbers: peak compute P (operations per second) and memory bandwidth B (bytes per second from DDR or HBM). Each workload has an arithmetic intensity I: operations per byte moved from that memory. The attainable rate is:

attainable = min(P, I × B)

The crossover point, P / B, is the ridge. Below it you are memory-bound and extra MACs buy nothing. Above it you are compute-bound.

A worked example (illustrative numbers)

Take an accelerator with P = 1 TOPS (10^12 int8 operations per second, a multiply and an add counting as two) fed by one 64-bit DDR4-3200 channel, theoretical peak 25.6 GB/s. These numbers are illustrative, not measurements of any product.

Ridge: 10^12 / 25.6×10^9 ≈ 39 ops/byte. Below that, no RTL can reach peak on this memory system.

Case 1: matrix-vector, batch 1. A fully connected layer with int8 weights uses each weight byte exactly once, for one multiply and one add. Intensity is about 2 ops/byte (activations are small next to the weights). Attainable rate: 2 × 25.6 GB/s = 51.2 GOPS, about 5% of peak. Batching b inputs reuses each weight b times and raises intensity to roughly 2b ops/byte. That is weight reuse, and it is why batch size is a hardware question.

Case 2: tiled matrix multiply. Compute C = A × B with int8 inputs, int32 outputs, and inner dimension K = 256. Use output-stationary tiles: for each T×T output tile, stream a T×K strip of A and a K×T strip of B, accumulate on chip, and write the tile once.

• Operations per tile: 2·T²·K
• Bytes per tile: T·K + K·T (inputs) + 4·T² (int32 outputs)

With T = 16: 131,072 ops over 4,096 + 4,096 + 1,024 = 9,216 bytes, so I ≈ 14.2 ops/byte. Attainable: 14.2 × 25.6 ≈ 364 GOPS, about 36% of peak. Memory-bound.

With T = 64: 2,097,152 ops over 16,384 + 16,384 + 16,384 = 49,152 bytes, so I ≈ 42.7 ops/byte, just past the ridge. Now the same array can, in principle, be compute-bound.

Same MACs, same DDR. The difference is on-chip buffer size, which sets tile size, which sets reuse. Use achieved bandwidth, not the theoretical figure; it depends heavily on access pattern. The roofline is an upper bound; the rest of this article is about the cycles that keep you below it.

Where the cycles go

Serialized load, compute, store

The simplest controller loads a tile, computes it, and writes it back, in sequence. Time per tile is the sum of the three, and the engine is busy only during the middle phase.

Double (ping-pong) buffering fixes this: two copies of each input and output buffer. While the engine computes from buffer 0, the DMA fills buffer 1 and drains the previous result. Time per tile becomes roughly max(load, compute, store) instead of their sum. Two caveats:

1. The second copy costs real SRAM, which competes with tile size, which sets intensity. Budget both together.
2. If load time exceeds compute time, double buffering hides latency but not missing bandwidth; you are back on the sloped part of the roofline.

DMA latency and outstanding transactions

A DMA engine that issues one AXI read burst and waits for its last beat before issuing the next is limited by latency, not bandwidth. The fix is to keep enough requests in flight. Little's law gives the target:

bytes in flight = bandwidth × round-trip latency

Illustrative example: a 512-bit (64-byte) AXI read port at 300 MHz can move 19.2 GB/s. If the round trip from AR handshake to first data beat is 200 ns (60 cycles), you need 19.2 GB/s × 200 ns = 3,840 bytes in flight. With 16-beat bursts (1 KB each), that is four outstanding reads. With 4-beat bursts, fifteen. Measure your own latency; it varies with the controller, interconnect, and traffic.

Practical consequences:

• Make bursts as long as the access pattern allows; they amortize address and controller overhead. An AXI4 INCR burst may not cross a 4 KB boundary, so the DMA must split there.
• Make the outstanding-transaction limit a parameter, and check that the interconnect and memory controller actually accept that many.
• If each descriptor costs setup cycles and moves one small strided row, overhead dominates. Prefer descriptors that cover a whole tile, or a 2D/3D DMA that walks strides in hardware.

Buffer and credit sizing for latency

If the DMA reserves receive space before issuing a read (a credit scheme, so returning data never blocks the interconnect), the buffer must hold the full bytes in flight: 3,840 bytes, or 60 64-byte beats, in the example. Fewer credits cap throughput at credits ÷ round-trip cycles beats per cycle. On the write-back side, a result FIFO shallower than write latency plus a burst turns write backpressure into compute stalls.

Bank conflicts in on-chip SRAM

On-chip buffers are split into banks so several readers can access them per cycle. If two same-cycle accesses hit one bank, one waits. A classic case: four banks interleaved on the low address bits, and a column walk through a row-major tile whose row stride is a multiple of four words. Every access lands in the same bank, and the read rate drops to a quarter.

Fixes: pad the row stride by one word so consecutive rows rotate across banks, use an XOR bank hash, or lay data out in the order the array consumes it. A short script that replays each dataflow's address sequence and counts conflicts is far cheaper than finding this in a waveform.

Weight versus activation reuse

The dataflow decides which operand stays put. Weight-stationary designs hold weights in the array and stream activations, which pays off when weights are reused across many inputs. Output-stationary designs, as in the example above, accumulate partial sums in place and save write bandwidth. Compute bytes moved per operation for every real layer shape, not just the largest; small and late layers are often where utilization collapses.

FPGA-specific notes

The memory controller usually runs in its own clock domain, behind an asynchronous FIFO whose synchronizer latency adds to the round trip and raises the bytes-in-flight target. DDR efficiency drops with read/write turnarounds and row misses, so group reads and writes into long runs. HBM offers many independent channels, and the design must spread traffic across them to use that bandwidth. More on that side of the work is on the FPGA development page.

Instrument it: stall counters

All of the above is a hypothesis until measured. The block below classifies each cycle into the four states. They are mutually exclusive, so the four counters always sum to the total, and if the total saturates every counter freezes so the ratios stay valid.

module stall_counters #(parameter int W = 32) (
  input  logic         clk,
  input  logic         rst_n,
  input  logic         clear,     // synchronous clear of all counters
  input  logic         enable,    // count only inside the measurement window
  input  logic         busy,      // engine did useful work this cycle
  input  logic         in_wait,   // engine needs operands, input not valid
  input  logic         out_wait,  // engine holds results, output not ready
  output logic [W-1:0] cyc_total,
  output logic [W-1:0] cyc_busy,
  output logic [W-1:0] cyc_stall_in,
  output logic [W-1:0] cyc_stall_out,
  output logic [W-1:0] cyc_idle
);
  wire sat = &cyc_total;          // freeze everything when the total saturates
  wire cnt = enable && !sat;

  // Priority: busy > stalled on output > stalled on input > idle.
  // Output pressure wins over input starvation: it is the upstream cause.
  wire c_busy = busy;
  wire c_out  = !busy && out_wait;
  wire c_in   = !busy && !out_wait && in_wait;
  wire c_idle = !busy && !out_wait && !in_wait;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      cyc_total     <= '0;
      cyc_busy      <= '0;
      cyc_stall_in  <= '0;
      cyc_stall_out <= '0;
      cyc_idle      <= '0;
    end else if (clear) begin
      cyc_total     <= '0;
      cyc_busy      <= '0;
      cyc_stall_in  <= '0;
      cyc_stall_out <= '0;
      cyc_idle      <= '0;
    end else if (cnt) begin
      cyc_total <= cyc_total + 1'b1;
      if (c_busy) cyc_busy      <= cyc_busy      + 1'b1;
      if (c_in)   cyc_stall_in  <= cyc_stall_in  + 1'b1;
      if (c_out)  cyc_stall_out <= cyc_stall_out + 1'b1;
      if (c_idle) cyc_idle      <= cyc_idle      + 1'b1;
    end
  end
endmodule

Drive busy from the engine's "did useful work" condition, in_wait from "needs an operand and input is not valid," and out_wait from "result valid and output not ready." Put one instance on each major stage (DMA read, compute, write-back) and expose the counters as registers.

Reading the counters

• High stall-in on compute: operands are late. Check the DMA read counters. If the DMA itself is starved, look at outstanding transactions and burst size; if it runs at its bandwidth limit, you are on the roofline slope and need more reuse.
• High stall-out on compute: write-back cannot keep up. Check result FIFO depth, write burst size, and whether reads and writes share a memory port.
• Busy high, throughput low: "busy" includes non-useful work such as padding or partial tiles. Tighten the definition.
• Idle dominates: the bottleneck is control or software: descriptor setup, interrupts between layers, configuration writes. Queues of pre-built descriptors usually help more than faster datapaths.

Where one stage shows heavy output stalls and the next shows heavy input stalls, the cycles are lost in between.

Utilization investigation checklist

• [ ] Peak compute, achieved bandwidth (measured, not theoretical), and the ridge point are written down.
• [ ] Arithmetic intensity is computed for every real layer shape, not just the largest.
• [ ] Tile sizes are chosen against the SRAM budget with double buffering included.
• [ ] Load, compute, and store overlap; per-tile time is close to the max of the three, not their sum.
• [ ] Outstanding reads × burst bytes ≥ bandwidth × measured round-trip latency.
• [ ] Bursts are as long as the access pattern allows and split correctly at 4 KB boundaries.
• [ ] Receive buffers and credits cover the full round trip; write-back FIFOs cover write latency plus a burst.
• [ ] SRAM bank mapping is checked against every dataflow's address sequence.
• [ ] Clock-domain crossings at the memory controller are included in the latency budget.
• [ ] Busy / stall-in / stall-out / idle counters exist on every major stage and are readable on hardware.

Where this comes from

I learned most of this the direct way: building an FPGA AI accelerator from the ground up, after RTL design, integration and CPU work at Intel and iPhone camera hardware at Samsung. In every one of those, the numbers that mattered were set at the boundaries between blocks, not inside the arithmetic. More on accelerator work is on the AI accelerators page.

Accelerator delivering less than the math says? Send a short description of the design and the gap you are seeing through the contact page, and I'll reply with where I would start looking.