Megakernels on Mac: What Fusion Saves and What It Costs

Bandwidth and memory measurements

The useful starting point is how much data the selected execution path moves, and which part of that movement fusion can remove. These experiments separate an observed streaming rate, a source-derived payload estimate, and the time spent between real kernels.

The streaming probe swept three access patterns, four grid sizes and two threadgroup sizes. Each number above is the best configuration in its run, rather than a repeated sample of one fixed configuration. An older 8 GiB probe reported 512.9 GB/s, showing how strongly the access pattern and probe geometry can affect the result.

The saved streaming receipts do not record a sustained warm-up duration or a thermal plateau. They support an observed bandwidth reference, not a validated warmed maximum. A separate staging fixture recorded 3.02329 accumulated GPU seconds of warm-up outside its timing window; that preparation applies to that fixture alone. Measurement notes and source receipts

The 27.65 ms estimate

The projection audit counted 14.434467840 GB of target payload and 1.323417600 GB of draft and context payload. The target feed-forward gate/up/down subset accounted for 9.625927680 GB. The restricted draft head already includes its mutable segment, so it is counted once.

This calculation prices a particular logical payload at a particular reference rate. Cache reuse, repeated reads, discarded drafting, context length, model format and the selected graph can change actual traffic. A rigorous lower bound would need both a minimum amount of traffic across the limiting interface and an upper bound on its sustained service rate. The best observed streaming result establishes neither for every future design.

Subtracting 27.65 ms from a measured 40 ms cycle therefore does not identify 12.35 ms of removable overhead. The remainder includes computation, state and cache traffic, and dependencies. Those costs can overlap. Fusion needs a specific read, write or scheduling boundary to remove, rather than a gap between two numbers measured at different scopes.

The distinction between GPU warm-up, sustained heat and cache residency is covered in Pulsar's operating-condition measurements.

Projection shape changes the achieved rate

The streaming reference does not predict every matrix operation. An eight-row projection sweep reported the following best payload rates:

The spread makes shape-specific controls important. A deeper K split reduced a dependent-call FFN-down probe from 118.8 to 99.7 µs, but changed results by about one BF16 unit in the last place. The short-K output projection instead moved from 36.7 to 37.3 µs, and independent-call timings were approximately unchanged. The numerical difference matters for byte-identical paths; the benefit also depends on the projection shape and call ordering. Projection and control receipts

Results from the persistent prototype

The measurements cover three levels: a captured kernel window, isolated projection variants and a paired serving comparison.

Small gaps between dispatches

In one captured FFN-to-GDN window, the separate-kernel path took about 521.2 microseconds. Approximately 7.7 microseconds lay between its kernels.

That is about 1.5% of that window. It does not mean all 7.7 microseconds can be removed, and it is not a full serving-cycle measurement. It establishes that the visible gaps were a small part of this particular region.

The producer's final work and the consumer's initial work also take time inside their kernel intervals. That work remains even when the dispatch boundary disappears.

Projection geometry and the persistent schedule

Different projections needed different amounts and shapes of parallel work. One persistent worker arrangement did not suit all of them.

Keeping the input projection as a native kernel recovered roughly half a millisecond in a hybrid experiment. The hybrid still did not beat the separate-kernel baseline.

An improved persistent prototype was then compared with both its predecessor and stock execution:

This was a September 27 screen on Qwen3.8-27B, with three sampled seeds at each of 1K, 5K and 10K input lengths. The ratios were geometric means over nine paired comparisons; output hashes and useful-token counts matched. Each request had representative GPU warm-up. The metric was decode output tokens divided by the engine's recorded decode wall time, not latency to a completed answer.

Scheduling and synchronization costs

Instrumentation exposed costs hidden by the dispatch count. In one worker-timeline experiment, the separate chain took 407.6 µs, the persistent kernel 428.0 µs, and the instrumented persistent kernel 450.7 µs. The software clock itself added 5.3%, so its worker timings describe the instrumented run.

These diagnostics cover different variants and cannot be added into a single explanation of the serving regression. They show why removing a dispatch is insufficient: work distribution, readiness signalling and normalization can cost as much as the boundary they replace. The worker clock also perturbs execution, and average worker idle time can overlap useful work elsewhere. Diagnostic receipts and scope

Threadgroup staging

Two further variants loaded Q4 down-projection weights through threadgroup storage.

Across 24 alternating pairs, the one-slot version added about 20.714 microseconds per projection, and the two-slot lookahead version added about 22.670 microseconds. Neither won a pair.

These operator measurements used real weights and synthetic activations. Both tested implementations lost; the result does not generalize to every staging design or isolate occupancy as the cause.

The extra copies, synchronization and resource use must be repaid by the reuse they enable.

Register pressure remained unproven

The megakernel used more registers in the driver metadata, so register pressure was an attractive explanation.

A later control tried to match the extra loads and arithmetic while changing value lifetimes. Direct paired medians were +0.043 and -0.044 ms/step, without a repeatable penalty, and the inspected kernels reported no register spills. The control was imperfect, but it was enough to withdraw the earlier numerical attribution.

Saved frame-level counters reported approximately 15.5% and 18.0% occupancy, while active-core counters reported approximately 92.0% and 95.3%. These measure different things; low occupancy alone did not establish idle cores. Neither observation established the cause of the loss.

Packing and geometry in the unfused control

Lithos's published GDN study provides a useful independent example:

The study attributes substantial improvement to operand packing and projection geometry. Fusion did not establish an advantage over the corresponding packed controls. These were fixed eight-row layer measurements, not generation throughput, and numerical equivalence to the original model was evaluated with tolerances. Author's study

Choosing a fusion boundary

A useful candidate removes identifiable work: an intermediate read/write, redundant transformation, repeated weight load or measured scheduling gap. Existing primitives in the runtime, Metal Performance Shaders (MPS), Metal Performance Primitives (MPP) or MLX may already cover it. When a new implementation needs better packing or tile geometry, the separate-kernel control needs the same improvement for the comparison to isolate fusion.

The numerical requirement also determines which changes are available. Byte equality, a documented tolerance, target-distribution preservation and task quality require different checks. A narrow candidate with explicit supported shapes and failure behavior is easier to validate against that requirement, including a case that breaks its central assumption. Operator measurements then feed into graph, stateful-model and serving tests before the implementation expands.

Fusion changes the cost of a target pass. Speculative decoding changes the useful progress obtained from it; Pulsar's engine measurements cover both in the complete serving path.

Further reading

• Apple: performing calculations on a GPU

• Apple: threadgroup and grid sizes

• Apple: resource synchronization

• Apple: resource storage modes

• Metal Shading Language specification

• Metal Performance Primitives programming guide

Scheduling and correctness

Inside the persistent kernel, a compiler or hand-written scheduler describes which tasks exist, where their inputs live, and when they can run.

Conceptually:

while there is work in this region:
    obtain a task whose prerequisites are satisfied
    compute the task's output
    make the output visible to its consumers
    publish completion

This scheduling sketch leaves the synchronization unresolved. A Metal implementation needs a safe way to detect completed prerequisites and make their writes visible to consumers.

There are several reasonable fusion boundaries:

Larger regions offer more reuse, but also bring more varied work under one schedule.

For example, a layer's multilayer perceptron (MLP) might already have a very effective matrix kernel. Pulling it into a persistent worker layout can make that matrix operation slower even if the surrounding dispatch count falls.

An indirect command buffer, or another form of pre-encoded replay, addresses a different problem. It can reduce repeated host encoding work while preserving multiple GPU dispatches. An engine can use both command replay and mixer megakernels.

Dependencies and synchronization

After a down projection produces a hidden vector, root-mean-square (RMS) normalization needs the sum of squares of all its elements. A worker that finishes the first quarter must still wait for the remaining quarters before the final normalization factor is available.

Fusion can change where the partial sums live and how workers signal completion. It cannot generally normalize using an unfinished sum and remain equivalent.

The same principle applies to recurrence. If a state update is:

the next token's update depends on the previous state. Parallelizing independent state columns may help. Simply launching all token updates simultaneously does not preserve the recurrence.

Worker queues, completion flags and software barriers add coordination costs. A worker arrangement that suits one projection may undersupply another. A threadgroup barrier is not a whole-grid barrier, and a grid whose resident workers spin for unscheduled producers can fail to make progress.

The implementation must specify which worker owns each value, when writes become visible, and what happens on timeout or cancellation. Buffers cannot be released while submitted work can still read them. The next consumer must see the logically committed state, even if materializing that state was deferred.