Faster MiniMax-H3 with Sage and Sparse Attention

MiniMax-H3 generates video and audio together, and due to its large model size it is the most attention-heavy model in our lineup. Attention is well known to be the most expensive part of inference: attention grows with the square of the token count.

In this post we follow two changes to the attention of MiniMax-H3. Warm measurements on RTX 4080:

Quantizing the kernel with SageAttention

MiniMax-H3 has 1344×768 as its official 16:9 canvas. At that shape, cuDNN flash attention is close to the FP16 Tensor Core roof. SageAttention takes a different route: it keeps attention dense, but moves QKᵀ to INT8 and P·V to FP8. The lower-precision Tensor Core paths have a much higher throughput roof on Ada.

For Nox we implemented SageAttention2 as a native CUDA kernel. The port is faithful: its output is bitwise identical to upstream, and it matches upstream’s speed to within 0.2% at this shape. The attention output stays within about 3.6% relative L2 against cuDNN with a cosine similarity of 0.9994.

The complete attention call falls from 476.9ms to 167.1ms, 2.85× faster, and a denoise step falls from 32.0s to 16.1s.

As we’ve seen in the benchmark post, floating point addition is not associative, leading to small differences. In this comparison the composition, detail and motion remain the same take; the two outputs are not byte-identical.

Doing less work with sparse attention

We’re still at the hardware roof (although now a different one with SageAttention). The next lever is to compute fewer attention scores. With SageAttention, attention is still about 7.7s of each 16.1s denoise step, so it remains the largest cost.

Veda showed that H3 does not need to compute its full attention map. A lot of the attention calculation is actually redundant as we will show below with examples. Attention is a grid of tiles. A small part of that grid carries most of the result. Veda trains a small distilled predictor to say in advance which tiles matter, computes roughly the top 10% and skips the rest (paper).

That result started our own research path: implement sparse attention in Nox’s native kernel, then see how far it can be optimized.

What dense attention computes

The figure below draws one attention call from a real render. Green is the source query tile, gold is every video key tile it attends to, and the strip underneath shows how much of each grid frame is covered. This is rather boring for dense attention as every tile is used for the calculation.

Raster tiles: the first sparse router

Our first router is simple: block-level top-k over pooled queries and keys is the standard selection rule (SpargeAttn, XAttention). In video it appears inside trainable routers (VSA, VMOBA) and as fixed block patterns (SVG). We run that in natural token order: compute the mean of Q and K in every tile for every head, and score each query tile against each key tile with a dot product. We only keep the top 15%.

Text, conditioning and audio keys are always kept in addition to the two key tiles that cover the query tile’s own tokens. The first denoising step and the first transformer block remain fully dense (especially important for T2V to get the composition similar to dense).

That token order is why it draws horizontal stripes in the example above: a row has 42 tokens, so a 64-token key tile is one full row plus 22 tokens of the next, and a 128-token query tile is just over three rows.

What routing buys

The kernel scales almost perfectly with the budget: at 15% of the key tiles it takes 15.4% of the dense time. Routing itself costs 3.4ms, 2.2% of a dense call. On a captured 38,304-token H3 problem, the complete call is 4.8× faster. Over a complete request, the warm time falls from 84.3s to 66.7s, a saving of 17.6s (21%), and denoising falls from 64.5s to 46.8s (27%).

What routing costs

The SSIM is not nearly as high as with the attention swap, because the result is a different sample. The overall video survives, but high-frequency details can change. We’ll investigate how much detail exactly in the following sections.

Why raster hits a ceiling

We measure attention mass to optimize what a route keeps. This mass is calculated by softmax over the QK scores, summing the weights across all keys to 1. Each token has a budget of 1 to spread over the keys. Usually a few high-scoring keys take most of it.

That gives us a score for any route without rendering anything: we can compare what part of the dense weights a route keeps by comparing the weights of the route, averaged over the query tiles. A route that keeps 1.0 of the mass has kept every key that dense attention gave weight to. A low value means the route skipped keys that dense attention relied on.

To put the current raster numbers in perspective, we first calculate the floor: with the always-kept tiles and picking the same number of tiles at random. For the desert I2V shot below, this would keep 0.275, a low score. The raster routing at 15% keeps 0.751 of the mass, meaning the simple pooled score is doing real work already. However, the raster routing has a limit. The score between two tiles is the average of every Q·K score between them, and a stripe crosses the whole frame. It can average various features of the video into one vector and dilute a small strong match inside it, which is exactly the kind of match that softmax rewards.

Even if we could calculate the perfect scoring, the raster route wouldn’t be able to pick the best tiles: with perfect knowledge we can only keep 0.818 of the dense attention mass.

Nox sparse attention: choosing what the router keeps

We don’t have to keep the tokens in the raster order: changing them to a different token order can be beneficial. Following earlier research, using a 3D window shape is a well-known improvement, so we opted to do the same here. Each head permutes its video tokens so that consecutive tokens form a 3D block:

Reordering costs nothing measurable: on the orbital shot from the Nox table above, the warm request takes 66.7s, the same as raster tiles.

Attention mass also decides the layouts. The geometry searches we compare against pick their per-head shape by measuring output error: STA searches a window size, Veda a tile factorization, SVG profiles whole patterns, and each needs an attention forward pass per candidate. We rank candidate layouts by how much dense attention mass their 15% route keeps instead, which needs only the captured Q and K. The result is a static per-head layout table plus dynamic top-k over mean-pooled queries and keys, with no predictor, no clustering, no online profiling and no extra weights.

The reordering raises the attention mass on the leviathan shot from 0.751 to 0.803. A router with perfect knowledge of the mass would reach 0.867 on the same tiles, so better scoring still has room.

At 15%, the output lands closer to dense on both shots compared here: SSIM rises from 0.567 to 0.668 on orbital and from 0.673 to 0.693 on arctic.

Veda attention

Veda attention uses the same 3D tile shape setup, but where it differs is how it picks the tiles to match. Veda’s predictor is a 275 MB file of learned projections, one for each block and head. To make a fair comparison between the two, we implemented Veda’s router in Nox and verified it against a golden export from Veda’s own reference code, which it reproduces exactly: same tile order, same routes. Below you can compare which tiles Veda picks vs our Nox sparse attention:

Nox sparse attention uses no trained weights, so the obvious question is how much that costs against the router that does.

One caution applies to everything below. The only predictor available is a preview trained on 8-step T2V, and these renders are 4-step FL2VA I2V. Veda is outside its training setup here, and a predictor trained for this setup may do better.

Nox sparse attention against Veda

This is where SSIM stops being useful. A different sample is not a broken one. So how can we measure what’s actually lost? We’ve chosen to use the Laplacian loss. This measures how much fine detail a render lost, without needing that detail to land on the same pixels as in the dense render.

The Laplacian is a small filter that compares each pixel with its four direct neighbors. In a flat area the result is close to zero, while edges and fine texture produce large positive and negative values. The variance of the filtered frame therefore summarizes how much fine detail the frame holds. We compute this variance on the luma of each frame and average it over all 124 frames, which gives one number per render. The loss is the relative drop against dense:

Laplacian loss = 1 − (variance of the render / variance of the dense render)

This number complements SSIM rather than replacing it. The measure won’t move much if a texture moved, but it also cannot tell whether the detail is the right detail or merely noise and artifacts.

Each render is scored against the Nox dense render of the same seed:

The additional predictor weight loading and execution costs about ~1.5s over the request before it (simulating what an actual warm request would do on an RTX 4080).

Attention mass on the same tensors

SSIM only compares final frames, so we also ran all the routers on the same captured Q and K tensors from a nearly dense Nox render, which asks one question: how much of dense attention’s softmax mass does each route keep?

At a similar kernel budget, 5,760 token slots for us and 5,632 for Veda, the two are level in the middle blocks and close on all captured blocks. The budgets are not quite equal in real tokens, because Veda pads tiles that run past the grid and those slots hold nothing.

Plain mean pooling on Veda’s tiles, with no predictor, reaches 0.815 at 15.1% of the tokens in the middle blocks. On these tensors the trained predictor adds little over mean pooling, although it is outside its training setup here.

Is 15% the right budget?

We swept Nox sparse attention over five budgets on the same three shots. SSIM says how far a render moved, so we also measured how much fine detail it lost against its dense render: the Laplacian variance of the luma.

Laplacian loss is the mean across the three shots. Attention mass is from the leviathan shot, in the middle half of the blocks.

Detail loss shrinks steadily as the budget grows, so routing trades fine detail for speed gradually rather than falling off a cliff. The dense first step & block do not shrink with the budget, so going from 15% to 5% saves only another 2.1s, while the Laplacian loss grows from 13.6% to 19.5%.

We picked 15% as the default budget when sparse attention is enabled because below it the quality drop is slightly steeper, and it keeps most of the saving. We have not shown that it is optimal: this is three shots at one seed each, and closeness to dense is not the same as quality. Beyond that, the budget is a tradeoff between time and quality for the user to choose.

How we measured

Most of the setup is described in the benchmark post: the same checkpoint and lora setup are used, only at a different resolution and framecount.

To build the figures, we saved Q and K for each head immediately before the Sage call at full density, then reproduced the pooling and top-k selection offline and mapped every key tile back to its frame, row and column. The routes are therefore recomputed outside the kernel, and each grid frame is drawn over the closest final video frame.

Attention mass uses the same captures, at 12 sampled blocks over the three routed steps. Each call averages the kept dense softmax mass over 48 sampled video query tiles for each head and takes the median head, and the reported figure is the mean over calls. The middle half is blocks 12 to 33, 18 calls. The random figure is the expected mass when the non-forced key tiles are chosen uniformly, with the forced text, conditioning, audio and diagonal tiles kept as in the router, and a simulation with 64 draws per tile and head agreed to within 0.001.

In conclusion

SageAttention has high SSIM numbers and the complete request 1.75× faster. Sparse attention became an opt-in feature for us, where the user can decide how sparse. With the default setting it can cut a further 21% from a complete request at no extra memory, reaching 2.2× against cuDNN.