NVFP4 on a 5090 names two different machines

I quantized Qwen3-8B to NVFP4 and served it on an RTX 5090 twice. Same checkpoint, same card, same eval harness. The first run used native FP4 compute:

(EngineCore pid=3713015) INFO 08-07 13:40:14 [__init__.py:978] Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM

The second run forced the Marlin weight-only fallback, and vLLM warned me about it:

(EngineCore pid=3713665) WARNING 08-07 13:43:31 [marlin.py:34] Your GPU does not have native support for FP4 computation but FP4 quantization is being used. Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads.

Both of these runs are, by the label on the tin, “NVFP4 on a 5090.” They are different machines. One computes matrix multiplies in FP4 on Blackwell’s native hardware; the other dequantizes FP4 weights and runs the math at high precision through a kernel tuned for years. They produce different speeds and measurably different outputs. And note the warning text itself: it fired on a card that does have native FP4 support, and its performance prediction turned out backwards in my measurements. The only way to know which machine a published “NVFP4 on consumer Blackwell” number came from is the log line, and most write-ups do not quote it.

This post has three parts: what 4-bit quantization actually costs on this card, measured by a harness built to catch damage that aggregate metrics hide; what it took to get native NVFP4 to land on sm_120 at all; and what the two-kernel comparison says about trusting labels. The repo is sm120-quant-bench; everything reproduces from one script.

Prior art, engaged

Consumer Blackwell is not unmapped. A January 2026 study (arXiv:2601.09527) benchmarks NVFP4, W4A16, and MXFP4 across three consumer cards including the 5090, with confidence intervals, energy telemetry, and a released harness. It is careful work and its economic conclusion, that self-hosted inference on consumer cards undercuts API pricing dramatically at moderate volume, deserves its audience.

Two questions stay open. First, kernel truthfulness. vllm#47749 documents a ModelOpt NVFP4 checkpoint on sm_120 silently routing to the Marlin fallback (vLLM 0.24, July 2026). Any consumer-Blackwell NVFP4 number published without its serving log leaves the reader unable to say which of the two machines above produced it. That is a question about reporting norms, not anyone’s integrity, and it applies to my own runs too, which is why every result in this repo carries its kernel-selection stderr as first-class data. Second, aggregate quality claims. The study reports suite-level deltas (“2 to 4% quality loss”). The rest of this post tests whether that framing can hide anything. It can.

A benchmark is a claim, and claims need provenance

Two ways a clean-looking quantization table lies. It measured a different system than it named: a kernel fell back, a dtype coerced, a template mismatched. Or it averaged away the damage: a sixty-task mean dilutes an eight-point fall in one capability below noise, and perplexity, the README standard, is blinder still.

I have been burned by the first class before, in another domain: a speech evaluation judge went stale in production and quietly inflated scores until it nearly inverted a licensing decision (post-mortem). A stale judge and a mislabeled kernel are the same failure. The name in the table stops matching the thing measured.

So the harness does the boring things. Test splits are carved once, before any quantization, with content hashes in a manifest, and the freeze script refuses to run twice. The calibration pool is disjoint from every eval split by construction. Capabilities are probed separately: math, code, knowledge split by domain, numeric fidelity, long-context retrieval. Deltas carry bootstrap 95% CIs, so a delta whose interval spans zero cannot be narrated as signal. And every run logs which kernel actually served it.

Environment, because sm_120 results without an environment line are folklore: one RTX 5090 32 GB (consumer Blackwell, sm_120), driver 580.173.02, torch 2.11.0+cu130, vLLM 0.26.0, flashinfer 0.6.14, transformers 5.10.1, llm-compressor 0.12.1.dev100+g170a2e19, Ubuntu 24.04.

What 4-bit costs, measured honestly

One model: Qwen3-8B, pinned revision, BF16 baseline. Two W4A16 algorithms (GPTQ, AWQ) and one NVFP4 checkpoint served two ways. All calibrated on the same 128 to 512 disjoint chat-templated samples, seed 3407.

Finding 1: perplexity ranks the formats backwards. NVFP4 posts the best perplexity of any quantization here, 18.28 against 19.66 and 19.73 for the W4A16 pair, a third of their distance to baseline. By the README-standard metric it is clearly the highest-quality 4-bit option. The capability probes disagree: NVFP4 has the worst GSM8K (86.6), the worst humanities score, and the worst long-context retrieval of every variant tested. Meanwhile GPTQ and AWQ sit 0.06 perplexity apart, statistically identical, while the numeric fidelity probe separates them by 6.7 points and drops AWQ 8.7 under baseline with intervals that barely graze. The aggregate does not merely fail to discriminate. On this table it actively misleads, twice, in both directions.

Finding 2: damage pools where digits live. Numeric fidelity and GSM8K degrade across every format; humanities is nearly flat, with GPTQ statistically indistinguishable from baseline there. A suite mean would have blended all of this into a true and useless “2 to 4% loss.” True, and no help to anyone deciding whether their invoice model still does arithmetic.

Finding 3: the napkin math holds. W4A16 reads 2.69 times fewer bytes per token (16.4 GB to 6.1 GB) and decodes 2.36 times faster (11.1 to 4.7 ms), 88% of the bandwidth-predicted ideal. Decoding is a memory problem. Here is the receipt.

Honest nulls. HumanEval deltas sit inside their wide intervals (164 problems) in both directions, including NVFP4-native’s apparent 8-point edge over NVFP4-Marlin; I will not narrate noise as signal in a post whose thesis is that people narrate noise as signal. And look at the v1 long-context row: a probe that hands every format 100 is not reporting quality, it is reporting its own saturation (the lone 99.5, on the Marlin path, was the first crack in the streak). A row of identical perfect scores is what a broken probe looks like, and it sat in my table claiming everything was fine. Version 2 plants three needles per document, adds near-miss distractor keys (one digit changed, different value), and never queries the last-inserted needle. That de-saturated it and exposed depth effects v1 could not see: every format holds 100 at 4k and sheds accuracy at 8k and 16k, with NVFP4-native worst at depth.

What it took to get native NVFP4 to land on sm_120

The quantization step failed first with this, surviving both pipeline="basic" and full CPU weight offload:

File ".../transformers/masking_utils.py", line 52, in and_mask
    result = result & mask(batch_idx, head_idx, q_idx, kv_idx).to(result.device)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a
total capacity of 31.33 GiB of which 8.74 GiB is free.

Weight offload not moving an OOM is the diagnostic: the allocation is activations, not weights. The longest calibration sample ran 3,994 tokens and 10 of 128 exceeded 2,048; passing max_seq_length=2048 to oneshot() fixed it. I have not resolved the exact broadcast shape (calibration batch size is 1), so I will not overclaim the mechanism; it is filed upstream as llm-compressor#3011. A 288 GB B300 would never see this wall. The largest consumer card NVIDIA sells hits it on the default recipe with an 8B model.

The working recipe pins llm-compressor git main (0.12.1.dev100+g170a2e19, which also bumps compressed-tensors to 0.17.2a20260806). The released 0.12.0.1 initially failed for me, but a clean-environment retest could not reproduce that failure, so the honest statement is “the pinned commit works,” not “the release is broken.”

Serving natively JIT-compiles flashinfer’s fp4_gemm_cutlass_sm120, which needed three one-line fixes on a conda toolchain: curand headers onto nvcc’s include path (they ship in site-packages/nvidia/cu13/include), MAX_JOBS=4 because 16-way ninja got OOM-killed at roughly 8 GB RSS per CUTLASS translation unit (FAILED: [code=137]), and LIBRARY_PATH=/usr/lib/x86_64-linux-gnu for conda’s ld failing to find -lcuda. One more flag pair mattered for honest numbers: full fp4_gemm autotuning tunes every capture-size by layer-shape combination at about 1.5 minutes each and was still running at 3.5 hours, so quality evals ran with VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS=fp4_gemm, and the speed table is explicitly default-tactic.

The two machines, compared

With both paths running, the label question gets an empirical answer. At these settings, on this card, the forced Marlin fallback is faster than native FP4 single-stream: 4.6 against 6.6 ms per token, with the W4A16 pair at 4.7 confirming it is the same well-worn weight-only path doing the same work. The warning that fires on the Marlin route predicts degraded performance; measured, the fallback won. Years of Marlin kernel optimization against a freshly JIT-compiled sm_120 CUTLASS path is not a fair fight yet, and the autotuned rematch is the follow-up post.

Quality between the two paths is parity or better for Marlin except one wide-interval HumanEval gap (57.3 against 65.2, intervals overlapping). The silent-fallback quality horror story that vllm#47749 made me fear did not reproduce here.

So the sharp conclusion is not “the fallback is a scam.” It is that the label tells you nothing. “NVFP4 on a 5090” named two machines in my lab that differed by 43% in decode speed and disagreed on benchmark scores, and only one log line distinguishes them. Any number without that line is unfalsifiable.

If you quantize for production

Check your stderr before your slides: one INFO line tells you which machine you measured, and on sm_120 today the difference is large. Gate on the capabilities your users exercise, with intervals, against a frozen split; on this table perplexity would have ranked your options backwards. And at single-stream default tactics on this card, the pragmatic serving choice for this NVFP4 checkpoint is currently the path the warning message tells you to fear.

Scope, confessed

One model, one card, PTQ only, single-stream speed at default tactics. The autotuned fp4_gemm rematch, a batch sweep, and more model families are follow-ups, not footnotes pretending otherwise. Everything reproduces from ./reproduce.sh; if your card disagrees, file an issue, because that disagreement is data.