Per-quant memory needs and a static "can you run it?" reference - no rig entry required
Q2_K
240.0GB
240.0GB min
260.0GB rec
Smallest footprint, noticeable quality loss
Q3_K_M
365.0GB
365.0GB min
400.0GB rec
Compact, moderate quality loss
Q4_K_M
410.0GB
410.0GB min
450.0GB rec
Balanced - the usual local sweet spot
Q5_K_M
510.0GB
510.0GB min
560.0GB rec
High quality, larger than Q4
BF16
1488.0GB
1488.0GB min
1600.0GB rec
Full quality, largest
Fit tiers use the same will-it-run logic as the rig finder. For comfortable fits, the badge reflects decode speed: fast >=20 t/s, ok 8-20 t/s, slow <8 t/s. t/s is a bandwidth estimate, not a measured benchmark.
How can a 24GB GPU run a 744B model? It does not load the model into VRAM. The quantized weights (e.g. ~410GB at Q4) sit in system RAM; the GPU only holds the small shared attention and router tensors and accelerates prompt processing. Because GLM 5.x is a Mixture-of-Experts model, each token activates only ~40B of its 744B params, so llama.cpp streams just those active experts from system RAM to the GPU each token (the -cmoe offload path).
That makes decode speed bound by system-RAM bandwidth, not GPU bandwidth - single digits on DDR4, which is why these rigs show 3-8 t/s even though they “fit.” A bigger GPU (e.g. 2x 3090) keeps more experts resident on-card and raises tok/s; a smaller GPU still runs it but pays the bandwidth tax. A 744B dense model could not run this way - only MoE’s small-active-params trick makes it possible.
Aggressive quants (1-2 bit) trade accuracy for size - roughly 17% accuracy loss at 2-bit vs full precision, and real long-context work often needs Q5 or Q6 even when lower quants “fit.”
Formula estimates here are conservative; real tuned setups can exceed them (one HN user reports ~6 tok/s on a 512GB DDR4 + 2x 3090 rig).