Run it locally
Per-quant memory needs and a static "can you run it?" reference - no rig entry required
Q2_K
21.5GB
21.5GB min
25.0GB rec
Smallest footprint, noticeable quality loss
Q3_K_M
30.0GB
30.0GB min
34.0GB rec
Compact, moderate quality loss
Q4_K_M
42.0GB
42.0GB min
48.0GB rec
Balanced - the usual local sweet spot
Q8_0
74.0GB
74.0GB min
80.0GB rec
Near-lossless
The reference hardware
22 reference configs, drawn in-house. Scroll for more.
Can you run it? - reference rigs
| Rig |
Q2_K |
Q3_K_M |
Q4_K_M |
Q8_0 |
| NVIDIA Jetson Orin NX 16GB |
no -> cloud
|
no -> cloud
|
no -> cloud
|
no -> cloud
|
| Single GTX 1080 Ti (11GB) |
no -> cloud
|
no -> cloud
|
no -> cloud
|
no -> cloud
|
| Single RTX 4090 (24GB) |
tight
|
offload
|
no -> cloud
|
no -> cloud
|
| 4x H100 80GB (320GB) |
fast 342.7t/s
|
fast 245.6t/s
|
fast 175.5t/s
|
fast 99.6t/s
|
| NVIDIA DGX Station 748GB |
fast 204.6t/s
|
fast 146.6t/s
|
fast 104.8t/s
|
fast 59.5t/s
|
| 8x RTX 3090 rack (192GB) |
fast 191.5t/s
|
fast 137.3t/s
|
fast 98.1t/s
|
fast 55.7t/s
|
| 4x RTX 5090 (128GB) |
fast 183.3t/s
|
fast 131.4t/s
|
fast 93.9t/s
|
fast 53.3t/s
|
| AMD Instinct MI300X (192GB) |
fast 136.2t/s
|
fast 97.6t/s
|
fast 69.7t/s
|
fast 39.6t/s
|
| 4x RTX 4090 (96GB) |
fast 103.1t/s
|
fast 73.9t/s
|
fast 52.8t/s
|
fast 30.0t/s
|
| 2x RTX 5090 (64GB) |
fast 91.7t/s
|
fast 65.7t/s
|
fast 46.9t/s
|
offload
|
| 2x RTX 3090 (48GB) |
fast 47.9t/s
|
fast 34.3t/s
|
fast 24.5t/s
|
offload
|
| Single RTX 5090 (32GB) |
fast 45.8t/s
|
tight
|
offload
|
no -> cloud
|
| RTX PRO 6000 Blackwell (96GB) |
fast 45.8t/s
|
fast 32.9t/s
|
fast 23.5t/s
|
ok 13.3t/s
|
| Mac Studio M4 Ultra 192GB |
fast 30.5t/s
|
fast 21.8t/s
|
ok 15.6t/s
|
ok 8.9t/s
|
| Mac Studio M4 Ultra 512GB |
fast 30.5t/s
|
fast 21.8t/s
|
ok 15.6t/s
|
ok 8.9t/s
|
| MacBook Pro M5 Max 128GB |
ok 17.1t/s
|
ok 12.3t/s
|
ok 8.8t/s
|
slow 5.0t/s
|
| Dual EPYC 9004 + 768GB DDR5-4800 |
ok 11.8t/s
|
ok 8.5t/s
|
slow 6.0t/s
|
slow 3.4t/s
|
| DGX Spark 128GB unified |
slow 7.0t/s
|
slow 5.0t/s
|
slow 3.6t/s
|
slow 2.0t/s
|
| Ryzen AI Max+ 395 128GB |
slow 6.6t/s
|
slow 4.7t/s
|
slow 3.4t/s
|
slow 1.9t/s
|
| Jetson AGX Orin 64GB |
slow 5.2t/s
|
slow 3.8t/s
|
slow 2.7t/s
|
no -> cloud
|
| Epyc + 512GB DDR4-3200 + 2x RTX 3090 |
slow 5.2t/s
|
slow 3.8t/s
|
slow 2.7t/s
|
slow 1.5t/s
|
| Epyc + 512GB DDR4-2400 + 2x RTX 3090 |
slow 3.9t/s
|
slow 2.8t/s
|
slow 2.0t/s
|
slow 1.1t/s
|
Fit tiers use the same will-it-run logic as the rig finder. For comfortable fits, the badge reflects decode speed: fast >=20 t/s, ok 8-20 t/s, slow <8 t/s. t/s is a bandwidth estimate, not a measured benchmark.
Or run it in the cloud
Live per-provider pricing, throughput and uptime - refreshed about 19 hours ago via OpenRouter. Click a column to sort.
some pricing may be stale - last verified 2026-10-24
|
Provider
|
Type
|
Input $/M
|
Output $/M
|
Cache $/M
|
Tok/s
|
Latency
|
Uptime
|
Value
|
|
DeepInfra
|
API
|
0.10 |
0.32 |
- |
- |
- |
100.00% |
best uptime
|
|
Parasail
|
API
|
0.22 |
0.50 |
0.110 |
- |
- |
100.00% |
|
|
Cloudflare
|
API
|
0.29 |
2.25 |
- |
- |
- |
100.00% |
|
|
|
API
|
0.59 |
0.79 |
0.295 |
- |
- |
100.00% |
|
|
CoreWeave
|
API
|
0.71 |
0.71 |
0.710 |
- |
- |
100.00% |
|
|
Google
|
API
|
0.72 |
0.72 |
- |
- |
- |
- |
|
|
|
API
|
0.88 |
0.88 |
- |
- |
- |
- |
|
|
|
API
|
0.90 |
0.90 |
- |
- |
- |
- |
|
|
AkashML
|
API
|
0.20 |
0.52 |
0.100 |
- |
- |
96.67% |
|
|
Novita
|
API
|
0.14 |
0.40 |
- |
- |
- |
96.30% |
|
|
SambaNova
avoid
|
API
|
0.45 |
0.90 |
- |
- |
- |
85.19% |
|
Default order: throughput among 95%+ uptime providers, then latency; subscriptions last. Sort by any column. Subscription rows show $/mo in the Value column - per-token columns are "-". Affiliate links are marked sponsored / nofollow. Confirm current pricing on the provider's site before committing.
PRICE HISTORY
Inference cost over time
Data accumulates from the first daily sync - longer ranges populate over time. Prices come from OpenRouter snapshots, not a historical API.
Loading price history...