rituraj

Five numbers that change your LLM self-hosting cost estimate

8 min read
#llm #infrastructure #production-ml #devops #cost

DeepSeek V4 Pro needs 966 GB of VRAM. Most self-hosting cost estimates say 24. The gap is one wrong variable, and four more like it sit between any napkin calculation and the GPU invoice that lands on your desk.

The LLM Deploy Cost Calculator gets all five right, calibrated against real benchmark runs on A100 hardware. Every number is reproducible via the "Show calculation formulas" panel.

The five traps:

  1. VRAM scales with total MoE parameters, not active. Get this wrong and a 24 GB estimate becomes 800 GB.
  2. KV cache and weight quantization are independent. A Q4 model can still need an FP16 cache twice its own size.
  3. Fitting in memory is necessary but not sufficient. Decode is bandwidth-bound, prefill is compute-bound, and the max of those plus VRAM drives GPU count.
  4. One GPU has no redundancy. Production HA doubles the bill the day you take it seriously.
  5. Tensor-parallel topology has hard limits. An L40S has no NVLink. PCIe all-reduce throughput collapses past 4-way. Recommending 21× L40S for a 966 GB model looks cheaper on paper than 7× H200, but it is not a deployable configuration.

DeepSeek V4 Pro: 49B active, 1.6T in VRAM. Use the wrong one and you're off by 32×

MoE models load every expert at inference time. DeepSeek V4 Pro activates 49B parameters per token but stores 1.6T in weights. A calculator that uses active_params for VRAM will tell you it needs ~24 GB at Q4. The correct answer is 800 GB of weights. That's a 32× gap before you even add the KV cache and overhead.

With a 20% overhead buffer (CUDA context, HBM thermal headroom, allocator fragmentation) and a KV cache floor to handle sessions longer than the average, the total comes to about 966 GB. That's where the topology trap closes: an A100 80GB cluster would need 13 cards, but 13-way tensor-parallel over NVLink is not a standard topology. One DGX A100 node has 8. The calculator enforces a max_tp cap per GPU class (NVLink datacenter cards: 8, L40S PCIe: 4, consumer cards: 1) and excludes options that exceed it. The actual minimum deployable configuration is 7× H200 141GB on a single NVLink node, totalling 987 GB of capacity.

I benchmarked the MoE split on Qwen3-30B-A3B: 30.5B total, 3.3B active per token. On A100 80GB with vLLM BF16, 134 tok/s. A dense 7B on the same GPU: 92 tok/s. Decode throughput tracks active params, but VRAM loads all 30.5B at startup regardless of which experts fire. At Q4 that's 15 GB of weights, not 1.6 GB.

Your KV cache is twice your model. Quantizing weights won't fix it

Most calculators apply the same quantization factor to the KV cache as weights. Q4 weights are 4-bit integers. KV cache tensors run in FP16, FP8, or INT4 independently.

Qwen3-32B at 32K context with 4 concurrent: KV cache in FP16 is 34 GB. Q4 model weights are 16 GB. Using 0.5 bytes per KV parameter (because you quantized weights) would report 8.5 GB. The real answer is 34 GB.

There is a second subtlety: KV should be sized on the sessions that run, not your theoretical maximum. Sizing purely on average session length leaves you vulnerable to OOM when traffic includes sessions above the mean. The calculator uses max(avgTokens, contextLength ÷ 4) as a safety floor. If your average session is 2K tokens on a 32K context window, it sizes KV on 8K. This is what caught the Code Assistant preset: the KV floor pushed total VRAM from 22 GB to ~29.5 GB, correctly excluding the RTX 3090 (24 GB) and recommending an L40S instead.

Architecture changes the KV formula too. GQA stores 2 × layers × kv_heads × head_dim × bytes per token. DeepSeek V4 uses shared K=V vectors (both key and value read the same tensor). The vLLM team's V4 writeup reports ~9.6 GiB at 1M context, an 8.7× drop from V3.2.

A 32B model fits on an H100. It still can't serve 5,000 calls/day

Prefill is compute-bound: MFU × GPU_FP16_TFLOPS / (2 × active_params). Quantization doesn't help. Kernels dequantize and run FP16 math.

Decode is memory-bandwidth-bound: batch × HBM_GBW / (weights_bytes + batch × KV_per_seq). Quantization does help here. Fewer bytes to read from HBM means higher throughput.

The calculator takes the max of (gpus_for_prefill, gpus_for_decode, gpus_for_vram) and picks the cheapest GPU that satisfies all three. The throughput model is calibrated from 113 real benchmark runs, not assumed. On Qwen2.5-7B FP16 at c=1, measured throughput is 63% of theoretical bandwidth. Efficiency drops to 36% at c=64. A calculator assuming 100% overstates throughput by 1.6×.

Key insight: The five mistakes compound. Plug V4 Pro's active parameters (49B) into a standard VRAM formula and you get ~24 GB at Q4, one GPU. Correct for total parameters (1.6T): 800 GB of weights. Add KV cache and 20% overhead: 966 GB total. Apply the NVLink topology constraint: A100 80GB is excluded (needs TP=13, max is 8 per node). Minimum deployable config: 7× H200 141GB on a single DGX node. Add two replicas for HA and that's fourteen GPUs. The naive estimate is wrong by 32× on VRAM and roughly 40× on cost.

The break-even you care about is the next GPU

The break-even chart is a step function. Self-hosted cost is flat until you need another GPU. Then it jumps. API cost rises linearly. Try it. Change any input and the chart updates.

The flat regions are capacity headroom. Each vertical jump is where you don't have any. A workload near a step boundary should treat the next-tier GPU as the realistic plan since traffic spikes push you over.

The presets

Seven production-realistic scenarios. Specific dollar figures drift with GPU market pricing; open the live tool for current numbers. The direction and magnitude of each scenario are stable.

Customer Support Bot. Qwen3-14B Q4, 8K context, 8 concurrent, 2 replicas, reserved 1-year, peak 3×, 500 calls/day. At this volume API wins. Conversational SaaS rarely justifies the ops overhead of self-hosting until traffic climbs past a few thousand requests per day.

Code Assistant. Qwen3-32B Q4, 32K context, 4 concurrent, on-demand, 200 calls/day. The 32K context window pushes the KV safety floor to 8K tokens per session, which combined with 20% overhead brings total VRAM to ~29.5 GB, above the RTX 3090's 24 GB. An L40S (48 GB) is the minimum viable card. At 200 calls/day this is nearly break-even; self-hosting makes sense only if you're already committed to the infrastructure for other models.

Enterprise RAG. Qwen3-32B Q4, 64K context, avg 12K session, 8 concurrent, 2 replicas, on-demand, 1,000 calls/day. The 64K window and 12K average session push VRAM well past 48 GB. L40S is excluded and the recommendation steps up to A100 80GB. At this volume self-hosted is roughly break-even with managed APIs, which means the cost argument alone doesn't settle it. Data residency usually does.

Startup MVP. Gemma 3-27B Q4, 16K context, spot pricing, 100 calls/day. API wins by a wide margin at this volume. Use the managed API, revisit self-hosting if you cross 500 calls/day.

High-Volume API Replacement. Qwen3-30B-A3B MoE Q4, 8K context, 32 concurrent, 2 replicas, reserved 1-year, 20,000 calls/day. Self-hosting wins here. MoE pays off: 30.5B total but 3.3B active per token means decode throughput punches well above the GPU count. API cost at this volume runs 10-15× the self-hosted number.

Llama 4 Scout: high volume. 109B MoE, 16K context, 16 concurrent, 2 replicas, reserved 1-year, 10K calls/day. Active params are 17B so decode is fast, but the 109B total weight requires multi-GPU. Large-MoE economics at scale.

Llama 3.3 70B: enterprise chat. Dense 70B, 8K context, 8 concurrent, 2 replicas, reserved 1-year, 2K calls/day. The comparison point for teams evaluating whether a 70B dense model justifies self-hosting at enterprise chat volumes.

What this tool does not model

Real vLLM deployments run 10-30% below ideal batching throughput. The tool assumes ideal batching. Treat output as a floor, not a guarantee. Prefix caching reduces KV pressure on repeated prompts, and the tool assumes every call starts cold. Multi-token prediction (MTP) is not modeled but typical speedups are 1.6× to 2.2× (Qwen3.6 MTP lands 1.7-2.4× on llama.cpp; DeepSeek V3/V4 native MTP heads do ~1.8× on SGLang). Treat as headroom, not a reason to drop a GPU.

The max_tp cap covers one NVLink node. Models exceeding 8× H200 VRAM capacity (above ~1,128 GB) are shown as ineligible. Cross-node tensor-parallel over InfiniBand is not modeled. That is a multi-node distributed serving problem that deserves its own analysis. The tool also doesn't price your time: production self-hosted adds roughly 0.3 FTE for monitoring and incident response, $6K-$12K/month at market rates.

When the dollar figures don't settle it

Data residency. HIPAA, defense contracts, GDPR. The calculator stops being "should we self-host" and becomes "cheapest compliant path." The comparison isn't $1,357 vs $1,440. It's $1,357 vs "we cannot take this contract."

Consumer GPUs in production. RTX cards lack NVLink and ECC. NVIDIA's EULA forbids datacenter deployment. The calculator's max_tp field enforces this: consumer cards cap at 1 (single GPU only), so the model must fit entirely on one card. Single-GPU replicas behind a load balancer are still valid, just not tensor-parallel.

Provider tiers. Not all "$0.22/hr" is equivalent. The calculator tags every pricing entry as marketplace (Vast.ai: individual GPU owners, no SLA), dedicated (RunPod, Lambda: managed cloud), or hyperscaler (AWS, GCP: enterprise SLA). When the cheapest option is a marketplace GPU, a warning is shown alongside the cheapest dedicated-cloud alternative. Budget dev environments and production services call for different tiers even when the GPU model is the same.

API pricing. Self-hosted costs track hardware deflation. API costs track quarterly vendor decisions. The calculator gives you a snapshot, not a forecast.

Where to push back

If your number disagrees with the calculator, the math is the product. Open an issue with your inputs on GitHub. Pricing rows in src/data/constants.ts update monthly. Model rows ship as models ship. If you have benchmark data for a GPU not yet measured, a pull request against inference-bench is the right place.

References