AI Infrastructure

Modern AI infrastructure is evolving into “AI token factories” where the primary goal is maximizing processing speed and minimizing cost per token.

The Cost per Token Metric

Traditional metrics like “cost per GPU hour” or “FLOPS per dollar” focus on raw compute power. Cost per Token accounts for real-world performance, software optimization, and hardware utilization.

Hardware Comparison (Blackwell vs. Hopper)

NVIDIA’s Blackwell architecture delivers a 35x lower cost per million tokens than the Hopper generation when running DeepSeek-R1, despite a higher hourly compute cost.

Full-Stack Optimization

Lowest token costs are achieved by co-designing:

• Hardware: Next-gen GPUs (Blackwell).
• Networking: High-bandwidth interconnects.
• Software: Frameworks like TensorRT-LLM and vLLM.

Strategic Insights

• The “Inference Iceberg”: True value lies beneath the surface in factors like token output per megawatt and software stack optimizations NVIDIA.
• Increasing Value Over Time: Open-source inference software updates can increase token output on existing hardware long after purchase.