NVIDIA H100: the Hopper workhorse, explained

What the H100 is

Hopper is NVIDIA's 2022 datacenter architecture, and the H100 is its flagship. Each SXM module pairs 80 GB of HBM3 at 3.35 TB/s with a fourth-generation Tensor Core whose Transformer Engine runs 8-bit floating point (FP8) natively. FP8 roughly doubled transformer throughput overnight, and that one change made Hopper the standard unit of AI compute for the next three years.

In 2026 the H100 is no longer the frontier part, which is exactly why it is the safe buy. Blackwell (B200, B300) sits above it for the largest models. The H100 sits at the center of the market: the part every cloud stocks, every framework targets, and every finance team already knows how to price. It ships as an SXM module in 8-GPU HGX and DGX nodes, with a lower-power PCIe card for standard servers.

The H100 remains the reference GPU because it sits at the cost-performance sweet spot for most real workloads, not the largest ones. The Transformer Engine and FP8 precision give it strong throughput per dollar on exactly the model shapes teams run in production: 7B to 100B-class dense and mid-size mixture-of-experts models, both for fine-tuning and for serving. Its 80 GB of HBM3 holds a large fraction of production models on a single GPU, and an 8-GPU node handles the rest through NVLink without leaving the box.

Two structural facts keep it the default. First, availability: Hopper shipped in enormous volume, so H100 capacity is broad and easier to secure than frontier Blackwell parts. Second, maturity: the entire training and inference stack (CUDA, cuDNN, TensorRT-LLM, vLLM, the major frameworks) is tuned for Hopper, so teams get predictable performance without chasing early-silicon issues. For any workload that fits comfortably in 80 GB per GPU, or a 640 GB node, the H100 is usually the rational choice, and buyers step up to the H200 or Blackwell only when memory capacity or model scale forces the move.

Most production inference is not frontier-scale, and this is where the H100 shines. A 7B model needs roughly 7 GB at FP8, and 8B to 70B models fit comfortably on a single 80 GB GPU with room for large batches and long context. In practice that means high-throughput, low-latency serving of the models most teams actually deploy, often several models or many concurrent sessions per GPU, without ever leaving one card.

The node-fit question only gets interesting for the largest open models. An 8-GPU HGX H100 node pools 8 x 80 GB into 640 GB of NVLink-connected HBM3. Today's frontier models are mixture-of-experts, and MoE keeps every expert resident in HBM even though only a slice fires per token, so total parameters, not the active count, set the memory floor. The H100 has no FP4 datapath, so FP8 (1 byte per parameter) is the lowest-precision weights floor here. Weights-only arithmetic shows where each model lands:

Weights-only figures at published parameter counts and FP8; production serving adds KV cache, activations and framework overhead on top. Only the smallest of today's frontier MoE models fits inside a single 640 GB H100 node, and anything larger shards across nodes over the cluster fabric. That multi-node tax on the biggest models is a large part of why buyers of frontier-scale workloads move to B200 or B300, where far more memory sits inside one node.

H100 specs

Official NVIDIA Hopper H100 SXM figures; the PCIe variant carries a lower 350 W TDP and lower clocks. Tensor Core FP8 and the Transformer Engine are the defining Hopper features. Real inference throughput and cost per token depend on the model and serving stack, not raw FLOPS alone.

H100 form factors: SXM, PCIe, and NVL

The H100 ships in three form factors that share the Hopper GPU but differ on power, interconnect, and even memory. The form factor decides where the card deploys and what it is good at, so it is worth matching to the workload rather than defaulting to whatever a provider lists.

SXM5 is the flagship: 80 GB, up to 700 W, and 900 GB/s NVLink, mounted eight to a board in HGX and DGX systems. It is what you want for training and any job that spans multiple GPUs. PCIe is the same 80 GB Hopper at 350 W with lower bandwidth and a standard slot, ideal for mainstream inference and mixed servers where a full HGX node is overkill. H100 NVL is a two-card pair bridged by NVLink, and unusually it carries 94 GB per GPU at higher bandwidth, tuned specifically for serving large language models on a compact footprint.

H100 vs A100

The A100 is the previous Ampere generation. The H100 is a full generational step: more memory bandwidth, roughly a fourfold jump in transformer throughput from FP8 and the Transformer Engine (which the A100 lacks entirely), and faster NVLink. The A100 still runs, but for new transformer workloads the H100 is the more efficient unit.

H100 vs H200

The H200 is the same Hopper generation as the H100, refreshed for memory. Compute is identical; the H200 adds capacity and bandwidth. If your workload is memory-bound (long context, large KV cache, bigger models per GPU), the H200 helps. If it fits in 80 GB, the two perform alike and the H100 is the value pick.

What an H100 costs

H100 pricing has settled into a clear shape. Because Hopper shipped in volume and Blackwell now sits above it, the H100 has become the market's reference rate: on-demand has softened as supply broadened, spot surfaces more readily than on any frontier part, and reserved terms land well below on-demand, which is where sustained workloads belong. Era Compute tracks every level, every day, across the major clouds and European datacenters, and publishes it as one live index.

Open the live H100 price index for today's composite, the chart, and on-demand, spot and reserved levels side by side with what each provider charges. The full index covers the rest of the fleet, H100 through GB300.

H100 availability in Europe

Era Compute sources H100 capacity in European datacenters built to run Hopper nodes at density, from a single HGX node to multi-thousand-GPU clusters, with per-GPU pricing that falls as the cluster grows. Because Hopper is a mature, high-volume part, European H100 supply is deep and quick to secure, not the months-long wait that frontier Blackwell demands.

This is reserved capacity: contracted on multi-year terms and qualified per customer, not on-demand rental. For European buyers the model pays off twice. EU placement keeps training data and inference traffic inside GDPR and data-residency boundaries, and European facilities frequently undercut equivalent US capacity.

Questions, answered