HBM (High Bandwidth Memory)

HBM (high bandwidth memory) is a type of DRAM in which memory dies are stacked vertically and connected to the processor over a very wide interface through a silicon interposer, delivering far higher bandwidth than conventional DDR or GDDR memory. HBM sits millimeters from the GPU die, which matters because AI workloads are memory-bandwidth-bound. Current HBM3E stacks deliver over 1 TB/s each; NVIDIA's B200 GPU carries 192 GB of HBM3E at about 8 TB/s aggregate bandwidth, and HBM4-generation parts push per-GPU bandwidth well beyond 10 TB/s. Only three suppliers exist — SK hynix, Samsung, and Micron — and HBM supply has repeatedly been the binding constraint on AI accelerator production. HBM capacity per GPU largely determines the maximum model size servable per device in inference.