New Flash Memory Tech Aims to Solve AI's Growing Memory Demands
The rapid expansion of Large Language Models (LLMs) is creating an unprecedented demand for memory, prompting memory manufacturers to accelerate plans for new High Bandwidth Memory (HBM) and DRAM production facilities, with initial output expected in 2027. This growing need also presents an opportunity for innovative memory solutions, such as High Bandwidth Flash (HBF). HBF adapts HBM's successful chip-stacking approach to NAND flash memory, commonly found in devices like SD cards and thumb drives. While traditional NAND flash is known for its slow write speeds, HBF aims to significantly improve read performance. Jim Handy, general director at Objective Analysis, notes that HBF will enhance flash's read capabilities, making it faster for specific applications. Unlike DRAM, NAND flash is non-volatile and denser, storing data without continuous power. However, its insulated gate mechanism makes writing data slower. The latest flash interface supports up to 4.8GB/s per die, but this lags far behind HBM's potential speeds. Hoshik Kim, senior vice president at SK Hynix, explains that HBF utilizes advanced 3D packaging and vertical stacking techniques, similar to HBM, to boost bandwidth. SanDisk has outlined plans for HBF, projecting up to 16 stacked NAND chips for 512GB capacity and read bandwidths reaching 1.6TB/s in its first generation, with future iterations targeting 2TB/s and 3.2TB/s. The key advantage of HBF lies in AI inference, where model weights are largely static and read-only. This makes flash memory, with its slower write performance, a suitable option for storing these large, read-heavy datasets, freeing up HBM for faster operations. Handy believes HBF is a practical approach for inference workloads, likening it to advanced caching. Although HBF is still in early development, SanDisk and SK Hynix have partnered to standardize the technology through the Open Compute Project (OCP). Kim views HBF not as a competitor to HBM but as a complementary technology that can address HBM's capacity limitations, potentially reducing the number of accelerators needed for large models, thereby improving energy efficiency and lowering costs for AI inference hardware.
The burgeoning demand for AI, particularly LLMs, highlights a critical bottleneck in memory capacity and speed. While HBM has emerged as a high-performance solution, its cost and capacity limitations necessitate complementary technologies. High Bandwidth Flash (HBF) represents a strategic effort to leverage the density and non-volatility of NAND flash for AI inference, a less write-intensive workload. By applying advanced packaging techniques to NAND, HBF aims to bridge the performance gap, offering a more cost-effective and power-efficient alternative for storing massive, read-heavy datasets like model weights. This development underscores a broader trend in computing: optimizing hardware for specific workloads rather than seeking a one-size-fits-all solution. The collaboration between SanDisk and SK Hynix within the Open Compute Project suggests a move towards industry standardization, which could accelerate HBF's adoption. The success of HBF will depend on its ability to deliver on its projected performance gains while maintaining cost advantages over HBM, ultimately influencing the scalability and economic viability of future AI deployments.
AI-generated to prompt reflection — not editorial opinion, not advice, not a statement of fact. How this works.