TurboQuant: Google’s New Trick for Squeezing AI Models Without Breaking Them

Google Research just dropped three new compression algorithms at ICLR and AISTATS 2026, and they’re worth paying attention to if you’ve ever hit a memory wall with large language models or vector search.

The trio — TurboQuant, Quantized Johnson-Lindenstrauss (QJL), and PolarQuant — targets the same fundamental problem: high-dimensional vectors are memory hogs. Every LLM relies on them for the key-value cache, that fast-access scratchpad that stores recent context so the model doesn’t have to recompute everything. But as context windows grow, that cache balloons. Traditional vector quantization helps, but it usually adds its own overhead — 1 or 2 extra bits per number for storing quantization constants. That partially defeats the purpose.

TurboQuant is the headline act, and it’s clever. Instead of treating compression as a single pass, it splits the job into two stages. First, it randomly rotates the data vectors — a neat geometric trick that makes the data easier to quantize uniformly. Then it applies PolarQuant, which handles the heavy lifting, capturing most of the vector’s meaning with the bulk of the available bits. The remaining error — tiny but systematic — gets mopped up by QJL, which uses just 1 bit per number as a mathematical error-correction layer. The result is a compression scheme that, according to their tests, achieves massive size reduction with zero accuracy loss.

QJL is the simpler of the two supporting algorithms, but I think it’s the most elegant. It’s based on the Johnson-Lindenstrauss lemma, a classic dimensionality reduction technique that preserves pairwise distances. QJL takes each vector component and reduces it to a single sign bit — +1 or -1 — with zero memory overhead. To maintain accuracy, it uses a special estimator that balances a high-precision query against the low-precision data. It’s essentially a high-speed shorthand that doesn’t cheat.

PolarQuant takes a different angle — literally. Instead of representing vectors in standard Cartesian coordinates (x, y, z), it converts them into polar coordinates (angle, magnitude). This lets it quantize the angle with very few bits while keeping the magnitude in higher precision, which turns out to be a much more efficient allocation of bits for many real-world vector distributions. It’s not a new idea — polar quantization has been around in signal processing for decades — but applying it to LLM key-value caches is smart.

What’s refreshing here is that Google isn’t just throwing compute at the problem. These algorithms are theoretically grounded — the blog post is unusually rigorous for a research announcement — and the results seem legit. They report being able to compress KV cache entries to as little as 2 bits per number without degrading model performance on standard benchmarks. Compare that to typical 16-bit or 8-bit quantization, and the savings are substantial.

Of course, there’s always a catch. These methods add preprocessing overhead: the random rotation step in TurboQuant, for instance, requires generating and applying a random matrix, which isn’t free. For offline compression of static vectors, that’s fine. But for real-time KV cache compression during inference, the latency budget might be tight. The paper (which I’d love to see in full — the blog post truncated some details) should clarify whether this works efficiently on modern accelerators.

Still, this is the most interesting compression work I’ve seen in a while. Most quantization papers focus on weight compression, but the KV cache is becoming the real bottleneck as models scale context windows to 128K, 256K, or beyond. If TurboQuant delivers on its promises, it could make long-context LLMs practical on consumer hardware, not just giant clusters.

I’m also curious to see how these techniques perform on non-transformer architectures or retrieval-augmented generation pipelines where vector search is the bottleneck. The blog post hints at applications beyond pure LLM inference, and the math is general enough to apply anywhere you have high-dimensional vectors and limited memory.

For now, this is one to watch. Google’s track record with quantization is mixed — some of their earlier work was overly theoretical — but TurboQuant feels different. It’s grounded, it’s practical, and it solves a real pain point. I’ll be digging into the full papers when they’re available.