This is one of those "Armen discovers electricity" kind of posts. Feel free to skip it, especially if you're experienced in the area of Large language models and/or are fed up with seeing articles on LLMs and "AI" popping up in every corner of the internet for the past five or so years.
I'm a hardcore "optimization guy". If there's something that can be made smaller/faster/more efficient — I'll be on it like a dog with a bone.
My first GPU was NVIDIA GeForce2 MX with only 32 Megabytes of memory. It ran almost anything like a champ in 2001/2002: Quake III Arena, Unreal Tournament, Half-Life & Counter-Strike, Deus Ex, Max Payne and, of course, GTA III. But as it got older, it started showing its age by struggling to keep up with The Elder Scrolls III: Morrowind which I was really into.
This set me on quite an upgrade journey over the following 25 years:
- GeForce4 Ti 4200
- GeForce FX 5200
- GeForce 6600 GT
- GeForce 8600 GT
- GeForce GTX 275
- GeForce GTX 580
- GeForce GTX 970
- GeForce GTX 1070 + GeForce GTX 1080 Ti
And, finally — my amazing, state-of-the-art 16GB GeForce RTX 4080 ❤️
There's something wonderful about going from 32MB of VRAM to 16GB. That's a 512-fold increase in graphics memory!
I gradually moved away from gaming and started using my computer more for creative and professional tasks like video editing, special effects, 3D, and in the last couple of years — image and text generation/manipulation with Large language models.
Recently I made a curious observation: when it comes to LLMs, we're getting better at using hardware that already exists, rather than waiting for GPUs with several hundred gigabytes of VRAM to become affordable. I can clearly see this when running local Large Language Models (like Gemma 4 or QWEN3.8) — over time, token generation speeds have been steadily increasing, seemingly without any extra effort (or new hardware) required from me.
This got me thinking about how my GPU fares against other models in the same lineup, as well as the 50 series GPUs, or even the "Professional" PRO workstation GPUs. Maybe it will be just as useful and eye-opening for you as it was for me. I've only looked into NVIDIA GPUs because these are the ones I'm most familiar with.
What's the key metric when comparing hardware for inference? — Correct! It's memory bandwidth! I've been using my RTX 4080 for a while to do all sorts of experiments with local models for text and image generation, and its high-bandwidth memory is what turned out to be key to running them successfully and at impressive speeds.
So here are the tables on each NVIDIA consumer/prosumer GPU generation, sorted by bandwidth. Starting with the RTX 20 series, as the first one designed and equipped with dedicated Tensor cores for massive matrix multiplications needed for stuff like ray-tracing or deep learning inference.
RTX 20 Series (Turing)
| GPU Model | Bandwidth, GB/s |
|---|---|
| RTX 2060 (6/12 GB) (GDDR6) | 336 |
| Quadro RTX 4000 (GDDR6) | 416 |
| RTX 2080 (GDDR6) | 448 |
| RTX 2070 (GDDR6) | 448 |
| RTX 2070 SUPER (GDDR6) | 448 |
| RTX 2060 SUPER (GDDR6) | 448 |
| Quadro RTX 5000 (GDDR6) | 448 |
| RTX 2080 SUPER (GDDR6) | 496 |
| RTX 2080 Ti (GDDR6) | 616 |
| Quadro RTX 6000 (GDDR6) | 672 |
| Quadro RTX 8000 (GDDR6) | 672 |
RTX 30 Series (Ampere)
| GPU Model | Bandwidth, GB/s |
|---|---|
| RTX A400 (GDDR6) | 96 |
| RTX 3050 (6 GB) (GDDR6) | 168 |
| RTX A1000 (GDDR6) | 192 |
| RTX 3050 (8 GB) (GDDR6) | 224 |
| RTX 3060 (8 GB) (GDDR6) | 240 |
| RTX A2000 (6 GB) (GDDR6) | 288 |
| RTX A2000 (12 GB) (GDDR6) | 288 |
| RTX 3060 (12 GB) (GDDR6) | 360 |
| RTX 3070 (GDDR6) | 448 |
| RTX 3060 Ti (GDDR6) | 448 |
| RTX A4000 (GDDR6) | 448 |
| RTX 3060 Ti (GDDR6X) | 608 |
| RTX 3070 Ti (GDDR6X) | 608 |
| RTX A4500 (GDDR6) | 640 |
| RTX 3080 (10 GB) (GDDR6X) | 760 |
| RTX A5000 (GDDR6) | 768 |
| RTX A5500 (GDDR6) | 768 |
| RTX A6000 (GDDR6) | 768 |
| RTX 3080 (12 GB) (GDDR6X) | 912 |
| RTX 3080 Ti (GDDR6X) | 912 |
| RTX 3090 (GDDR6X) | 936 |
| RTX 3090 Ti (GDDR6X) | 1008 |
RTX 40 Series (Ada Lovelace)
| GPU Model | Bandwidth, GB/s |
|---|---|
| RTX 2000 Ada (GDDR6) | 224 |
| RTX 4060 (GDDR6) | 272 |
| RTX 4060 Ti (8/16 GB) (GDDR6) | 288 |
| RTX 4000 SFF Ada (GDDR6) | 320 |
| RTX 4000 Ada (GDDR6) | 360 |
| RTX 4500 Ada (GDDR6) | 432 |
| RTX 4070 (GDDR6X) | 504 |
| RTX 4070 SUPER (GDDR6X) | 504 |
| RTX 4070 Ti (GDDR6X) | 504 |
| RTX 5000 Ada (GDDR6) | 576 |
| RTX 4070 Ti SUPER (GDDR6X) | 672 |
| RTX 4080 (GDDR6X) | 717 |
| RTX 4080 SUPER (GDDR6X) | 736 |
| RTX 6000 Ada (GDDR6) | 960 |
| RTX 4090 (GDDR6X) | 1008 |
RTX 50 Series (Blackwell)
| GPU Model | Bandwidth, GB/s |
|---|---|
| RTX PRO 2000 Blackwell (GDDR7) | 288 |
| RTX 5050 (GDDR6) | 320 |
| RTX PRO 4000 Blackwell (GDDR7) | 432 |
| RTX 5060 (GDDR7) | 448 |
| RTX 5060 Ti (8/16 GB) (GDDR7) | 448 |
| RTX PRO 4000 Blackwell (GDDR7) | 672 |
| RTX 5070 (GDDR7) | 672 |
| RTX PRO 4500 Blackwell (GDDR7) | 896 |
| RTX 5070 Ti (GDDR7) | 896 |
| RTX 5080 (GDDR7) | 960 |
| RTX PRO 5000 Blackwell (48/72 GB) (GDDR7) | 1344 |
| RTX 5090 (GDDR7) | 1792 |
| RTX PRO 6000 Blackwell Workstation (GDDR7) | 1792 |
If we pit some of the consumer cards directly against their "PRO" counterparts, an interesting picture emerges:
Looking at the table and the bar graph, you can clearly see that a "professional" card, although equipped with much more memory, still retains bandwidth fairly similar to the consumer card of the same class and with a chip of the same size. So even when using a much more expensive "Workstation-grade" GPU, we need to carefully think about how we structure the data and the algorithms we use.
If I were to swap my RTX 4080 with an overpriced RTX 6000 GPU of the same generation, I wouldn't necessarily see a huge boost in actual performance. The difference would primarily lie in the variety of models I could run and the amount of context I'd be able to squeeze into them. But for pure inference speed? That still calls for some clever tricks.
So... Why Are LLMs Getting Faster?
Big changes are happening in two areas: making models smaller without sacrificing too much intelligence, and making them generate tokens faster.
Model Compression (Or "Quantization")
When we move from 16-bit floating-point precision to 4-bit quantization, the storage required for model weights shrinks by roughly 4x. This makes it possible to fit much larger models into the memory of consumer-grade GPUs, without needing a supercomputer. But simply reducing the number of bits isn't enough. We need to be smart about how we quantize the weights, because different parts of a model have different sensitivities to precision loss. Some tensors can tolerate aggressive compression, while others need more precision to preserve the model's capabilities.
This is where GSQ (Gumbel-Softmax Quantization) and RCO (Riemannian Constrained Optimization) come in. Put simply, GSQ finds a better way to compress the model's weights, so they take up less space while staying as close as possible to the original values. RCO decides how much compression each part of the model should get: parts that need more precision get more bits, while parts that can handle it get fewer. Together, they help make the model smaller without sacrificing more accuracy than necessary.
When used together, GSQ and RCO help squeeze more intelligence into less memory, preserving as much of the original model quality as possible.
Inference Speed
Making a model smaller is only half the story. It also needs to be fast.
Traditionally, LLMs generate one token at a time: predict one token, feed it back into the model, then predict the next. This creates a sequential bottleneck because GPUs are designed to handle parallel computations.
And that's where Multi-Token Prediction (MTP) comes in.
MTP can be used to accelerate inference through "speculative decoding". Instead of waiting for the main model to generate every token sequentially, multiple future tokens are proposed. The main model then verifies these predictions together, potentially accepting several tokens in a single inference round.
The result? Fewer sequential decoding steps and potentially in most cases — much faster text generation.
It's all about finding the sweet spot: compressing models while preserving their intelligence, and reducing the time it takes to generate each response.
So most of the memory and speed improvements I've been seeing were achieved through several key techniques:
| Technology | What it actually accomplishes |
|---|---|
| MTP (Inference) | Generates and verifies multiple candidate tokens per inference round |
| GSQ (Quantization) | Improves the accuracy of very-low-bit weight quantization |
| RCO (Quantization) | Allocates different quantization types to different tensors under a fixed size budget |
| GQA (Architecture) | Reduces memory overhead by sharing keys/values across multiple query heads |
| MoE (Architecture) | Activates only a fraction of the model's parameters for each token |
I mean, of course all of this is well-known to those from the world of machine learning, but for me it was a revelation.
TL;DR
If you're thinking of running a local model for your own needs, look for the cheapest 16GB VRAM GPU with bandwidth of around 500 GB/s and obviously ignore the overpriced "PRO" card lineups. Something like RTX 5060 Ti 16GB would suffice.
Then, look for quantized models with more efficient optimizations like GSQ, RCO and ones with an MTP head — whether as part of a model or as a separate file. Here's where you get the same impressive speedups that I'm seeing with my GPU.
I can recommend a few models that combine most of the aforementioned techniques:
- Qwen3.8-27B-OrcaRouter-GSQ-RCO-IQ3_XXS-Uncensored — Built on the well-known Qwen3.8-27B, it's a very capable model, especially for coding and analysis tasks
- Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced — A very good and ridiculously fast general-purpose model (not fit for coding)
- Tiel-Coder-35B-A3B-GGUF-MTP — An amazing variant of Qwen3.6-35B-A3B mixture-of-experts (MoE) model that is extremely fast and highly specialized for coding tasks, while being too heavily lobotomized for anything else
Oh yeah, I prefer uncensored (or "abliterated") models because my models must do exactly what I tell them, no exceptions.
Finally, maybe try diving deeper into MoE models, since a lot of exciting development is happening in this area, as seen with Strata — a fairly new open source inference engine meant to make running giant models like Qwen3.8-Flash-Next 125B possible on systems with limited GPU memory and plenty of system RAM (over 48GB at least).
Yours truly,
Captain Obvious
