1. Use EXL2 at 4.0bpw for max speed on ur NVIDIA GeForce RTX 3090 24GB setup.
2. Tbh, GGUF IQ4_XS handles MoE quants better if the reasoning feels off.

+1 to what was said earlier! IQ quants like GGUF IQ4_XS are basically magic for MoE models cuz they keep those expert layers from getting too messy. Over the years, I've noticed reasoning reallyyy starts to tank below 4-bit. Since you're using dual NVIDIA GeForce RTX 3090 24GB cards, honestly try a 4.5bpw EXL2 quant. It's the best value for speed and smarts without needing a crazy expensive A100. Good luck!

> Basically, if you had 48GB of VRAM to play with, what quantization method and bit-rate would you recommend to get the best balance of speed and "smartness" out of DeepSeek-R1? Honestly, I’m still kind of a beginner at this, but I’ve been looking at the market trends for different backends and one thing I’m worried about is the KV cache overhead. Even if a specific quant "brand" like EXL2 or AWQ says it fits in 48GB, you have to be super careful because DeepSeek-R1’s reasoning involves massive context. I’ve seen some data suggesting that if you don't leave enough VRAM for the cache, the whole thing will just crash or crawl to a stop once it starts thinking hard. Basically, the "brand" of loader you pick matters just as much as the bits. Like, vLLM handles memory differently than llama.cpp, and if ur pushing right to the limit, you might get OOM errors right in the middle of a long chain-of-thought. Tbh, I’d caution against aiming for the absolute highest bpw just because it looks better on paper. Has anyone looked into how much VRAM the context window actually eats up for these MoE architectures at 32k or 64k context? I dont want you to waste time downloading a huge file only for it to fail because of the cache.

Ok adding this to my list of things to try. Thanks for the tip!

> I'm curious if EXL2 at 4.0bpw holds up better than a standard GGUF Q4_K_M In my experience, quantization basically squishes the model to fit VRAM, but it can lose detail. This matters cuz R1 needs precision for reasoning! With my current setup, I was sooo nervous about it breaking!! honestly i think 4.5bpw EXL2 is amazing and still feels reallyyy smart. I’m a beginner tho, so I’d stay cautious and not go below 4-bit? it’s just safer!! 👍

Saved for later, ty!

Any updates on this?

This ^

Noted!

tbh I am in the exact same boat and it is driving me crazy... i have been trying to figure this out since the release dropped and i still havent pulled the trigger on a specific quant because i am so scared of ruining the reasoning capabilities. with a 48gb setup like ours you really want to maximize that vram without hitting the wall where the logic falls apart but the concrete data just isnt there yet. it is honestly so frustrating waiting for a definitive side-by-side comparison so we dont waste hours downloading files only to find out the model has turned into a hallucination machine.

I totally agree with what everyone is saying about being careful with the reasoning logic. I tried an experimental build a few days ago on my current setup and it was honestly a mess. I thought I was being clever by trying to squeeze every bit of efficiency out, but the compatibility between the model format and the loader I usually use was just... off. It kept spitting out these weird repetitive loops after about 500 tokens. It really taught me that fitting in VRAM is only half the battle; if the software doesnt play nice with the specific quantization math, youre gonna have a bad time. I have become way more conservative lately because of that. Id rather lose a few GB of space to a more standard, well-tested format than risk the model losing its mind mid-sentence because of some weird optimization bug. Reliability is definitely underrated when you just want things to work.