> Does going down to 4-bit significantly hurt the chain-of-thought process, or is it better to stick to 6-bit or 8-bit even if it means slower inference?

Sooo, here is the deal with reasoning models like DeepSeek-R1: they are definitely more sensitive to quantization because that "Chain of Thought" (CoT) process relies on super precise weights to maintain logic. Basically, if you go too low, the model starts yapping nonsense or loses its train of thought midway through a complex problem. Ngl, 4-bit is usually the "danger zone" where you start seeing some IQ loss, but with your dual NVIDIA GeForce RTX 3090 24GB setup, you have 48GB to play with!

I honestly recommend aiming for EXL2 at around 4.5bpw to 5.0bpw. Since youre staying on GPU, ExLlamaV2 is sooo much faster than llama.cpp with GGUF. I tried a 4-bit GGUF and it felt a bit "lobotomized" on math tasks, but a 5.0bpw EXL2 felt almost identical to the full weights! If you can squeeze a 6-bit version in there, do it, but 5-bit is usually the sweet spot for speed and logic. Good luck!! 👍

yo, i feel u on the vram struggle... even with a dual NVIDIA GeForce RTX 3090 24GB setup, these r1 models are absolute units. i've been messing around with local llms for years now and honestly, reasoning models like DeepSeek-R1 are a bit more finicky when it comes to quantization compared to your average llama-3.

For your situation, here's what I recommend after testing a bunch of formats:

* **EXL2 (4.5bpw to 5.0bpw):** This is literally the way to go since you're on dual gpus. It's way faster than gguf when you're staying strictly on cuda. i noticed that at 5-bit, the logic stays pretty sharp, but once you drop below 4-bit, the chain-of-thought starts getting a bit... loopy?
* **GGUF (Q5_K_M or Q6_K):** If you want rock-solid stability and don't mind a slight speed hit via llama.cpp, this is the safest bet. it handles multi-gpu okay, but it's just not as snappy as exl2 on dual 3090s.
* **AWQ/HQQ:** These are cool for 4-bit, but personally, i think they lose a bit too much 'soul' in the reasoning process for a model this complex.

Basically, if you go below 4-bit, the intelligence degradation is REAL. it starts skipping steps in the logic, which kinda defeats the purpose of r1, right? i'd stick to 5-bit exl2 if you can squeeze it into that 48gb vram pool... it's like the sweet spot for speed and smarts. gl with the download, it's worth the wait tho! 👍

@Reply #4 - good point! Honestly, after messing with these things for years, Ive learned that being conservative is usually better than chasing the newest experimental format. I agree with the guys above that EXL2 is the way to go for speed, but there is more to it than just the bit-rate.

• In my experience, you should just stick with NVIDIA hardware. You really cant go wrong with their ecosystem when it comes to stable drivers and community support.
• I've tried many different setups and anything that lets you stay purely on the GPU is gonna be less headache. Just get any software from a reputable developer that supports EXL2 and call it a day.
• Honestly, just go with the most popular brand in the space and dont overthink it. Stability matters way more than a 5% speed boost when youre trying to get a reasoning model to actually think without it crashing your whole system...

Building on the earlier suggestion, you should really stick to 5-bit or 6-bit if you want to keep that reasoning sharp, because 4-bit can get a bit messy with complex logic. I've tried many different setups over the years and the power draw is usually what ends up being the bottleneck for me. It reminds me of when I tried to run a whole cluster off a single outlet in my old apartment.

• The circuit breaker tripped every time I started a training run.
• I had to run an extension cord from the kitchen across the hallway.
• My roommate kept tripping over it and nearly took out the whole rack. We eventually had to tape the cord to the ceiling just so we could walk around without dying. It looked like some kind of weird sci-fi movie set in there with all the black cables hanging down. Anyway lol sorry kinda went off topic there.

Following this thread

Interested in this too

Man I wish I found this thread sooner. Would have saved me so much hassle.

Hmm, I've had a different experience with the whole 'high bitrate or bust' mentality. While the previous guys are right that reasoning models are sensitive, jumping straight to an 8-bit GGUF is basically a VRAM death sentence on a NVIDIA GeForce RTX 3090 24GB dual setup if you want any decent context window. Honestly, the 6-bit and 8-bit files are huge and slow as molasses.

I'd actually suggest a different approach—stick with EXL2 quantization around the 4.5bpw to 5.0bpw range. Basically, EXL2 is way more efficient for dual-GPU setups like ours cuz it keeps everything on the VRAM and avoids the massive overhead of llama.cpp GGUF offloading.

* **4.5bpw EXL2:** Best bang for your buck. Logic stays sharp and it actually fits.
* **Flash Attention:** Make sure it's toggled on to save VRAM.
* **Cache:** Use 4-bit cache to squeeze in more tokens.

I tried the 4-bit GGUF and it was... okay? But the speed was disappointing. Seriously, stick to EXL2 for the dual 3090s. It’s way more cost-effective for your time. gl!

Any updates on this?

+1