14/07/2026 2:25 am
Topic starter
Notifications
Clear all
DeepSeek Forum
3
Posts
4
Users
0
Reactions
104
Views
0
Im losing my mind trying to figure out the right quantization for DeepSeek V4 Flash. I really need to get this running locally for a research project on legal document analysis by next Tuesday, but my hardware budget is basically non-existent and Im stuck using an older 3090 rig in my basement. I keep reading that GGUF is the way to go because of the llama.cpp support, but then I see people on discord swearing by EXL2 for faster speeds on consumer cards. Its all just a mess. I tried a 4-bit EXL2 quant earlier and the memory usage was fine but the output started hallucinating like crazy, and then when I tried a 8-bit GGUF it just crawled at like 0.5 tokens per second which is totally unusable for my timeline. Is there some kind of middle ground or a specific calibration dataset I should be using to keep the performance somewhat stable? Does anyone know if there is a massive quality drop-off between Q4 and Q6 for this specific architecture or am I just wasting my time messing with these settings? Honestly I feel like Im gonna miss my deadline at this rate...
3 Answers
11
14/07/2026 2:25 am
TL;DR: Stick to Q5_K_M GGUF. Honestly, avoid 4-bit EXL2 for legal work, it gets messy fast. Just grab it for your Nvidia GeForce RTX 3090 24GB and stay patient.
11
14/07/2026 3:00 am
Honestly, the hallucination issues you're seeing at 4-bit are pretty standard for complex legal text where nuance is everything. The architecture on DeepSeek V4 is super sensitive to compression because of the MoE setup. If you're stuck on a Nvidia GeForce RTX 3090 24GB, you really have to balance the VRAM overhead against the quantization precision. Here is how to look at the trade-offs:
- EXL2 is fine for general chat, but for legal analysis, you need more bits. Try moving to a 6-bit EXL2 quant if you can manage the VRAM. It preserves the logical reasoning chains way better than 4-bit.
- If you keep using GGUF, ignore the Q8. It is overkill and kills your speed. Aim for Q5_K_M or Q6_K. The Q6_K is the sweet spot for this model where the perplexity drop-off is actually negligible compared to FP16.
- You asked about calibration: definitely use a custom dataset. If you use the standard wiki test, it won't help with legal terminology. Run your quant script using a sample of your actual document corpus. It helps the model weights stay aligned with the domain-specific language you're actually analyzing. Basically, don't waste time with 8-bit. The memory bandwidth on the Nvidia GeForce RTX 3090 24GB is the bottleneck, not the compute power. Use a 6-bit quant to keep the context window large enough for your documents while maintaining enough weights to stop the hallucination loop.
3
14/07/2026 3:10 am
@Reply #2 - good point! Honestly, the MoE architecture is notorious for falling apart when you squeeze it too hard. I've had issues with 4-bit quants on legal docs too, it just misses all the nuance. Since you're on a Nvidia GeForce RTX 3090 24GB GDDR6X, have you actually tried a Q5_K_M GGUF yet or are you still jumping between extremes? I personally think you should try using GGUF IQ4_XS quantization. It usually strikes that weird middle ground where it keeps the logic intact without killing your VRAM, but tbh I'm curious, are you using specific system prompts to steer it? Sometimes these models just need a bit of guidance to stop the hallucinations. Also, are you running this off a Samsung 980 Pro 2TB NVMe SSD to help with swapping? Let me know so I can give you a better idea of how to fix this before your deadline.
