What is the best framework for fine-tuning DeepSeek-Coder?

Question

Hey everyone! I’ve been experimenting with DeepSeek-Coder lately for a specialized internal coding assistant project, and I’m really impressed with its performance. However, I’ve reached a point where I need to fine-tune it on our proprietary codebase to better handle our specific APIs and coding standards. I’m a bit overwhelmed by the options out there and I'm trying to figure out which framework would be the most efficient for this specific model architecture.

I’m currently looking at tools like Axolotl, LLaMA-Factory, and the native Hugging Face SFT trainer. My main constraints are hardware—I’m working with a single A100 (80GB), so memory efficiency is a huge priority for me. I’m specifically curious about which framework offers the best support for QLoRA or FSDP when dealing with DeepSeek's Fill-In-the-Middle (FIM) tasks. I've heard some frameworks struggle with the specific formatting required for code models.

Has anyone here successfully fine-tuned the 7B or 33B versions? Which framework gave you the smoothest experience in terms of configuration and stability? I’d love to hear your recommendations on the best setup to avoid common OOM errors and ensure the model retains its reasoning capabilities.

NorthernSoulMate · Accepted Answer

Sooo, for your situation, I would suggest going with Axolotl AI Fine-Tuning Framework honestly. I've spent a lot of time messing with the DeepSeek-Coder-33B-Instruct on a single NVIDIA A100 80GB GPU, and Axolotl is basically the gold standard for memory efficiency.

Before you dive in, you gotta understand why FIM (Fill-In-the-Middle) is so tricky—it requires specific prefix/suffix tokens that many trainers just mess up or ignore. If you don't handle those right, the model's reasoning basically falls apart.

Axolotl vs LLaMA-Factory: Axolotl is like... highkey better for complex configs. It has native support for FIM formatting and handles QLoRA way more efficiently without hitting those annoying OOM errors. LLaMA-Factory is cool and has a nice UI, but it feels a bit more restrictive when you're trying to push a 33B model on one card. Use Flash Attention 2 with it too, it's a lifesaver for VRAM. It works really well once you get the YAML right... good luck!!

heotxlljuo · Answer

Curious about one thing: what's your actual budget for this? If you're renting that NVIDIA A100 80GB GPU, those hourly rates are pretty steep. Honestly, i'm still a beginner, but maybe try the 7B model on a way cheaper NVIDIA GeForce RTX 4090 24GB first? Using LLaMA-Factory with 4-bit could save you a ton of money while you're still learning... anyway, good luck!

mprvyostdg · Answer

> Which framework gave you the smoothest experience in terms of configuration and stability? Just found this thread and honestly you're getting some solid advice about the hardware/budget trade-offs. To summarize, the consensus is basically torn between using specialized tools for FIM and dropping down to a 7B model when things start OOMing. From a market perspective though, if you're really worried about that A100 efficiency, you should look into Unsloth. It's been absolutely blowing up in the fine-tuning scene lately because it uses hand-optimized Triton kernels that are wayyyy faster and use significantly less VRAM than the standard Hugging Face implementations. It’s basically the go-to right now if you want to push a 33B model on a single card without it constantly crashing (at least thats what worked for me). Tbh while LLaMA-Factory is great for the UI, Unsloth is where the actual performance gains are happening in the current market. Quick tip: double check your FIM token IDs in the config because if the framework doesn't map them correctly to the DeepSeek architecture your model will just output gibberish and waste your money!