Honestly, 5k is a fantastic budget for a local rig! You definitely dont need an H100 to get this running, that would be overkill for distillation work. With your budget, you should absolutely go for a multi-GPU consumer setup. Here is what I recommend checking out:

• NVIDIA GeForce RTX 3090 24GB GDDR6X: Still the king of budget VRAM! Grab two of these used if you can find them. They give you a massive 48GB total which is amazing for distillation workflows.
• NVIDIA GeForce RTX 4090 24GB GDDR6X: If you want better efficiency and faster FP8 support, go for one or two of these. Theyre expensive but super powerful for inference. Dual 3090s are basically the sweet spot for VRAM/price ratio. Just make sure your power supply can handle it! You'll be golden with that setup.

@Reply #2 - good point! VRAM is honestly the biggest bottleneck. Over the years, I've learned that cooling is just as important as the card itself. If you're building this, grab two NVIDIA GeForce RTX 3090 24GB GDDR6X cards and make sure your case has serious airflow. I've tried many setups, and overheating killed my progress more than lack of compute ever did. Don't cheap out on the PSU either, get a reliable Corsair AX1600i 1600W Titanium.

> Am I gonna hit a vram wall immediately? Look, honestly, the vram wall is the only thing that actually matters here. If you are doing distillation, you need space for the weights plus the gradients and activation buffers, so running consumer cards is totally doable if you manage the quantization right. Forget the H100 stuff. You can easily build a machine that handles this for 5k. I would suggest grabbing two NVIDIA GeForce RTX 3090 24GB GDDR6X cards since you can find them used for a steal now. They give you 48GB of total VRAM, which is honestly the sweet spot for running larger models at 4-bit or even 8-bit quantization without crashing your system.

• Motherboard: Make sure you get one with enough PCIe lanes, like an ASUS Pro WS WRX80E-SAGE SE WIFI if you want to expand later.
• PSU: Dont skimp here, get at least a Corsair AX1600i 1600W Titanium to keep those cards stable. Basically, dont overthink the enterprise gear. Just keep an eye on your memory overhead. If you use something like EXL2 quantization, you can squeeze way more into that 48GB than you'd expect. It basically makes distillation research super accessible without needing a server farm. Good luck with the build, it is definitely gonna be a beast for your project.