Late to the party but be careful with the power draw if you go the multi-gpu route. Make sure to check your home wiring because too many cards will trip a breaker fast. Quick question tho, are you planning to run the full 671B model or just a smaller distillation? You might save some serious cash looking for used NVIDIA RTX A6000 48GB GDDR6 units instead of stacking multiple consumer cards.

I've been using the Apple Mac Studio M2 Ultra 128GB RAM for a while now and I'm very satisfied. It works well for these massive 671B architectures even at 4-bit. Quick tip: if you stick with the Nvidia GeForce RTX 3090 24GB route, check your motherboard for PLX chips. It keeps the P2P communication fast... that methodical hardware choice saved me hours of debugging.

Totally agree on the memory logic! You absolutely need that capacity for something this big.

• it just works Seriously, that unified memory is amazing!

Man I wish I found this thread sooner. Would have saved me so much hassle.

@Reply #4 - good point! Coming back to this and honestly... it's just disappointing how much work it takes to get this stuff running. I tried to make it work with my current setup of several NVIDIA GeForce RTX 3090 24GB cards and it was honestly not as good as expected.

• The heat output is basically like having a space heater under my desk.
• I had so many issues with the drivers failing during long runs.
• The memory bandwidth just isnt there when you're spanning across multiple PCIe slots. I even looked into some pro-grade gear like the NVIDIA RTX A6000 48GB GDDR6 thinking it would fix my VRAM anxiety, but the price is just painful for what you get. Nvidia is usually the way to go for CUDA support, but for these huge 671B models, the consumer hardware just feels like it's hit a wall. I kinda wish I hadn't spent so much on my current rig because the performance just feels underwhelming compared to the effort I put in. Tbh, the whole multi-GPU route is way more headache than it's worth.