Honestly, 12k is a decent budget but space is gonna be your biggest enemy here if you are cramming this into a closet. Honestly, two rigs with NVIDIA GeForce RTX 4090 24GB GDDR6X cards are great for dev work, but they run super hot and the power consumption is no joke. If you go that route, you need to make sure your airflow is solid or you will definitely thermal throttle those cards during long inference tasks. Here is how I see the hardware trade-offs:

• The NVIDIA RTX 6000 Ada Generation 48GB GDDR6 is way better for LLMs because of that massive VRAM. It handles larger context windows way smoother than consumer cards.
• Dual 4090s give you 48GB of VRAM total, but syncing them across PCIe lanes for inference is a bit clunky compared to a workstation card.
• If you really have a tiny closet, the blower-style cooling on pro cards like the NVIDIA RTX A6000 48GB GDDR6 keeps the heat exhausting out the back instead of dumping it into the case. Personally, I would lean toward a single workstation build if you have the budget. It is just less of a headache to manage one rig. Dont worry too much about tokens per second, the RTX cards are fast enough for internal dev work. Just focus on getting enough VRAM so the model actually loads without offloading to system RAM, cause that will kill your performance instantly. Let me know if you need help picking out a case that actually fits those cards.

@Reply #1 - good point! Honestly, running consumer cards in a cramped closet is just asking for thermal throttling. I've dealt with this exact issue before, and those RTX 4090s become useless when they start downclocking because the airflow is trash. In my experience, you should look into workstation-grade blowers if you're stuck in a closet. Check out the NVIDIA RTX 6000 Ada Generation 48GB GDDR6 if you can swing the cost for one or two units. It has way better thermal management for dense environments. If that breaks the bank, look at the NVIDIA A4500 20GB GDDR6 in a multi-GPU cluster. It runs much cooler than the gaming cards, even if the raw power is lower. You'll get more stability for your compliance stuff without melting your office.

Subbing for updates

Can confirm