sooo i've been messing around with this exact problem lately and ngl it's kinda a headache cuz of those VRAM requirements!! honestly for your situation i would suggest looking at RunPod GPU Cloud or maybe Lambda Labs GPU Cloud instead of the big guys like AWS. i tried setting up a cluster on AWS but it was literally so expensive and i couldnt even get the NVIDIA H100 Tensor Core GPU instances i needed because of availability issues...

anyway i ended up trying a multi-node setup with NVIDIA A100 80GB PCIe cards on RunPod and it was actually pretty smooth?? i'm still a beginner at the networking side tho so be careful with the bottlenecks. DeepSeek-V3 is a beast and if you dont have fast interconnects between nodes it just crawls. make sure to check if the provider offers NVIDIA NVLink or InfiniBand because without that the latency is gonna be highkey terrible for a 671B model.

i also tried DigitalOcean GPU Droplets but for the full-scale R1 model it just didnt feel beefy enough for concurrent requests. if youre worried about costs maybe look into spot instances on Vast.ai GPU Rental? it's way cheaper but kinda risky for production cuz your instance might get killed lol. plus you gotta manage the auto-scaling yourself which is... a lot. definitely benchmark the vLLM or SGLang inference engines too cuz they helped me a ton with the VRAM usage!! gl dude!

Similar situation here - I went through this last year when my team tried to scale a similar MoE architecture. Honestly, I had some pretty bad experiences with the big providers because of the way they handle inter-node bandwidth for things like the NVIDIA H100 Tensor Core GPU clusters. We tried Amazon EC2 P5 Instances and the bill was just INSANE, plus the setup was a total headache for our engineers.

We switched to RunPod Dedicated Bare Metal and Together AI Inference API to see if we could cut costs. Unfortunately, we found that while RunPod is great for price, the networking bottlenecks for the full 671B model were highkey frustrating during peak hours. I mean, we were basically getting timeouts. We ended up trying Anyscale Ray Platform for the orchestration and it helped, but yeah, finding stable H100s without a massive commit is still a struggle. GL! 👍

In my experience, if you're hitting those VRAM walls with the 671B model, you gotta look at Vultr Cloud GPU or CoreWeave NVIDIA H100 PCIe. I've tried many setups over the years, and while the big guys are okay, the networking on specialized clouds is actually BETTER for multi-node clusters.

* Go for DigitalOcean H100 GPU Droplets if you want easy auto-scaling.
* Watch out for inter-node latency—it literally kills performance if your InfiniBand isn't configured right.

Honestly, I'd stick to H100s for the FP8 support alone; it makes DeepSeek-V3 way more manageable. gl!

I've been doing some market research on high-scale deployments for MoE architectures, and honestly, everyone focuses on the price per GPU hour while ignoring the *interconnect* bandwidth. With DeepSeek-V3's 671B parameters, you're looking at significant inter-node communication. If your networking isn't up to snuff, your latency will spike the moment you hit high concurrency because the experts are scattered across different cards. From a market perspective, Oracle Cloud Infrastructure (OCI) is actually the dark horse here. Their OCI Supercluster architecture uses a non-blocking RoCE v2 network that provides the kind of low-latency RDMA fabric you usually only find in specialized labs. It's often more stable than the 'burstable' networking you get on other providers. Alternatively, Microsoft Azure ND H100 v5 VMs are probably the benchmark for scale right now because they use NVIDIA Quantum-2 InfiniBand. Basically, if you're worried about stable latency under load, you need to prioritize providers that offer 400Gbps+ cross-node throughput. OCI usually beats AWS on price for these specific high-end clusters, and there availability for H100s has been surprisingly better lately. Just watch out for the egress costs if youre moving a lot of data out of their ecosystem (at least thats what I found during my last benchmark).

Following this thread

@Reply #9 - good point! But honestly i have been so disappointed with how some of these newer providers handle their orchestration lately. Massive latency jitter on those high-bandwidth clusters is something nobody seems to mention. Its just not as good as expected for real production loads, especially with the 671B model requirements. Id say just go with any of the gpu instances from FluidStack Cloud. You really cant go wrong there if you want to avoid the headache of manual networking config. It kind of sucks that the market is so fragmented right now, but sticking to a more established aggregator is probably your best bet for keeping things stable while managing those MoE expert routing overheads.

Subbing for updates

> I’m curious about specialized AI clouds... Since DeepSeek models can be quite VRAM-intensive—especially if we're talking about the full 671B versions So basically the consensus here is that the big names are pricey and hard to get chips from, while the specialized ones offer better networking but can be a bit more of a wild west setup. Everyone seems to agree that the interconnect is the hidden killer for these MoE models because of how they handle experts across nodes. I ran a bunch of performance benchmarks on my current setup recently and honestly the results were eye opening. On paper, the specs looked amazing, but when I actually started pushing high concurrency, the latency spikes were insane. I learned the hard way that raw GPU power doesnt mean much if ur data is just sitting there waiting for the network to catch up. I spent weeks messing with different configurations just to get a stable throughput. My main takeaway from testing is that you really need to look at ur actual workload patterns rather than just raw TFLOPS. For these huge 671B models, the bottlenecks always seem to pop up in places you dont expect during peak traffic. It reallyyy depends on how youre handling the KV cache too. Anyway, just my two cents from the trenches!

This is exactly what I needed to hear. Youre a lifesaver honestly.

I've handled large scale deployments for years and honestly people sleep on the smaller specialized providers that actually have stock and better support for modern LLM stacks. For DeepSeek-V3, you absolutely need the SXM versions of the chips because the interconnect speed on PCIe will bottleneck your MoE experts during inference once you're running multiple nodes.

• Massed Compute NVIDIA H100 80GB SXM5 clusters are currently the most cost-efficient way to get high-bandwidth memory for those 671B params.
• Hyperstack NVIDIA H100 80GB PCIe works if you're doing aggressive quantization like FP8, but sticking to SXM is much better for low-latency requirements.
• Crusoe Cloud NVIDIA A100 80GB is a solid choice if you want stable pricing without the big cloud tax. One thing to watch is the driver compatibility with vLLM or SGLang kernels specifically for DeepSeeks architecture. In my experience, some boutique clouds use outdated templates that break the custom ops needed for MoE. I usually just start with a clean image and do a manual install to avoid those headaches.