Top web hosting services for deploying large language models?

Question

So I have been building standard web apps for like 8 years now, mostly Docker stuff on AWS ECS or just throwing things on Vercel. But I am finally dipping my toes into hosting open-source LLMs for a new client project and man, the infra side of this is a whole different beast. I am super excited to get this out there but the hosting costs for GPUs are throwing me for a loop.

We are building a custom support agent using a fine-tuned Llama 3 8B model. The client is based in Germany so I need something with decent European latency. Here is what I am trying to figure out:

Budget is around 100 to 150 bucks a month max for the model hosting part

Must support GPU acceleration because CPU inference is just way too slow for a chat interface

Decent scale-to-zero or serverless options would be amazing so we arent paying for idle time at 3 AM

Easy deployment pipeline, hopefully docker based since thats what I know

I looked at Replicate and RunPod but keeping a dedicated GPU instance running gets pricey fast, and the serverless cold starts on some of these platforms seem brutal. What hosting services are you guys using to deploy these mid-sized models without breaking the bank?

qennkjwekj · Accepted Answer

> serverless cold starts on some of these platforms seem brutal Yeah, I tried running that on a NVIDIA A10G GPU on RunPod but the cold starts unfortunately killed our UX. Had to switch to a dedicated NVIDIA L4 GPU on GCP instead.

wpdperkhgi · Answer

In my experience, serverless is the only way to hit that budget for Llama 3 8B. I've tried many setups over the years, and you should look at Baseten or Modal. Here is what works best:

Use NVIDIA A10G 24GB GPU instances on Baseten.

Set up cold-start optimization using TensorRT-LLM. This keeps idle costs at zero and latency low for EU users.

RegentsCanalBoat · Answer

Like someone mentioned, those cold starts are a total vibe killer for real-time chat! I ran into the same wall with a project last year and it was honestly so frustrating to watch the loader just... spin. Before I dive into what worked for me, are you gonna run it at FP16 or are you looking at quantization? Tbh that makes a massive difference for staying under that 150 budget!