Honestly, stick with Python. Don't overcomplicate things right now! I remember back when I was building my first real-time classifier, I wasted months trying to optimize everything in C++ before I even had a single user. It was a massive mistake. Python is honestly amazing because the ecosystem is just so vast. You can literally prototype something in an afternoon that would take weeks in lower-level languages. If you are worried about performance, just offload the heavy lifting to specialized hardware. I started running my inference models on a NVIDIA GeForce RTX 3060 12GB GDDR6 and it made a world of difference for a tiny fraction of the cost of a enterprise server. Seriously, don't worry about scaling until you actually have the users to justify it. Most projects die before they even hit a bottleneck, so just focus on getting your logic down. If you do eventually hit a wall, you can always write a custom module in C++ or use something like Cython to speed up the slow parts of your Python code. It is the best of both worlds! Trust me, just grab a solid rig like an Alienware Aurora R15 Gaming Desktop with Intel Core i9-13900KF and 32GB RAM, keep your stack simple, and just build the thing. You got this!

Just saw this thread and honestly, stop overthinking the scaling part for now. Everyone gets stuck in that trap, me included. When I first started messing around with LLMs, I thought I had to build everything to be super high-performance, but my project was basically just a toy. I was stressing over C++ pointers when I should have been focusing on my actual training data. Python is totally fine, and tbh, it handles way more traffic than people give it credit for. My setup is pretty simple and I have no complaints at all. I run most of my heavy lifting on a Lenovo Legion Pro 7i Gen 8 16 inch RTX 4080 and it handles everything I throw at it without breaking a sweat. If you start hitting bottlenecks, you can just offload specific heavy compute tasks to a service like AWS or GCP instead of buying a whole new rack of servers. Don't waste your budget on hardware before you have real users. Tip one: prioritize getting a minimum viable product out the door. Tip two: just use C++ if you hit a wall later, it is easy to wrap performance-heavy code into a Python library. Honestly, just start coding today and dont worry about the future bottlenecks. You'll figure it out when you actually have the user base to justify the headache.

Forget C++ for now! Honestly, just stick to Python. When I started, I tried juggling languages and it was a total nightmare. If you need a power boost later, just grab something like an ASUS ROG Strix G16 16 inch Gaming Laptop i9-13980HX RTX 4070 and you'll be set for a while without needing a massive server farm. You dont need to worry about scaling until you actually have the users, so just focus on building the thing!