Jut cut my AWS bill with 82%, and sped up my ML inference from 8 sec to 0.02 sec.
My tips:
- Use TorchServe with TensorRT, batch size 4
- Deploy with FastAPI, Gunicorn (w/ Uvicorn worker), and Nginx
- AWS auto-scaling cluster (NVIDIA A10G), ECS, LB w/ WAF
- Reserved instances