vLLM
High-throughput LLM inference server with PagedAttention, continuous batching, and an OpenAI-compatible API.
API & Serving Intermediate
Category
API & Serving
Difficulty
Intermediate
When to use
You're self-hosting an open-weights LLM and care about throughput, latency, and GPU utilization.
When not to use
You only need to run a model on a laptop — llama.cpp or Ollama is simpler.
Alternatives
TensorRT-LLM TGI llama.cpp Ollama