vLLM optimization pack
The vLLM optimization pack provides support for optimizing the serving configuration of Large Language Models served through vLLM, balancing latency, throughput and GPU memory efficiency.
Component Types
The following component types are supported.
Component Type
Description
A vLLM inference server
Installing
Here’s the command to install the vLLM optimization pack using the Akamas CLI:
akamas install optimization-pack vLLMLast updated
Was this helpful?