vLLM optimization pack

The vLLM optimization pack provides support for optimizing the serving configuration of Large Language Models served through vLLM, balancing latency, throughput and GPU memory efficiency.

Component Types

The following component types are supported.

Component Type	Description
vLLM	A vLLM inference server

Installing

Here’s the command to install the vLLM optimization pack using the Akamas CLI:

akamas install optimization-pack vLLM