vLLM

This page describes the Optimization Pack for the component type vLLM.

Metrics

Metric	Unit	Description
e2e_request_latency_p95	milliseconds	95th percentile of the end-to-end request latency
e2e_request_latency_avg	milliseconds	Average end-to-end request latency
request_success_rate	requests per second	Rate of successfully completed requests
time_to_first_token_p95	milliseconds	95th percentile of the time to first token (TTFT)
time_to_first_token_avg	milliseconds	Average time to first token (TTFT)
prefill_token_throughput	tokens	Prefill-phase token throughput
inter_token_latency_p95	milliseconds	95th percentile of the inter-token latency (ITL)
inter_token_latency_avg	milliseconds	Average inter-token latency (ITL)
decode_token_throughput	tokens	Decode-phase token throughput
num_requests_running	requests	Number of requests currently being processed
num_requests_waiting	requests	Number of requests waiting in the queue
kv_cache_usage_avg	percentage	Average KV cache utilization
kv_cache_usage_max	percentage	Peak KV cache utilization
preemption_rate	preemptions per second	Rate of request preemptions
prefix_cache_hit_rate	percentage	Prefix cache hit rate
input_sequence_length_p95	tokens	95th percentile of the input (prompt) sequence length
input_sequence_length_avg	tokens	Average input (prompt) sequence length
output_sequence_length_p95	tokens	95th percentile of the output (generated) sequence length
output_sequence_length_avg	tokens	Average output (generated) sequence length
fleet_e2e_latency_percentile	milliseconds	Fleet-wide end-to-end request latency percentile
fleet_inter_token_latency_percentile	milliseconds	Fleet-wide inter-token latency percentile
fleet_request_success_rate	requests per second	Fleet-wide request success rate
fleet_kv_cache_usage_avg	percentage	Fleet-wide average KV cache utilization
inter_token_latency_p95_5m	milliseconds	95th percentile inter-token latency over a 5-minute window
inter_token_latency_p95_max_per_pod	milliseconds	Maximum per-pod 95th percentile inter-token latency
inter_token_latency_p95_min_per_pod	milliseconds	Minimum per-pod 95th percentile inter-token latency

Parameters

Parameter	Type	Unit	Default	Domain	Restart	Description
gpu_memory_utilization	real	percentage	0.9	0.0 → 1.0	yes	Target fraction of GPU memory reserved for the model executor and KV cache
max_num_seqs	integer	sequences	128	1 → 1000000000000	yes	Maximum number of sequences processed together in a single iteration (batch)
max_num_batched_tokens	integer	tokens	1024	1 → 1000000000000	yes	Maximum number of tokens processed together in a single iteration (batch)
max_model_len	integer	tokens	8192	1 → 1000000000000	yes	Maximum model context length, in tokens
tensor_parallel_size	integer	parallelism degree	1	1 → 1000000000000	yes	Number of GPUs across which each model instance is sharded (tensor parallelism)