Skip to main content

One post tagged with "Autoscaling"

Autoscaling LLM inference workloads

View All Tags

From workload-variant-autoscaler to llm-d-autoscaling: retiring a custom engine for KEDA + EPP

ยท 7 min read

llm-d-workload-variant-autoscaler (WVA) is being renamed to llm-d-autoscaling. At the same time, the part of WVA that made it an "autoscaler" โ€” the analyzers and cost-aware optimizer that turned Prometheus metrics into a wva_desired_replicas signal โ€” is deprecated and will be removed from the repository. Instead, we're pointing KEDA directly at metrics that EPP now produces on its own. Quota-constrained environments get the same treatment: instead of WVA's own GPU quota limiter, we're pointing operators at Kueue, Slurm, and LSF โ€” tools already built and trusted for that job.

That scaling-decision pipeline โ€” the part of WVA that reads metrics and computes a replica count, as opposed to the CRDs and controller code around it โ€” ships as-is in release v0.9 and won't get further updates.