Buoyant launches Kinference for Kubernetes, with alpha limits

Buoyant launched Kinference on November 4, 2026, giving Kubernetes teams a new control layer for running shared AI inference. It places routing policy and cache aware load balancing around an existing vLLM fleet. The immediate opportunity is better control over where requests go, rather than a new model or a hosted inference subscription.

The release needs a careful reading. The current documentation identifies version v0.1.0-alpha.7, and the repository warns that the software has not had sustained production exposure. Several capabilities promoted in the launch remain on the roadmap. Teams evaluating it should start with the implemented routing and observability features, not assume the full product description is available today.

What Kinference can route today

The policy guide describes three outcomes for an incoming request: send it to a local vLLM worker, send it to an external inference provider, or reject it. Operators can choose candidates and filter them using request information, tenant identity and the state of their serving infrastructure. Policies are attached to inference pools, so creating a policy alone does not activate it.

This separates application choices from infrastructure decisions. A client can request a model while the platform determines which eligible instance serves it. Changing the routing rule can therefore happen centrally, without asking every application owner to maintain a different list of endpoints. The kinfctl command line tool validates policy configuration before it reaches a cluster. Validation catches configuration problems. Teams still need to test whether a policy produces the behavior they intend.

The important gaps in the alpha

The repository’s known limitations list four missing areas: Anthropic style /v1/messages calls, rate limiting, semantic classifiers and support for multiple Kinference replicas. Those limits qualify the broader launch language about API compatibility and platform features. Multiple inference workers are part of the documented serving design, but that should not be confused with multiple replicas of the control plane itself.

The feature pages separately label token rate limiting and semantic classification as future releases. In particular, examples involving automatic detection of personal information or prompt complexity are not evidence that the current alpha supplies those classifiers. Do not make a data handling guarantee depend on an unimplemented detection feature.

Engine support is narrower than a general inference platform might suggest. The getting started guide uses vLLM and describes SGLang and TensorRT-LLM as future options. The supported engine, API path and control plane redundancy should all be explicit parts of an evaluation plan.

Why cached context changes the routing decision

According to the load balancing documentation, Kinference tracks each worker’s cache events and queue state, then predicts time to first token for a particular prompt. Routing by that prediction is the default. It requires vLLM to emit KV cache events through ZMQ, so simply placing a gateway in front of an unchanged deployment is not enough to obtain the advertised cache awareness.

For a repeated conversation, a worker that already holds useful context may need less computation than one starting cold. A busy worker can still be a poor destination. The useful decision combines reuse with load, rather than always selecting the least occupied server or always returning to the same one.

Buoyant’s launch reports a 95% reduction in median time to first token and a 45% reduction in median end to end latency against standard Kubernetes load balancing on one workload. These are vendor reported results, not independent ByteForward measurements or an expected improvement for every application. They should motivate a workload matched test, not become a savings estimate in a purchasing plan.

What a deployment actually requires

The installation consists of a traffic gateway and a separately released inference handler, with Helm charts and Kubernetes resource definitions. You must already have a working vLLM deployment. Kinference does not provide model weights or turn an empty cluster into a model server.

The serving guide calls for a recent vLLM version, giving 0.28.0 or later as an example, plus exported cache events and prompt token details. An InferenceWorkerSet describes each vLLM service, an InferencePool combines workers, and a Gateway API HTTPRoute sends requests to the pool. The example route uses /v1/chat/completions.

The configured KV block size also matters: the guide says Kinference does not detect it automatically. A gateway that responds successfully is only the first check. Operators should confirm that cache events, token counts and the selected worker configuration are correct before interpreting performance graphs. ByteForward’s Intel Inference Microservices deployment guide covers the related distinction between a working model container and the wider platform around it.

Tenant identity and monitoring need deliberate setup

The tenant guide says Kinference reads tenant identity from a configurable header and trusts that value unconditionally. Operators must prevent clients from arbitrarily setting it. That makes trustworthy identity handling in the surrounding infrastructure a prerequisite for policies that distinguish one team from another.

Per-tenant metrics also have an operational cost. The documentation warns that each tenant, model and quality of service combination creates almost 100 time series. It offers an allowlist or the option to disable per-tenant tracking when that volume would overload monitoring. Decide how much detail is useful before enabling an unbounded set of tenant labels.

Kinference exports Prometheus compatible metrics and provides Grafana dashboards. Time to first token, the gap between tokens and complete request latency answer different questions. A convincing evaluation should measure all three alongside failures and resource use, with realistic concurrency and both repeated and unfamiliar prompts.

Availability and pricing

Kinference’s public code is licensed under Apache 2.0. The software is available to deploy, but running it still requires infrastructure, model access and operational work. Calls routed to an external provider can also carry that provider’s charges. Open source availability does not make the resulting inference service costless.

Buoyant separately sells Buoyant AI, a solution built on Kinference and vLLM for coding agents, with prepared model deployments. Its pricing page directs buyers to contact the company rather than listing a public price. That commercial offering should not be conflated with the downloadable project.

The strongest reason to try this alpha is a concrete routing problem in an existing vLLM cluster. Compare it with your current service using the same models, hardware and request mix. Check failover requirements and API compatibility before connecting a coding harness. For teams whose immediate need is enforced token budgets, built-in semantic classification or Anthropic style messages, the documented gaps are reasons to wait or retain other components.

AI-generated editorial illustration by ByteForward using the official Kinference wordmark. Conceptual routing scene, not a product interface or hardware photograph. No endorsement implied.