Together Compiler targets inference costs, with customer access in beta
Together AI introduced Together Compiler on November 4, using AI agents to optimize the software that serves models. The company says it already runs on some production serverless endpoints. Customers can request limited beta access, while self-service optimization for dedicated endpoints remains a planned capability.
The practical question for developers is whether an optimization improves their application under real demand. A faster benchmark can justify a trial. A production decision needs evidence about response times, operating costs and reliability.
Who can use Together Compiler?
Together places the release within its Polymorphic Inference initiative, which aims to adapt serving systems to changing workloads. Its announcement directs interested customers to contact the company for early access.
Before arranging a trial, ask which model and hardware combinations your account can use, what access includes and how results will be measured. Request a clear description of any additional charges and the process for reverting an unsuccessful change. These are evaluation questions, rather than features confirmed by the announcement.
How the optimization works
The technical report describes a system that takes the model, hardware and traffic characteristics as inputs. Multiple agents propose changes, while persistent inference engines test candidates without repeatedly restarting. Smaller engine configurations support rapid experiments, and larger ones check performance with more representative inputs.
Together reports up to 2.6 times the tokens per second against optimized baselines in its frontier-model evaluation. It also reports 1.7 times the performance of available open-source megakernels on a Llama 1B, batch-size-one workload. These are vendor-reported results for specified tests.
One methodological detail matters: the report says its Trainium2 evaluation used a synthetic multi-turn dataset instead of AgentX because of a concurrency limitation. Readers comparing hardware should retain that qualification.
These measurements concern inference execution. They do not establish that a model has become more knowledgeable or better at reasoning. Nor should a peak result become an assumed speed increase for every customer request.
The billing model changes the calculation
Together’s dedicated inference overview distinguishes serverless billing per token from dedicated deployments billed per GPU-hour. Dedicated endpoints reserve compute resources, while serverless uses shared capacity. The page also describes shadow traffic and A/B testing for evaluating candidate deployments.
For a buyer, those billing differences affect how an efficiency gain could turn into savings. With hourly infrastructure, completing more useful work on the same provisioned hardware can improve utilization. With token-based billing, faster execution alone does not determine the price charged for those tokens.
Consider a team generating internal reports overnight. Its priority might be completing the queue before employees arrive. A customer-facing assistant has a different concern: how long someone waits for an answer during the busiest period. Both teams need performance data, but they should set different acceptance criteria.
What to measure before adopting it
Start with a fixed collection of representative requests and record the current result. Include short conversations, long documents and busy periods if those occur in your application. Keep the model version and generation settings documented so a configuration change does not quietly become a comparison between different systems.
Measure time to the first token, time to a complete answer, error rates and total cost. Look at slower requests as well as the average. A system can appear responsive most of the time while leaving a smaller group of users waiting much longer.
Check output quality alongside speed. For structured extraction, that could mean valid fields and correct values. For coding, use the same tests on generated changes. Agree on a rollback threshold before shifting live traffic, and retain the previous configuration until the new one passes those checks.
ByteForward’s recommendation is to treat this as a workload-specific infrastructure evaluation. Ask for a measured trial with a defined budget and success criteria. Adopt it when the results improve your service or operating economics, rather than on the strength of the largest reported multiplier.
