Private LLM deployment: the cost model nobody publishes

A private LLM costs whatever your GPUs cost divided by the tokens they actually serve, so utilisation decides the answer. At published September 2026 rates, self-hosting an open-weight model only beats a managed endpoint at high sustained throughput. Most teams should self-host for data residency or a specialised fine-tune, rarely for price.

Why the usual self-host versus API comparison is wrong

The standard comparison takes a per-million-token API price, takes a GPU hourly rate, divides the hourly rate by an assumed token throughput, and declares a winner. Every term in that calculation is defensible. The calculation is still wrong, because it compares a variable cost against a fixed cost as though both behaved the same way.

An API price is consumption-based. If your traffic halves next month, the bill halves. A rented GPU is a standing charge. It costs the same whether it is saturated or idle, and idle is the normal state for a private LLM serving an internal workforce that goes home at six. The self-hosted number in most comparisons is the number you would pay if the hardware were busy every second of every hour, which is not a deployment anyone has ever run.

There are three further omissions, and each of them moves the result by more than the GPU line does.

Redundancy is the first. A single self-hosted LLM instance behind a production workflow is an outage waiting for a driver update. Availability needs at least two replicas, usually in different availability zones, which doubles the fixed cost without doubling the tokens served.

Operational headcount is the second, and in our delivery experience it is the largest line in the model for any deployment under roughly ten GPUs. Someone has to own the inference stack: CUDA and driver versions, vLLM upgrades that change scheduler behaviour, out-of-memory incidents at 2am when a user pastes a 90,000-token document, capacity planning, and the quarterly argument about whether to move to newer hardware. That person exists whether or not you put them in the spreadsheet.

Throughput is the third. Tokens per second per GPU is not a property of the GPU. It is a property of the GPU, the model, the quantisation, the batch size, the sequence length distribution and the KV cache pressure, and it moves by an order of magnitude across those variables. A cost model that treats it as a constant has hidden its biggest uncertainty inside its most confident-looking number.

The four prices in the private LLM market, and what each one buys

People say "self-host or use an API" as though there were two options. There are four, and the middle two are where most sensible answers land. All prices below are the providers' own published list prices, checked on 12 September 2026.

Two things in that table are worth stopping on.

The same open-weight model carries a 6.7x price spread between two managed providers. Llama 3.3 70B works out at $0.155 per million tokens blended at DeepInfra and $1.04 blended at Together, on a 3:1 input-to-output mix. Identical weights, identical licence, same class of hardware underneath. Whatever you are buying from an inference provider, it is not the model. It is their utilisation, their batching, their capacity contracts and their appetite for margin.

The hyperscaler premium is real but not the whole story. An H100 on AWS on-demand works out at $6.88 per GPU-hour against $3.49 at RunPod's secure tier, roughly double, on prices checked 12 September 2026. RunPod publishes its per-GPU rates openly; AWS does not publish GPU instance rates on its on-demand landing page, and the p5.48xlarge figure here comes from the Vantage price tracker's reading of the AWS price list, updated 12 September 2026. What you get for the premium is a VPC you already have, IAM you already run, a network path to data that never leaves your account, and a support contract. For a regulated buyer those are the product.

What does a private LLM actually cost per million tokens?

Here is the whole model in one line. Everything else in this article is an argument about what to put in it.

Cost per million tokens = (G x N x R x 730 + Ops) / (T x N x R x U x 3,600 x 730 / 1,000,000)

Where:

• G is the GPU price per GPU-hour, in US dollars
• N is the number of GPUs one replica of the model needs
• R is the number of replicas you run
• T is sustained output tokens per second per GPU, at your prompt and batch profile
• U is sustained utilisation, as a fraction of wall-clock time the cluster is actually generating
• Ops is the fully loaded monthly cost of the people who own the stack
• 730 is hours in an average month

Drop the Ops term and the replicas cancel, which collapses it to something you can do in your head:

Cost per million tokens = G x 1,000,000 / (T x 3,600 x U)

That collapse is the useful part. Buying more GPUs does not change your unit cost. Only three things do: the hourly price of the hardware, how many tokens per second each GPU sustains, and what fraction of the time it is doing that. Almost everything a self-hosting vendor sells you on, and almost everything an engineering team argues about, is an attempt to move T or U.

The Ops term does not cancel. It is fixed against the whole cluster, so its contribution per token falls as the cluster grows. That is the actual economy of scale in self-hosted inference, and it is why a private LLM at ten GPUs can be sensible when the same architecture at two GPUs is not.

Why does utilisation decide everything?

Take the collapsed formula and hold the GPU price at $3.49 per GPU-hour, RunPod's published secure-tier H100 SXM rate on 12 September 2026. The table below is arithmetic, not a benchmark. It is what the formula produces at each combination of sustained per-GPU throughput and sustained utilisation. The throughput column is a range on purpose, because it is the one number you have to measure rather than read.

Cost per million tokens, in US dollars, derived from G = $3.49 per GPU-hour. Hardware only: no people, no storage, no egress.

The spread across that table is 250x, and the model, the vendor and the hardware never changed. Utilisation alone moves the answer by 9x within any single row.

Now ask what utilisation you will actually achieve. An internal assistant used by staff in one or two time zones has demand concentrated into maybe nine hours on five days, which is 27% of wall-clock time before you account for the fact that demand inside those hours is bursty and you must size for the peak rather than the mean. Sizing for a 4:1 peak-to-mean ratio inside the working window puts sustained utilisation nearer 7%. Customer-facing traffic across three continents does better. A batch pipeline that can run overnight does much better, because it backfills the idle hours, and this is the single highest-return change available to most private LLM deployments: find the asynchronous work, and schedule it into the trough.

The uncomfortable implication is that an inference provider's advantage over you is neither hardware nor engineering skill. It is that they aggregate thousands of bursty tenants onto the same fleet and run it near saturation. You are buying their utilisation. That is the product.

The worked model, with every assumption labelled

Below is a complete monthly model for a small private LLM deployment. Every input is an assumption, and each one is labelled with why it was chosen, so you can replace it with your own number rather than arguing with mine.

At those values the cluster is 4 GPUs. Fixed monthly cost is 4 x $3.49 x 730 = $10,190.80 of hardware plus $8,333 of people, so $18,524 a month before a single token is served.

Now the part the usual comparisons skip: the break-even is not a number, it is a curve. Here is the same fixed cluster at different sustained request rates, assuming 500 output tokens per request.

Assumptions: 4x H100 at $3.49 per GPU-hour, 730 hours a month, 500 output tokens per request, $8,333 a month of operational headcount. The cluster is held fixed on purpose, to isolate utilisation from capacity. The right-hand column is a published list price, not a modelled one.

Two readings of that table, and they point in opposite directions, which is why the curve matters more than any single break-even figure.

The cluster only reaches 8 requests per second if each GPU sustains 1,000 output tokens per second. At 8 requests per second, hardware-only cost is $0.97 per million tokens, six times the managed price. Add the half engineer and it is $1.76, eleven times. A four-GPU private LLM serving a plausible internal workload does not come close to beating a managed open-weight endpoint on price, and no amount of negotiating the GPU rate closes an 11x gap.

Push the cluster to 64 requests per second, which requires sustaining 8,000 output tokens per second per GPU, and hardware-only cost falls to $0.12, finally under the $0.155 managed price. Add the people back and it is $0.22, still above. The crossing point sits in a regime that needs both near-ceiling per-GPU throughput and near-continuous saturation, which is precisely the regime a managed provider operates in and you probably do not.

We are aware this arithmetic argues against work we sell. It is what the published prices say in September 2026, and a cost model that only ever recommends the expensive option is marketing with a spreadsheet attached. Our generative AI engineering practice runs this model before scoping anything, and it ends in a managed endpoint more often than it ends in a cluster.

CTA, mid-article. If you are weighing a private LLM against an API right now, send us your traffic profile and token mix and we will run this model with your numbers, including the case where the answer is that you should not build it. Book a working session.

One GPU is not a deployment

Every published comparison prices one GPU. Almost nobody runs one.

A single replica means a host failure, a kernel panic, a bad driver upgrade or a routine node drain takes your private LLM offline entirely. If the model sits behind an internal search tool, that is annoying. If it sits inside a customer-facing workflow it is an incident, and the workflow needs a fallback model on a managed endpoint anyway, which means you are paying for both.

Two replicas is the working minimum, and if the model is load-bearing you want them in separate availability zones, which adds cross-zone network cost and a slightly harder deployment. Some teams run active-passive instead, keeping a warm standby. That halves effective utilisation for the same hardware bill, which in the table above is the difference between $3.88 and $7.76 per million tokens.

Cold start is why the standby has to stay warm. Loading a 70B open-weight model from object storage into GPU memory takes minutes, not seconds, and autoscaling that must pull weights before it can serve is not autoscaling in any sense a user would recognise. That is also why "just scale to zero overnight" solves less than it appears to: the moment the first morning request arrives, somebody waits several minutes for it. Scale to zero works for batch, not for interactive.

In a cost model the honest version of redundancy is a multiplier on fixed cost with no matching multiplier on tokens served, and it is roughly 2x.

Who owns the inference stack on a Tuesday night?

This is the line nobody publishes, and in small deployments it is larger than the hardware.

A managed API has an operational cost of approximately zero. Somebody owns the client library and the retry logic. A self-hosted stack has a real, recurring, named owner, and the work is not glamorous.

Driver and CUDA version compatibility with your serving framework. vLLM and TensorRT-LLM both move quickly, and an upgrade that improves throughput can change scheduler behaviour under load in ways staging traffic will not reveal. Out-of-memory incidents caused by a user pasting something enormous into a context window your KV cache budget did not anticipate. Capacity planning against a demand curve nobody has modelled. Quantisation regressions, where an INT4 build that scored fine on your eval set behaves worse on one customer's document format. Guardrails and prompt injection defences that a frontier provider ships by default and you now own. Model card review and licence compliance every time somebody wants to swap in a newer open-weight model.

In our own delivery work the pattern is consistent. The first three months of a self-hosted deployment need close to a full engineer, and steady state needs somewhere between a fifth and a half of one, depending on how much of the stack is managed for you. We used half an engineer in the model above. If your organisation cannot name that person before the project starts, the deployment has no owner and the cost model is fiction.

There is a legitimate counter-argument. If you already run a platform team with GPU experience, the marginal operational cost of one more service on the fleet is genuinely small, and the Ops term should reflect the margin rather than the average. That is a real and common situation, and one of the cleanest cases for self-hosting. It is also rarer than the number of teams who believe it applies to them.

Why throughput is not a constant

T is the term people treat as a hardware specification. It behaves more like a configuration outcome.

Batch size is the dominant lever. Generation on a modern GPU is memory-bandwidth-bound at small batch sizes, so serving one request at a time wastes most of the card. Continuous batching, which vLLM lists as a core feature alongside PagedAttention and chunked prefill in its own documentation, keeps the batch full by admitting new requests as others finish rather than waiting for a whole batch to complete. The effect on aggregate throughput is large. The effect on individual latency is a trade: a fuller batch raises tokens per second across the fleet and raises time to first token for the person waiting.

KV cache is the constraint that decides how large the batch can get. Every active sequence holds key and value tensors for every token in its context, and that memory competes directly with model weights on the same card. A deployment advertising a 128,000-token context window, with users who occasionally fill it, has a very different batch ceiling from one capped at 8,000 tokens. This is the most common reason a private LLM performs worse in production than in the load test. The load test used short prompts.

Sequence length distribution matters more than mean sequence length. Prefill cost scales with input tokens and decode cost with output tokens, so a workload of long inputs and short outputs (classification, extraction, retrieval-augmented answering over a large retrieved context) has a completely different cost profile from short inputs and long outputs (drafting, code generation). A RAG pipeline pulling twenty chunks from a vector database is mostly prefill, and prefill parallelises better than decode does.

Numerical precision moves T as well, and it moves the fixed floor underneath the cluster at the same time. That interaction is large enough to deserve its own arithmetic, two sections below.

The practical consequence for a cost model: measure T on a replay of your own production traffic, at your own context window, with your own quantisation, on the serving stack you intend to run, at the batch size that keeps time to first token inside whatever the product needs. Any earlier number is a placeholder. We make the same argument in our guide to when private LLM hosting is worth it, which covers serving stacks and GPU sizing in more depth than this piece does.

What a single-user latency test hides

The most common way a private LLM cost model goes wrong is not a bad GPU price. It is a throughput figure measured with one request in flight.

At concurrency one, a decoding GPU spends most of each step reading model weights out of memory in order to produce a single token. Those weights get read whether one sequence is waiting on them or a full batch of them. Continuous batching exists to amortise that read: the scheduler admits arriving requests into the running batch as others finish, so a single decode step emits a token for every sequence in flight at close to the cost of emitting one. Aggregate tokens per second therefore climbs steeply as concurrency rises from one, while the rate any individual user perceives barely moves.

That is the regime your cost model lives in and your latency test never visits. A benchmark with one request in flight measures the worst cost per token the hardware will ever produce and the best latency it will ever produce. Neither is a number you will operate at, and putting the first one into the formula above understates what the hardware can do by a margin you have no way to bound from that test.

The climb does not continue. Two ceilings stop it, and they stop it for different reasons.

The first is KV cache capacity. The number of sequences the scheduler can hold is roughly the free memory on the card divided by the KV bytes one sequence consumes at its context length, and that divisor is set by your traffic rather than by your configuration file. When the blocks run out, the scheduler either queues arrivals or preempts sequences already running and recomputes their cache when they resume. Recomputation is work you pay for twice. Past that point, additional load converts into waiting rather than into tokens, so cost per token stops improving while the queue keeps growing.

The second is the compute-bound regime. Once enough sequences share each weight read, the matrix multiplications themselves become the limit and every further sequence adds real time to the step. Aggregate throughput still rises, sublinearly, and the per-user token rate now falls in a way people notice.

Little's Law does the conversion between the number you forecast and the number that decides your hardware. Concurrency equals arrival rate multiplied by residency time: to serve L requests a second, each occupying the system for W seconds end to end, you must hold L x W of them resident at once. Take your own target rate and your own measured residency, and check the product against your KV budget at the 95th percentile context length rather than the mean. If the budget cannot hold that many sequences, the cluster you sized from the cost table is the wrong cluster, and you will find out under peak load rather than in staging.

Prefill is what decides which ceiling arrives first. A long prompt admitted into a running batch has to be prefilled before it can decode anything, and unless the serving stack chunks that prefill it stalls the decode step every other user is waiting on. Chunked prefill gives back a little aggregate throughput in exchange for a lot of tail latency. Which side of that trade you want is a product decision, and it changes T, which changes the cost per million tokens directly.

So the load test to run is a concurrency sweep, not a latency check.

T for the cost model is the aggregate output rate at the knee divided by the number of GPUs. Not the peak. The peak sits past your latency budget, and throughput you are not allowed to ship is not throughput.

How quantisation moves the break-even

Quantisation is usually discussed as a way to fit a model onto smaller hardware. In a cost model it is a break-even lever, and it pulls on two terms at once.

Storing weights at 8-bit or 4-bit precision instead of BF16 shrinks the weight footprint. That frees memory on the same card for the KV cache, which raises the batch ceiling described above, which raises T, which lowers cost per million tokens at any given utilisation. It can also reduce N, the number of GPUs one replica needs.

N is the more interesting of the two, because in the collapsed formula N cancels. Cost per million tokens at a given utilisation does not care how many GPUs you run. What N changes is the floor. A model needing two cards per replica, run at the two replicas availability demands, cannot cost less than four GPU-hours every hour no matter how little traffic arrives. Fit the same model on one card and that floor halves, from $10,190.80 a month to $5,095.40 at the RunPod rate checked on 12 September 2026. For a deployment with modest, bursty demand, the floor is the bill. Quantisation therefore matters far more to a small private LLM than to a large one: at scale it buys throughput, at small scale it buys the right to exist at all.

The cost is output quality, and the reason nobody can hand you a single number for it is that the number is a product of four things you have not told them.

Which model, because sensitivity to low-precision weights differs by architecture and by how the model was trained. Which method, because GPTQ, AWQ and the various weight-only and activation-quantised formats do different work, fail in different places, and depend on the calibration data used to produce the build. Which task, because a constrained extraction with a short output and a validating schema absorbs a degradation that multi-step reasoning does not, errors in a chain of steps having somewhere to compound. And which threshold, because "acceptable" is a product decision you own rather than a property of the bit depth.

A published quantisation benchmark answers all four of those questions for somebody else. It is evidence that a method works, not evidence about your bill.

Run the decision as a priced comparison rather than a quality check. Freeze an evaluation set from real traffic and grade the incumbent build on it first, so a regression is distinguishable from a bad day. Write down the threshold you will accept per task class before you look at any throughput number. Serve the quantised build under the same sweep as the section above, at the same latency contract, so the T you compare is the T you would ship. Then convert: the measured gain in T becomes a measured fall in cost per million tokens, and you decide whether the measured quality delta is worth that saving on that traffic. Deciding the threshold after seeing the saving is how teams ship a degraded model and describe it as a trade-off.

Directionally, every row that raises T lowers the token volume at which self-hosting overtakes a managed endpoint, and every row that raises N raises the monthly bill you pay before the first request. The magnitudes are yours to measure. Ours would be a guess about your traffic.

The evaluation harness is a cost line

Nobody budgets for re-validation, and on a private deployment it is not optional, because nobody else in the world is running your weights at your precision against your prompts.

The trigger list is longer than most teams expect. A base model upgrade. A quantisation change. A serving-stack upgrade that alters sampling defaults or scheduler behaviour. A system prompt edit. A retrieval index rebuild that changes what lands in the context. A refreshed LoRA adapter. A driver or CUDA bump that arrives underneath you when a node is replaced. Each of those is a new artefact in production, and each needs the same evidence the last one needed.

What the evaluation set has to contain is fairly settled, and building it is a one-off. Real requests sampled from production and stratified by task class rather than chosen by whoever built it. A reference output or a grading rubric for each class. A pass threshold per class, agreed and written down before any run. Long-context and adversarial cases, held separately, because an average score hides the cases that generate complaints. A deterministic runner that pins sampling parameters, the seed where the stack supports one, and the exact build identifiers for weights, quantisation and serving version. Latency and throughput capture inside the same run, so quality and cost deltas are comparable rather than measured on different days. And the incumbent's scores, kept as the baseline.

The cost has a shape rather than a value: E = H_build, amortised, plus S x (H_run + H_adjudicate + H_decide), where S is model swaps per year and each H is engineer-hours. Measure your own from the first swap, because the constants are specific to your grading rubric and your tolerance for disagreement. In our delivery work adjudication dominates, not compute. Running the set is cheap. Deciding what to do about the cases where two builds disagreed and the automatic grader was not confident is a person reading outputs.

Two consequences for the break-even. E raises it, which argues against self-hosting, and it argues most strongly against self-hosting at low volume because E amortises over tokens exactly as Ops does. And E is smaller than it looks in isolation, because an API consumer needs a version of the same harness anyway: providers deprecate models on their own schedule and hand you a migration you did not ask for. What belongs in your cost model is the difference between the two harnesses, not the whole of one.

The compensation is that the harness is the only asset in this entire article that keeps paying. It prices the quantisation decision. It prices a model swap. It prices the exit. Built after the cluster, it is a cost. Built before the cluster, it is the instrument that tells you whether to buy the cluster.

What does it cost to be wrong?

Reversibility has a price, and it is the part of the decision most spreadsheets ignore entirely.

On a managed API, if the product changes direction you stop calling it. The bill goes to zero inside a billing period. If your traffic estimate was five times too high, you pay for what you used. If a better model ships next quarter, you change a string in a config file.

On rented GPUs, that flexibility is what the neocloud premium buys. On-demand GPU rental at $3.49 per GPU-hour is expensive relative to a commitment, and the expense purchases the right to stop tomorrow.

Commitment pricing is where the asymmetry becomes concrete. AWS's own documentation states that Savings Plans and Reserved Instances offer up to 72% off on-demand pricing in exchange for a one-year or three-year commitment, and that once purchased a Reserved Instance cannot be cancelled, only modified, exchanged or sold on the Reserved Instance Marketplace. Applied to GPU capacity, the three-year reserved rate for p5.48xlarge was $23.777 per hour against $55.04 on-demand in us-east-1, per the Vantage price tracker's reading of the AWS price list, updated 12 September 2026. That is a 57% discount, and also a commitment of roughly $625,000 over the term for one eight-GPU node.

Two observations follow, and the second is the one that matters.

The three-year AWS reserved rate works out at $2.97 per GPU-hour, slightly below what a neocloud charges on demand with no commitment at all. Three years of contractual exposure buys a rate you can get by the hour elsewhere.

And hardware moves faster than three years. B200 capacity already sits on the same rate cards as H100 capacity in September 2026, at $6.99 per GPU-hour on Lambda's single-GPU tier and $6.79 on RunPod's secure tier. A three-year commitment to current-generation silicon is a bet that the price per useful token on that silicon will not be undercut by newer parts or by falling managed prices before the term ends. Every generation so far has undercut the last.

The rule we apply is to match commitment length to the confidence interval on the demand forecast. If you cannot defend a traffic number eighteen months out, do not sign a thirty-six month commitment to serve it.

What does it cost to move back to an API?

The exit never appears in a monthly comparison, because it is not a monthly cost. It is an option premium, paid once, at the worst possible time, usually when the deployment has already stopped making sense.

Getting onto a private LLM is a project. Getting off is a project plus the removal of everything that grew around the private endpoint while it was running. The bill breaks into three parts: re-validation, re-integration, and whatever remains of a commitment you cannot cancel. The first is the harness you have already built, which is why building it early cuts the exit cost before you ever need it. The third is contractual and you knew the number when you signed. The second is where teams get hurt, and it is almost entirely self-inflicted.

Re-integration is expensive because of what accumulates against a specific model. Prompts tuned to the quirks of one open-weight build, which do not transfer. Output parsing written against that model's formatting habits, spread across whichever services needed it. Tool and function-calling schemas expressed the way your serving stack expresses them. A tokeniser that counts your text differently from the provider you are moving to, so both your context budgets and your cost forecast change on the day of the cutover. A latency profile that timeouts and interface behaviour were tuned around. Rate limits on the API side that need negotiating before, not during, the migration. And a data-handling review that has to clear the same compliance gate that pushed you private in the first place, which may be the item with the longest lead time on the list.

The decision that keeps all of that cheap costs nothing on day one, and it is a boundary rather than a tool. Application code never calls the serving stack and never imports a vendor SDK. It calls a gateway you own, speaking one internal contract, with the model identity held as configuration. The gateway carries routing, retries, per-route token accounting, prompt versioning and the fallback path. If the serving stack exposes an OpenAI-compatible interface and the gateway normalises on it, changing where inference happens becomes a configuration change plus one harness run. It is the same boundary discipline we argue for between orchestration and the tools underneath it in our comparison of agentic AI and AI agents, applied one layer lower.

There is a cheaper version of the same insurance. Keep a small share of live traffic on a managed endpoint permanently, as the fallback path redundancy requires anyway. It costs what you route through it, which at a few per cent of volume is noise against the GPU line, and it means the exit path is exercised every day rather than theorised in a document. A fallback nobody has ever served real traffic through is a diagram.

When self-hosting is right anyway

Cost is the wrong reason to self-host for most organisations. There are four good reasons, and none of them is financial.

Data residency and contractual restriction is the first, and the most common genuine driver we see in the UK and Germany. If your data cannot cross a jurisdictional boundary, or a customer contract names the sub-processors you may use, an API in another region is not an option regardless of price. That requirement is usually answered by a VPC deployment or a dedicated instance in the right region, which costs less than a fully self-managed cluster and satisfies the same clause.

Air-gapped environments are the second. Defence, parts of the healthcare estate and some industrial control environments have no egress at all. There is no commercial API answer, so the cost model does not apply: there is nothing to compare against.

Regulatory constraint on the model itself is the third. Under the EU AI Act timeline published by the European Commission, obligations for general-purpose AI providers took effect on 2 August 2025, transparency rules for AI systems that interact with people applied from August 2026, and high-risk obligations follow on 2 December 2027, with 2 August 2028 for AI embedded in regulated products. Where you need to pin an exact model version, reproduce an output months later, or evidence exactly what sat in the system prompt for an audit, controlling the weights and the serving stack makes that straightforward. Doing the same across a provider's deprecation schedule is work.

A genuinely specialised fine-tune is the fourth. If LoRA or another parameter-efficient fine-tuning run on your own data produces a model that materially beats a frontier API on your task, and the task carries enough volume to matter, you have a reason that is about capability rather than price. Be strict about the evidence. Most cases we are asked to look at turn out to be retrieval problems, and a better RAG pipeline over a properly tuned vector database beats a fine-tune on both cost and maintenance.

Licence terms belong here rather than in a legal appendix, because they constrain what "open-weight" means in practice. The Llama 4 Community License requires anyone with more than 700 million monthly active users in the preceding calendar month to request a separate licence from Meta, requires prominent display of "Built with Llama", and requires derivative model names to begin with "Llama"; Meta publishes the licence text in full. Apache 2.0 models carry none of that. Qwen3-32B, for example, is published on Hugging Face under Apache 2.0, with a 32,768-token native context extensible to 131,072 using YaRN. If your deployment involves redistribution, white-labelling or a derivative model name, read the licence before the benchmark.

Where model routing changes the arithmetic

The decision is usually framed as one model for everything. It rarely should be.

Most production workloads have a long tail of easy requests and a short head of hard ones. Classification, extraction, short summarisation and routing itself are handled well by small models. Multi-step reasoning and long-context synthesis are not. Sending everything to the most capable model you have is the most expensive possible policy, and sending everything to the cheapest is the fastest route to a quality complaint.

Model routing changes the cost model in a way that often dissolves the self-host versus API question. Route the cheap majority to a small open-weight model, which may well be worth self-hosting because that class of traffic is high-volume, predictable and tolerant of latency, so it holds utilisation up. Route the hard minority to a frontier API, paying a high per-token price on a small share of requests. Keep a fallback model on a managed endpoint so a cluster failure degrades quality instead of causing an outage.

Routing also has a quiet effect on the concurrency arithmetic. Sending short, uniform requests to the self-hosted tier narrows the sequence length distribution on that cluster, which makes the KV budget predictable, which raises the batch ceiling and therefore T. Mixed traffic, where a 90,000-token document lands in the same queue as a two-line classification, is the profile that produces the worst throughput and the least stable latency. A router is a cost control before it is a quality control.

This is where a private LLM most often earns its place: not as the whole system, but as the high-volume tier of a routed one, sitting alongside an API for the cases that need it. The architecture question is the same one we work through for tool-using systems in our guardrails and architecture piece on agentic AI, and it is the pattern our agentic AI development engagements start from. Routing decisions cannot be made at all without observability: per-route token counts, per-route latency, and a quality signal per route, which means instrumenting before optimising.

For conversational deployments the routing tiers usually fall out of the intent taxonomy you already have, and our enterprise conversational AI guide covers how those tiers get defined. The same logic sits underneath our AI chatbot product, where a cheap tier handles the bulk of turns and escalation is explicit rather than accidental.

What we would not do

We would not build a private LLM for cost reasons at under roughly ten sustained requests per second. The arithmetic above does not support it, and the deployments we have seen at that scale spend more on the person maintaining them than on the token bill they replaced.

We would not sign a three-year GPU commitment on the strength of a demand forecast nobody would defend at eighteen months, and we would not sign one at all while on-demand neocloud pricing sits near the reserved rate.

We would not quote a tokens-per-second figure for your workload from anyone's published benchmark, ours included. Measure it on your traffic, at your context window, on the stack you intend to run, at the concurrency where your latency budget runs out.

We would not size a cluster from a single-user latency test, in either direction. It flatters your latency and slanders your cost.

We would not ship a 4-bit build because a published benchmark said the quality loss was small. The threshold gets written down first, on your own task, and the throughput gain is priced against the measured delta rather than assumed to outweigh it.

We would not let application code call a serving stack directly. The endpoint goes behind a gateway you own in the first week, because that is the only decision on this list that costs nothing now and decides what leaving costs later.

We would not treat data residency as a cost problem. If the requirement is real it decides the architecture, and the cost model only tells you how to implement it efficiently.

And we would not present a self-hosted number without the utilisation assumption printed next to it. A cost per million tokens quoted without the utilisation it assumes is not a number. It is a hope.

If you want this run properly, our generative AI solutions team will model your specific case against current published prices, including the operational headcount line, and say plainly which of the four options in the table above you should be buying. Where the answer is a routed architecture rather than a cluster, our AI automation practice and full-stack engineering team build the routing, observability and fallback paths that make it safe to run. Book a free working session and bring your token logs.