Private LLM Hosting: When to Run Models on Your Own Infrastructure

Most teams should start with a hosted LLM API. The economics, the operational simplicity and the pace of model improvement all favour calling someone else's endpoint. That is the honest default, and any vendor who tells you otherwise is selling infrastructure you may not need.

"Most teams" is not "all teams", and "to start" is not "forever". There is a real and growing set of situations where running models on your own infrastructure is the better engineering decision, and a smaller set where it is the only legally viable one. The hard part is telling the two apart without over-building for a compliance fear that does not actually apply, or under-building and discovering at audit time that your data went somewhere it should not have.

This is the practical side of that decision: what the rules actually say, how much GPU memory each model class needs, what three real configurations cost per month, where the token break-even sits, how to migrate off an API without a big-bang cutover, and what you sign up to operationally. Every price below carries the provider, the page and the date we checked it, because GPU and token pricing moves quarterly and a cost article without dates is worthless six weeks after publication.

If you want the argument about whether those numbers justify the build at all, with utilisation modelled as a function of sustained requests per second, that lives in our companion piece on the private LLM cost model. This article assumes you have decided to look seriously and want to know how to do it.

What private LLM hosting actually means

The phrase covers a spectrum, not a single architecture, and the trade-offs differ sharply across it.

At one end is a dedicated or single-tenant deployment of a hosted model, where a provider runs the model but isolates your traffic and data on infrastructure reserved for you. Your data still leaves your network, but it is contractually and technically partitioned. In the middle is self-hosting open-weight models in your own cloud account, running Qwen, Mistral or Llama on GPU instances inside your own VPC, network policy and security groups. At the far end is fully air-gapped, on-premises hosting, where the model runs on hardware you own in a network with no route to the public internet.

Each step towards the air-gapped end buys more control and more isolation, and costs more in operational burden, capital and the speed at which you can adopt newer models. The decision is rarely "API versus on-prem". More often it is which point on this spectrum matches your actual constraints, and the answer is frequently one step less isolated than the first instinct.

A fourth option sits between the first two and gets overlooked: a regional processing endpoint from a frontier provider. OpenAI's pricing documentation, checked on 13 September 2026, states that regional processing endpoints for data residency carry a 10% uplift for models released on or after 5 March 2026 that are eligible for it. Ten per cent is a small price next to a GPU cluster, and for a residency requirement satisfied by processing location rather than by ownership, it can close the question entirely.

Which rules actually force self-hosting, by sector

"Compliance" is the reason most often given for self-hosting and the one most often given vaguely. The useful version names the rule, the obligation it creates, and whether a vendor contract can discharge it. In most cases it can. The cases where it cannot are specific, and they are worth knowing precisely.

Healthcare: the business associate chain under HIPAA

Sending protected health information to a third-party model provider makes that provider a business associate. Under 45 CFR 164.504(e), a covered entity must obtain satisfactory assurances, in a contract, that the business associate will safeguard the information. The part teams miss is 164.504(e)(5): the same requirements flow down to the business associate's own subcontractors, in the same manner as they apply between the covered entity and the business associate.

That flow-down is the real question to ask a model vendor. A signed BAA with the provider is necessary and not sufficient if the provider runs inference on a cloud it does not own, routes overflow traffic to a partner, or passes your prompts through a third-party safety classifier. Ask for the subcontractor list. Major providers offer BAAs and will answer that question, and for a large number of healthcare workloads the answer is satisfactory. Self-hosting becomes the cleaner posture when the chain is long, when you cannot get a complete subcontractor list, or when your own risk committee has decided that PHI does not cross the network boundary regardless of contract.

The same rule explains why a clinical product is usually designed for a controlled environment from the start rather than retrofitted into one, and why what such a system may safely say is settled before any of the hosting decisions here are made. Our AI Clinical Assistant and AI Medical Documentation Bot are built to run inside a boundary the customer controls for exactly this reason: the constraint is structural, not a preference about cost.

Financial services: record-keeping, not data protection

The financial-sector trigger is usually misdiagnosed. It is rarely about confidentiality. It is about whether you can produce the record.

17 CFR 240.17a-4(f) permits records to be preserved electronically, and the amended rule allows an electronic recordkeeping system that maintains a complete time-stamped audit trail of every modification and deletion, with the date and time of each action, as an alternative to the older write-once media requirement. The rule also requires that the firm has at all times filed an undertaking with its designated examining authority, signed by a designated executive officer or a designated third party, to furnish records promptly to the Commission, a self-regulatory organisation or a state securities regulator on reasonable request.

FINRA Rule 4511 sets the retention floor: at least six years for FINRA books and records with no other specified period, in a format and medium complying with SEA Rule 17a-4. In the UK, FCA SYSC 9.1.2 requires a common platform firm to retain records relating to its MiFID business for at least five years, in a medium that keeps them accessible for future reference by the FCA.

Read those together and the LLM implication is concrete. If a model output is a communication with a client, or an input to a recommendation, it is a record. You need the prompt, the response, the model version and the timestamp, retained for six years in the US or five in the UK, produceable on demand, with an audit trail over any change. A hosted API can satisfy all of that if you log it yourself at the application layer, which you should be doing anyway. Self-hosting helps in one narrow way: an open-weight model you have pinned does not get deprecated underneath a record you have to reproduce in year five.

Public sector: procurement and residency

Public-sector buying rarely forbids a hosted API outright. It constrains which suppliers are eligible and where processing happens. In the US that usually means a FedRAMP authorisation at the right impact level, and the practical effect is that your model choice narrows to whatever your authorised cloud offers rather than whatever is best this month. In the EU and UK, the operative constraint is cross-border transfer. Chapter V of the GDPR, at Article 44, makes any transfer of personal data to a third country conditional on an adequacy decision or an appropriate safeguard, and public bodies are typically the least willing to rely on a mechanism that could be challenged later.

Layered on top, the European Commission's AI Act timeline puts general-purpose AI obligations in force from 2 August 2025 and transparency rules from August 2026, with high-risk obligations following on 2 December 2027. None of that requires self-hosting. It does require you to know which model version produced which output and to keep documentation about the system, which is easier to guarantee when the weights are pinned in a registry you control.

The honest reading of that table: in four of six rows a well-contracted API is sufficient, and the work is logging and supplier diligence rather than infrastructure. We say that knowing it argues against selling a build.

The other four reasons to self-host

Compliance is the most cited driver. It is not the only one, and the remaining four are worth stating plainly because they are the ones that survive scrutiny.

Intellectual property is the second. If your prompts and outputs encode proprietary methods, unreleased designs, trading logic or source code, every API call transmits that material to an external party. Reputable providers contractually commit not to train on enterprise traffic, and for most companies that is enough. Some IP is valuable enough, or some threat models adversarial enough, that the only acceptable answer is keeping it inside the building. Semiconductor design, pharmaceutical research and proprietary trading are where we see this bind hardest.

Cost at sustained high volume is the third, and it is the one with an arithmetic answer rather than a judgement. The next section works it.

Latency and offline operation is the fourth. A model on a GPU inside your own network removes the public-internet round trip and the queueing behaviour of a shared endpoint. For interactive applications with tight tail-latency budgets, or for environments that must function with no connectivity at all, a factory floor, a vessel, a secure facility, self-hosting is a requirement rather than an optimisation.

Determinism is the fifth and the most underrated. Hosted models change. Providers update, deprecate and retire versions on their own schedule, and behaviour shifts under you. Anthropic's pricing page, checked 13 September 2026, still lists Opus 4.1, 4.5, 4.6, 4.7 and 4.8, and Sonnet 4.5 and 4.6, alongside the current Sonnet 5 and Opus 5. That is a lot of version churn in a short period, and wanting to opt out of it is a reasonable engineering position. Self-hosted open weights are frozen until you choose to move them, which matters for regulated validation, reproducible evaluation, and any system where a silent behaviour change counts as a defect.

At what token volume does self-hosting turn cheaper?

The arithmetic is simple enough to do in public, so here it is in public.

A self-hosted cluster costs the same every month whether it serves one token or a billion. Call that fixed cost F:

F = G x N x 730

where G is the GPU price per GPU-hour, N is the number of GPUs in the cluster, and 730 is the hours in an average month. An API costs P per million tokens and nothing when idle, so for a monthly volume of V million tokens the API bill is V x P. Setting them equal gives the break-even volume:

V* = (G x N x 730) / P

Take a production cluster of four NVIDIA H100 SXM 80GB cards. RunPod publishes a Secure Cloud rate of $3.49 per GPU-hour for that card, checked 13 September 2026, with a Community Cloud rate of $2.69. Using the secure tier, because production workloads generally cannot run on unvetted hosts:

F = $3.49 x 4 x 730 = $10,190.80 per month

Now the API price. It matters enormously which API you compare against, and this is where most published break-evens quietly cheat by picking the most expensive one. Two comparisons, both at a 3:1 input-to-output token mix, which is an assumption about your prompts that you should replace with your own:

• Claude Sonnet 5, at $2.00 per million input and $10.00 per million output on Anthropic's pricing page checked 13 September 2026, blends to (0.75 x $2.00) + (0.25 x $10.00) = $4.00 per million tokens.
• DeepInfra's managed Llama-3.3-70B-Instruct-Turbo endpoint, at $0.10 per million input and $0.32 per million output on DeepInfra's pricing page checked 13 September 2026, blends to (0.75 x $0.10) + (0.25 x $0.32) = $0.155 per million tokens.

Divide:

• Against Claude Sonnet 5: V* = $10,190.80 / $4.00 = 2,548 million tokens a month, about 2.5 billion.
• Against the managed open-weight endpoint: V* = $10,190.80 / $0.155 = 65,747 million tokens a month, about 65.7 billion.

Those two numbers are twenty-six times apart, from the same cluster, in the same month. That ratio is the most important thing on this page. The break-even for self-hosting is not a property of your infrastructure. It is a property of what you are comparing it to.

Then the sanity check almost nobody runs. Can four H100s actually serve 65.7 billion tokens a month? Output generation is what consumes GPU time; prefill is much cheaper per token. At a 3:1 mix a quarter of that volume is output, so 16.4 billion output tokens across 730 x 3,600 = 2,628,000 seconds, which works out at 6,254 output tokens per second across the cluster, or roughly 1,564 per GPU, sustained continuously, all month. That is at or beyond the ceiling for a 70B-class model on an H100 even with continuous batching, and it assumes no idle hours whatsoever. The break-even against a cheap managed open-weight endpoint is, for a four-GPU cluster, arithmetically real and physically out of reach.

Run the same check on the frontier comparison and it inverts. 2.5 billion blended tokens is 637 million output tokens a month, 242 output tokens per second across four GPUs, 61 per GPU. That is unremarkable. A four-GPU cluster clears the Sonnet 5 break-even without straining.

Prices are the providers' published list rates checked on 13 September 2026. Self-host columns are hardware only, at RunPod Secure Cloud rates of $1.09 per hour for an L40S 48GB and $3.49 per hour for an H100 SXM 80GB, multiplied by 730 hours. The output-tokens-per-second column assumes a 3:1 input-to-output mix and gives the sustained rate the self-hosted option would have to hold to serve that row. Whether your model and hardware reach it is a load test, not a number anyone should hand you.

Three readings of that table, and they do not agree, which is the point.

Against a frontier API, self-hosting crosses early. A single L40S at $795.70 a month beats Claude Sonnet 5 above roughly 199 million blended tokens, which at 1,500 input and 500 output tokens per request is about 99,500 requests a month. That sounds like an easy win and it is a trap, because an 8B open model is not a substitute for Sonnet 5 on most tasks. A break-even between two things you would never actually swap is not a break-even.

Against a managed endpoint running the same open weights, self-hosting almost never wins on price. You are not buying the model from an inference provider; the weights are free and the licence is identical. You are buying their utilisation. They aggregate thousands of bursty tenants onto one fleet and run it near saturation, and you cannot.

The volume at which the sums change is therefore set by quality substitutability, not by tokens. The right comparison is between the cheapest option that passes your evaluation. Run the evaluation set first, find the cheapest thing that clears the bar, then price against that and nothing else.

Talk to us. If you want this run against your traffic profile and token mix rather than our example numbers, our generative AI engineering team will do it, including the outcome where the answer is that you should stay on an API. Book a working session.

How much GPU memory does each model need?

Memory is the first sizing question, not compute. Two things have to fit in VRAM: the weights, which are fixed, and the key-value cache, which grows with the number of concurrent requests and the length of their contexts. The second is what kills deployments that tested fine.

Weight memory is parameters multiplied by bytes per parameter. BF16 is two bytes, FP8 is one, INT4 is half. A 32.8B model at BF16 is 65.6 GB before anything else is loaded.

KV cache per token is also arithmetic, straight out of the model's config file:

KV bytes per token = 2 x layers x KV heads x head dimension x bytes per element

The leading 2 is for the key and the value. For Qwen3-32B, whose the published model config on Hugging Face gives 64 layers, 8 key-value heads and a head dimension of 128, at BF16 that is 2 x 64 x 8 x 128 x 2 = 262,144 bytes per token. A quarter of a megabyte for every token held in cache. Thirty-two concurrent requests at 8,000 tokens of context each means 256,000 resident tokens, so 67.1 GB of KV cache, slightly more than the weights themselves.

Architecture figures come from each model's published config on Hugging Face, read 13 September 2026. Weight totals are parameter count multiplied by bytes per parameter, except where the model card states its own figure. Mistral's card says Mistral-Small-3.2-24B needs about 55 GB of GPU RAM in bf16 or fp16, which is higher than the 48 GB a naive parameter count gives. We have used the card's number, and the gap is a useful reminder that the naive calculation is a floor rather than an estimate.

Two things that table does not tell you, and a spreadsheet will not either.

Usable VRAM is not physical VRAM. vLLM's engine arguments documentation, checked 13 September 2026, gives a default gpu_memory_utilization of 0.92, so an 80 GB H100 offers about 73.6 GB to the model executor unless you change it. Qwen3-32B at BF16 needs 65.6 GB for weights alone, leaving roughly 8 GB of KV cache on a single card, enough for about 30,000 tokens of total resident context. That is four concurrent users at 8k. The model "fits" on one H100 and is useless there.

Attention architecture changes the KV arithmetic more than parameter count does. gpt-oss-120b has 36 layers, 8 KV heads and a head dimension of 64, so its KV cache is 72 KiB per token against Qwen3-32B's 256 KiB, despite being nearly four times the parameters. OpenAI's model card also notes sliding-window attention on alternating layers, which cuts resident cache further in practice. If your workload is long-context and high-concurrency, the attention design deserves more weight in model selection than the benchmark table does.

What does private LLM hosting cost per month?

Three configurations, priced at published list rates checked on 13 September 2026. Every figure is GPU hours multiplied by 730 hours. Nothing below is a quote, a discount or a negotiated rate.

Configuration A: single GPU, one internal tool. One NVIDIA L40S 48GB on RunPod Secure Cloud at $1.09 per GPU-hour is $795.70 a month. It holds Qwen3-8B at BF16 with modest concurrency, or Mistral-Small-3.2-24B at FP8. There is no redundancy: a host drain, a driver upgrade or a kernel panic takes the service offline entirely. This is the right shape for an internal assistant where an hour of downtime is an inconvenience, and the wrong shape for anything customer-facing.

Configuration B: two replicas, production. Four H100 SXM 80GB on RunPod Secure Cloud at $3.49 per GPU-hour is $10,190.80 a month. Two GPUs per replica holds Qwen3-32B at BF16 with roughly 80 GB of headroom for KV cache across the pair, and two replicas mean you survive one host failure and can do a rolling upgrade without a maintenance window. This is the smallest configuration we would put in front of customer traffic.

Configuration C: multi-node. Sixteen H200 141GB on RunPod Secure Cloud at $4.59 per GPU-hour is $53,611.20 a month: two nodes of eight, each node holding Qwen3-235B-A22B at BF16 with about 1,038 GB of usable memory against a 519 GB requirement. At FP8 you could run the same model on four cards per node and halve the bill, at the cost of an evaluation you now have to run. Multi-node tensor parallelism also needs high-bandwidth interconnect between cards, and both RunPod and DeepInfra put their high-bandwidth cluster offerings behind a sales conversation rather than a published rate, so treat any rate-card price for a multi-node cluster as a lower bound.

What those three figures deliberately exclude, named rather than folded into an invented percentage: a network volume large enough for the weights, and a BF16 checkpoint of a 235B model is roughly 470 GB on disk; egress; a gateway and load balancer; observability with per-request token accounting, which you need for the cost comparison to mean anything; a container registry and secrets management; and the people. On people, our companion piece models operational headcount explicitly as a line item and shows how it changes the answer at small cluster sizes, so we will not restate the argument beyond saying it is the line most often omitted and the one that most often flips the decision.

One structural point about commitment. Reserved capacity cuts the hourly rate substantially, and it also narrows your exit. AWS's Reserved Instances documentation, checked 13 September 2026, lists three things you can do with a purchased Reserved Instance you no longer want: modify it, exchange it if it is a Convertible Reserved Instance, or list it for sale on the Reserved Instance Marketplace. Straightforward cancellation is not among them. Signing a three-year GPU commitment to make a break-even work is a bet that your token volume and your model choice both hold for three years, in a market where the frontier price per token has fallen repeatedly. We would not make that bet before a year of production traffic data.

Which serving stack should you run?

Weights alone do not make an efficient endpoint. The inference server does, and the difference between a naive server and a properly batched one changes the cost arithmetic above by an order of magnitude.

A working mental model: Ollama is what you run on a laptop to try a model, vLLM or TGI is what sits behind a real application serving concurrent users, and Triton is what a platform team standardises on when it serves many models of many types. Continuous batching multiplies how many concurrent requests a single GPU sustains, which feeds straight back into the tokens-per-second column in the break-even table.

What quantisation buys and what it costs

Quantisation is the main lever for fitting a larger model onto smaller hardware. Storing weights at lower numerical precision, 8-bit or 4-bit formats such as GPTQ, AWQ and MXFP4, cuts the weight footprint roughly in half at 8-bit and roughly to a quarter at 4-bit relative to BF16. Applied to Configuration C above, FP8 takes Qwen3-235B-A22B from 470 GB of weights to 235 GB and halves the node count, which is $26,805.60 a month of difference at the rates checked on 13 September 2026.

There is a second effect that gets less attention: freeing weight memory frees KV cache memory on the same card, which raises the concurrency ceiling. On a single H100 running Qwen3-32B, moving from BF16 to FP8 takes the weights from 65.6 GB to 32.8 GB and turns roughly 8 GB of available KV cache into roughly 41 GB, a five-fold increase in how many concurrent long-context requests the card holds. The capacity gain is often larger than the memory saving that produced it.

The cost is quality, and the honest position is that nobody can tell you in advance how much. Eight-bit is usually close to lossless for most workloads. Four-bit is sometimes fine and sometimes visibly worse, and which one you get depends on the model and the task rather than on the bit depth alone. Note that gpt-oss-120b ships quantised by design, with MXFP4 applied to the MoE weights, and OpenAI's model card states all its published evaluations were performed at that same precision. That is the shape of a defensible quantisation claim: measured on the quantised artefact, on the task in question. Do the same on your own evaluation set, and treat any generic quantisation benchmark as evidence about someone else's workload.

Security and isolation in a private deployment

Self-hosting changes your security posture rather than simply improving it. You remove third-party access risk and take on full responsibility for securing the deployment, which is not free.

The controls that matter: the model and its endpoint sit inside a private network segment with no public ingress, reachable only through your own authenticated application layer; traffic is encrypted in transit, and anything stored, logs, caches, retrieval indexes, fine-tuning datasets, is encrypted at rest; access is governed by role-based least privilege so the set of people who can reach the model and its data stays deliberately small; and every request and administrative action is logged for audit, which is frequently a compliance requirement in its own right, as the record-keeping rules above make clear.

For genuinely air-gapped deployments the discipline extends to the supply chain. Model weights, container images, CUDA drivers and Python dependencies all have to cross the boundary through a controlled process and be verified on the way in, because there is no live connection to pull updates or patches. This is real operational weight and it belongs in the decision honestly. An air-gapped environment nobody patches is not more secure, it is differently insecure.

Do you need to fine-tune, or is retrieval enough?

A frequent assumption is that private hosting requires fine-tuning on your data. Usually it does not, and reaching for fine-tuning first is a common and expensive mistake.

Retrieval-augmented generation keeps the model fixed and supplies relevant information at query time from a knowledge base you control. It is the right default for the most common need, which is making a model answer accurately about your specific documents, policies, products or records. Retrieval keeps knowledge current because you update the data rather than the model, costs far less to maintain, and keeps proprietary content in a store you can audit and delete from rather than baked into weights you cannot. Deletion matters more than people expect: a GDPR erasure request against a fine-tuned model is a genuinely hard problem, and against a retrieval index it is a row delete.

Fine-tuning changes the weights and earns its keep for a narrower set of goals: teaching a consistent output format, adapting to domain language the base model handles poorly, or improving reliability on a narrow high-volume task. It does not reliably add knowledge the way people expect, and a fine-tuned model still goes stale as your information changes. The pragmatic sequence is retrieval first, measure, then fine-tune only the specific gaps retrieval cannot close. The two compose: a fine-tuned model serving a retrieval pipeline is a normal and good architecture, and it is how we build most generative AI systems that run on private infrastructure.

How do you migrate off an API without a big-bang cutover?

The cutover is where private LLM projects fail, more often than the infrastructure is. The pattern that works runs both systems at once for longer than anyone wants to.

Freeze an evaluation set from production traffic before you change anything. Not a synthetic benchmark. A few hundred real requests, sampled across your actual traffic classes, with outputs from the incumbent API captured and graded. Skip this and you will have no way to tell a regression from a bad day, and every subsequent argument about quality becomes a matter of opinion.

Run the candidate in shadow mode first. Mirror a sample of live requests to the private endpoint, compare offline against the incumbent, serve nothing from it. Shadow mode is where the unglamorous failures surface: a tool-calling schema the open model formats differently, a system prompt tuned against one model's instruction-following that does not transfer, a context length that was fine at the API's limit and overflows yours.

Account for tokens on both paths from day one. Different providers use different tokenisers, so the same text costs a different number of tokens on each side. Compare bills without comparing token counts and you will misattribute a tokeniser difference to a pricing difference, then make the wrong call on the strength of it.

Split traffic by class, not by percentage. A flat 10% split spreads risk evenly across everything, including the workloads where the open model's quality gap is widest. Cut over the classes where the gap is smallest first, summarisation, classification, extraction, structured rewriting, and leave open-ended reasoning on the frontier API until you have evidence. Teams that route by task rather than by coin flip get a better result and a much clearer signal. This is the same routing discipline that sits underneath any production agentic AI architecture, where different steps in a workflow legitimately want different models.

Keep the API path live as a fallback, with an automatic trigger. On timeout, on error, on queue depth above a threshold, requests go back to the managed endpoint. This costs a little and buys the ability to fail without an incident. It also means your private cluster does not have to be sized for peak, which moves the economics in your favour.

Write the rollback condition down before cutover, not during the incident. A named metric, a named threshold, a named person who can call it. If you cannot state it in one sentence, you are not ready to cut over.

Do not sign the reserved-capacity commitment until after the dual-run. You will learn your real token volume and your real utilisation during those weeks, and those two numbers are the inputs to every commitment decision. Buying first and measuring afterwards is the most expensive ordering available.

Plan for the dual-run to last a quarter for anything customer-facing. Throughout that period you pay for both, which is a real cost and belongs in the business case from the start rather than arriving as a surprise in month two.

What you take on operationally

This is the line item nobody budgets, and the reason self-hosting projects disappoint more often than they fail outright.

Who owns the pager

A private LLM has a pager, and someone has to be on it. The failure classes are specific: out-of-memory when KV pressure exceeds what the configured gpu_memory_utilization left you, driver or CUDA version mismatch after a node is replaced, a host drain in the middle of the night on a single-replica deployment, queue saturation with no backpressure so latency climbs until the application times out, and a model rollout that loads cleanly and produces different output.

Draw the ownership boundary explicitly and write it into the runbook. In the split that works, a platform team owns availability, latency and capacity of the endpoint, and the application team owns output quality, prompts and the evaluation set. Ambiguity here produces the worst kind of incident, the one where two teams each believe the other is investigating.

A sustainable rota needs enough people that nobody is permanently on call. If the honest answer is that one engineer would carry it alone, the deployment is not production-ready no matter how good the infrastructure is, and a managed endpoint is the correct engineering decision even when the token arithmetic favours the cluster.

Model and stack updates

The open weights are frozen until you move them. Everything underneath them is not. vLLM ships frequently, and upgrades change defaults; a value such as gpu_memory_utilization moving in a minor release alters your effective KV cache and therefore your concurrency ceiling without any change to your own configuration. CUDA drivers, container base images and Python dependencies all carry CVEs that someone has to track and patch, and in an air-gapped environment each patch is a manual transfer across the boundary.

Budget a regular upgrade window and treat the serving stack as production software with its own release process. The alternative, leaving it pinned because it works, is how a deployment reaches the point where the upgrade path spans three major versions and nobody wants to attempt it.

Evaluation drift

The model does not drift. Your traffic does, and your evaluation set goes stale while the numbers on it stay green.

The mechanism is boring and reliable. You build an evaluation set from traffic in month one. Over the following quarters your product changes, your users learn what the assistant is good at and start asking different things, a new document type enters the corpus. Your evaluation set still passes at 94% because it is measuring the workload you had a year ago. Re-sample every quarter, add the classes of request that now exist, and keep the original set alongside the new one so you can see regressions rather than just a moving number.

The same applies to the retrieval layer. Index freshness, chunking behaviour on new document formats, and embedding model version all change output quality without anyone touching the LLM. When quality complaints arrive, retrieval is the more likely culprit.

Licence obligations

Open weights are not uniformly open. Qwen3-8B, Qwen3-32B, Qwen3-235B-A22B, Mistral-Small-3.2-24B and gpt-oss-120b are all Apache 2.0 according to their published model cards, which is about as unencumbered as it gets. The Llama family is not: the Llama 4 Community Licence requires a separate licence from Meta above 700 million monthly active users in the preceding calendar month, requires prominent display of "Built with Llama", and requires derivative model names to begin with "Llama".

None of that is onerous, and all of it is an obligation someone has to own and re-check at each model change. It belongs in the same review as the security patch and the evaluation re-sample, not in a legal conversation that happens once at procurement and never again.

Running a private model and an API side by side

Many of the teams we work with never pick one. They run a private model for steady baseline load, where the volume justifies fixed hardware, and route spikes to a managed endpoint rather than buying GPUs that sit idle most of the month. That is the same architecture as the migration fallback described above, kept permanently instead of retired.

The cost is complexity: two inference paths to monitor, a routing layer that has to decide, and two sets of evaluation results to keep current. The rule we apply is that the hybrid is worth it only once your baseline load alone would already justify self-hosting. Below that threshold a hybrid setup is a private deployment you did not need plus an API bill you still have.

Routing quality is where the value actually accrues. Sending classification and extraction to a small private model and open-ended reasoning to a frontier API is a bigger cost lever than any hardware decision, because it changes the price per token by an order of magnitude for a large share of traffic. It is also the pattern behind most well-built conversational systems, whether that is a product like our AI Chatbot or a bespoke build, and the wider architecture of that sits in our enterprise conversational AI guide.

A decision framework you can apply this week

Start with the constraints, not the technology. Work through the compliance table above and establish whether any rule genuinely forbids a vendor contract, rather than whether someone is nervous about one. In four of the six rows the answer is a well-drafted contract and better logging.

If nothing forbids it, run your evaluation set across the candidate models, including the cheap managed open-weight endpoints. Find the cheapest option that passes. Price against that, not against the frontier model you happen to be using today. This single step reorders more decisions than any other on the page.

If self-hosting still looks right, size from memory upwards. Weights, then KV cache at your real concurrency and context length, then the serving stack's usable-memory fraction, then a load test before committing capacity. Assume your first sizing is wrong in the KV direction, because it usually is.

Then run both for a quarter before committing to anything you cannot exit. Measure real utilisation rather than projecting from peak, capture token counts on both paths, and only then decide about reserved capacity.

And staff the pager before the cutover, not after. Of everything here, the operational commitment is the part discovered latest and the part that most often turns a working deployment into a regretted one.

Where a workload genuinely needs private infrastructure, that is the work we do, and where it does not we will say so. Our AI automation practice runs this framework against a specific workload rather than in the abstract, and it ends in a managed endpoint more often than it ends in a cluster. The agentic architecture guide covers the guardrail and evaluation layer that sits on top either way. For a straight answer about your own case, including the answer that you should not build it, book a working session.