What do air-gapped models actually require?

Air-gapped models require much more than a GPU server with its network cable removed. The useful definition is a system whose inference data, model artifacts, administration path, logs, and update process cannot cross the security boundary except through an explicit, inspected transfer procedure. If an engineer can restore internet access for a package install, or a management controller still calls a vendor cloud, the design has a gap in the ordinary sense, not an air gap in the security sense.

That distinction changes the purchase. You are taking responsibility for hardware capacity, model custody, software dependencies, identity, observability, failure recovery, and every future update. You also accept that some hosted capabilities cannot be brought inside at any price because the provider does not distribute its weights. The result can be the right answer for source code, production records, export-controlled material, or regulated workloads, but only when the operating model is designed before the servers arrive.

An air gap is a governed data-flow boundary

A machine is air-gapped only when every path across its boundary is either absent or controlled as a transfer event. Teams often inspect the application network and forget the baseboard management controller, hypervisor management plane, storage replication, DNS, time synchronization, telemetry exporters, license checks, crash reporters, and the laptop an administrator uses on both sides. Each one can turn a supposedly closed environment into a partly connected environment.

Start with four flows: input data, output data, software supply, and administration. Draw each source and destination. Mark the protocol, the identity used, the party allowed to approve it, and the evidence retained. A diagram that says "offline cluster" tells an assessor almost nothing. A flow inventory that says "signed release bundle enters through transfer station T1 after two-person approval" describes a control that engineers can build and test.

There are several defensible isolation levels, and calling all of them air-gapped creates bad decisions. A private subnet with blocked outbound traffic still depends on connected routers, cloud control planes, and identity services. A disconnected environment may accept scheduled imports through a guarded gateway. A physical air gap has no live network path and moves approved artifacts on controlled media or through a purpose-built one-way mechanism. Pick the level from the threat model, then use the accurate name in contracts and architecture reviews.

NIST SP 800-53 treats media protection, media transport, boundary protection, configuration management, and audit as separate control families. That separation is useful. Removing a route does not answer who may carry an update, how the media is scanned, whether the receiving side verifies it, or how administrators prove what changed. An air gap needs all of those controls working together.

Test the claim instead of trusting the diagram. Inventory network interfaces and radios, trace switch ports, inspect management controllers, attempt DNS and outbound connections from every workload namespace, and review where logs go. Repeat those checks after maintenance, because temporary diagnostic access has a habit of becoming permanent infrastructure.

The threat model decides what must stay inside

The boundary should contain the assets and operations whose disclosure or external dependence you cannot accept. That sounds obvious, yet many deployments put inference inside while prompts are prepared in a connected service, results are copied to a hosted ticketing system, or traces containing source code leave through observability. The GPU did its work locally, but the workload did not.

Separate three planes. The data plane carries prompts, retrieved documents, model responses, embeddings, and tool results. The control plane handles user identity, scheduling, policy, secrets, logging, and administration. The supply plane brings in weights, containers, operating-system packages, drivers, firmware, and vulnerability information. Closing only the data plane leaves two broad paths for compromise or leakage.

Write a threat model with named adversaries and failure cases. A regulated records team may care most about accidental disclosure and provable custody. A defense program may also assume a capable supply-chain attacker. A factory may care about continued operation when external links fail. These needs produce different transfer rules, redundancy, and review depth. "Security requires it" is too vague to choose hardware or approve an exception.

Decide which outputs may leave. A transformed source tree can contain comments, credentials, customer names, and logic that was present in the input. A model answer is not automatically sanitized because it is newly generated. If results cross the boundary, treat that as an export with content checks, an approver, a destination, and a record. The same rule applies to support bundles: stack traces and prompt captures often contain the material the gap was meant to protect.

The awkward question is whether people defeat the design. If operators copy error messages onto personal phones, retype commands from connected chat, or move arbitrary USB devices between zones, the formal network boundary only shifts the leak path. Give them an internal documentation mirror, searchable runbooks, approved transfer media, and a support procedure that works under pressure. Controls that make recovery impossible will be bypassed during the first serious outage.

Hardware sizing begins with the workload, not the model card

Size an isolated inference cluster against measured requests, concurrency, context length, latency, and availability, then select the model and precision that fit. Buying enough GPU memory to load the weights proves only that one process may start. Production also needs memory for the key-value cache, activations, runtime workspaces, concurrent execution contexts, and the serving stack.

A first-pass weight estimate is simple:

weight_bytes ~= parameter_count * bits_per_weight / 8
required_vram = weights + kv_cache + activations + runtime_workspace + safety_margin

The first line is a lower-bound estimate, not a quote. Quantized formats carry scales and metadata. Some architectures activate only part of their parameters per token but may still need all experts resident. NVIDIA's TensorRT documentation makes the practical point that a serialized engine approximates weight memory, while execution contexts add persistent and enqueue memory. It also advises measuring free device memory and setting explicit workspace limits. That is better guidance than multiplying parameter count and ordering the nearest GPU.

Benchmark the actual serving build on the exact hardware. Use representative input and output lengths, including the long requests that dominate cache use. Measure time to first token, tokens per second, queue delay, peak device memory, host memory, storage reads at startup, and recovery after a worker fails. Run enough concurrent requests to expose scheduler behavior. A single interactive prompt hides the capacity problem.

CPU RAM and storage matter too. You may need room for current and previous weights, an unpacked staging copy, container layers, engine caches, evaluation sets, and audit logs. If a rollback requires deleting the only known-good model to make space, storage was undersized. Fast local storage can shorten restarts, while a slow shared store can make every node return to service at once and then wait on the same bottleneck.

Multi-GPU execution solves a fit problem or a latency problem, but it adds communication and another failure domain. NVIDIA documents that splitting execution reduces per-device memory pressure at the cost of inter-GPU communication. Validate the topology, peer-to-peer path, and collective libraries with the same driver and firmware versions planned for production. Two GPUs listed in a bill of materials do not guarantee useful tensor parallel performance.

Availability multiplies the requirement. If maintenance or a hardware fault cannot stop inference, provision capacity for the largest allowed failure while still meeting latency targets. That may mean a spare node, spare GPU capacity across nodes, redundant internal registries, and replacement parts held on site. A hosted provider hides much of that reserve in its service price. Inside the perimeter, you own it.

Capacity testing must cover power and thermal limits, not just software throughput. A dense accelerator node can pass a short benchmark and throttle during a sustained run because the rack cannot remove heat or a power feed cannot support every device at full load. Record clocks, temperature, corrected hardware errors, and power draw during a soak test. Ask facilities what happens when one feed or cooling unit fails, then repeat the performance calculation under that reduced condition.

Qualify a complete hardware and software baseline: server firmware, management-controller firmware, GPU firmware, driver, runtime, kernel, container image, and inference engine. An optimized engine may be built for a particular GPU generation or software combination, so importing a prebuilt engine without reproducing its target can create a startup failure inside the zone. Keep the build process available internally, or promote an engine only after it has run on the production hardware class.

Do not assume CPU inference provides a useful emergency mode. It may keep a small model alive, but latency and memory bandwidth can make the main workload miss every service objective. Measure it and label it accurately: degraded service, batch-only recovery, or no fallback. The same honesty applies to mixed accelerator fleets. Different devices can require different engines and can make scheduling and spare capacity harder than the purchase discount suggests.

Finally, reserve a qualification lane. If all GPUs carry production traffic, every driver, runtime, and weight update must be tested either on unlike hardware or directly in the live pool. A small representative node can validate imports, rebuild engines, run evaluation, and expose compatibility failures before promotion. It costs capacity, but so does discovering after an emergency patch that the only internal compiler image is missing a dependency.

Weights need custody, licenses, and reproducible identity

Model weights are executable inputs with legal terms, security consequences, and a precise identity. Treating them as a large file copied from a workstation loses the information needed to reproduce or investigate a deployment. The release record should bind the weight files to the model configuration, tokenizer, inference runtime, adapters, container images, licenses, and evaluation result.

Use immutable digests, not mutable names such as latest or a bare model label. The Open Container Initiative Image Specification requires descriptors to carry a content digest and size, and recommends verifying retrieved bytes before use. That principle applies even if weights travel as ordinary files. A signed manifest should list every artifact by SHA-256, its byte count, its source, the license reviewed, and the approval ticket.

A minimal bundle can be checked with tools present in almost any controlled build environment:

$ sha256sum -c SHA256SUMS
weights/model-00001-of-00004.safetensors: OK
weights/model-00002-of-00004.safetensors: OK
weights/model-00003-of-00004.safetensors: OK
weights/model-00004-of-00004.safetensors: OK
config/tokenizer.json: OK
images/inference-server.oci.tar: OK

$ sha256sum SHA256SUMS
4b7f...a921  SHA256SUMS

The shortened digest above shows the output shape, not a value to copy. In production, record the complete checksum, verify the signature on the manifest using a public key already trusted inside the perimeter, then recalculate every file digest after transfer. A checksum protects integrity only if the expected checksum arrived through a trusted and authenticated path. Putting a malicious file and its matching checksum on the same unreviewed drive proves nothing.

Keep the original bundle after promotion. If an output changes, you need to know whether the cause was a weight revision, tokenizer change, runtime rebuild, driver update, sampling configuration, or application code. Version the whole inference unit and make rollback an ordinary deployment operation. Never rebuild an old release from floating dependencies during an incident.

Licensing can block a deployment even when the files are technically downloadable. Review rights to run, modify, redistribute internally, create derivatives, and use outputs for the intended purpose. Record any acceptable-use terms that affect the workload. A model described casually as open source may use a license that does not meet your organization's definition of open source, so security approval does not substitute for legal review.

The update path is part of the production system

An air-gapped environment needs a designed import pipeline because models depend on a changing software stack. Drivers, firmware, runtimes, base images, Python or system packages, model weights, tokenizer files, vulnerability advisories, and revocation data all age. Freezing them forever avoids transfer work but accumulates defects and makes the eventual jump harder to test.

Use two staging zones. A connected acquisition zone fetches pinned artifacts and records provenance. A transfer station scans, verifies, inventories, and packages the release without holding production secrets. The receiving zone verifies the signed manifest again, imports artifacts into internal repositories, runs acceptance tests, and promotes by immutable digest. Do not let production pull directly from the transfer device.

Red Hat's documentation for disconnected OpenShift environments treats mirroring and disconnected updates as continuing operations, not installation tricks. That is the right mental model even outside OpenShift. Mirror the repositories your deployment consumes, retain release metadata, and practice upgrades and rollbacks without contacting public infrastructure. If a package manager silently reaches an external index during a rebuild, the bundle is incomplete.

Set different cadences for routine and emergency changes. Routine releases can bundle model, runtime, and operating-system updates after evaluation. An actively exploited driver or library flaw may need a narrow emergency bundle. Define who can declare that path, which tests may be shortened, how risk is accepted, and when the complete test suite catches up. Otherwise every urgent patch becomes an improvised policy debate.

For regulated legacy rewrites, CodeHero supplies the models and can run them air-gapped inside the customer perimeter on customer hardware or hardware it rents to the customer. That arrangement still requires the customer and delivery team to agree on transfer authority, physical access, logs, export review, and the final disposition of weights and hardware.

Plan removal as carefully as import. Expired weights, failed drives, transfer media, log archives, and rented hardware may contain protected data or model assets. Define sanitization and destruction evidence before decommissioning. A closed perimeter with an undocumented exit process merely postpones the custody problem.

Offline operations need their own dependencies

The cluster must keep working when every public convenience is unavailable. That includes identity, time, name resolution, certificates, package repositories, monitoring, alerting, documentation, backups, and support tooling. A model server that runs while its users cannot authenticate is not an available service.

Identity is often the first hidden dependency. If the organization uses a cloud identity provider, decide whether the closed zone gets an independent directory, a replicated subset through a controlled mechanism, or locally managed accounts. Plan how accounts are created, disabled, and reviewed. Cache duration is not an identity architecture, especially when a terminated administrator retains access until a disconnected token expires.

Time and certificates fail more quietly. Run an internal authoritative time source and document its own synchronization and drift checks. Operate an internal certificate authority or import certificates with a renewal process that starts well before expiry. Test what the inference gateway, registry, monitoring stack, and automation do when a certificate expires. Emergency clock changes can invalidate logs and signatures, so do not make manual time adjustment the normal repair.

Observability stays inside unless an approved export says otherwise. Collect request identifiers, latency, queue depth, token counts, memory pressure, hardware errors, model version, policy decisions, and operator actions. Avoid logging full prompts and outputs by default. When content capture is needed for debugging or parity tests, restrict it by role, retention, and case, because the log store can become a concentrated copy of the protected workload.

Build an internal source of truth for runbooks and known issues. Vendor support may ask for diagnostics that cannot leave. Agree in advance whether support staff can enter the perimeter, whether a sanitized bundle can exit, and which commands the customer will run. Rehearse a failed GPU, corrupted model bundle, registry outage, certificate expiry, and rollback. An isolation design earns trust during recovery, not during the architecture presentation.

Backups need independent restore tests. Keep configuration, manifests, internal repository metadata, policy, and any stateful application data, but do not assume weights must be backed up in the same system if signed release bundles already provide a recoverable source. Restore into a clean segment and prove that no external download is needed.

Closed models set a hard capability ceiling

You cannot self-host a model whose provider does not release the weights and a usable license. This is the clearest trade-off against hosted inference, and procurement cannot negotiate around physics or missing artifacts. A smaller distributable model may be good enough for classification, extraction, code transformation, or retrieval-assisted tasks, but it is not automatically equivalent to the strongest hosted model on broad reasoning or uncommon languages.

Hosted services also concentrate engineering work that the closed deployment must reproduce: optimized kernels, request batching, autoscaling, model routing, abuse controls, tool execution, multimodal preprocessing, structured-output handling, and frequent model refreshes. Some features can be rebuilt inside. Others depend on proprietary models or provider-side systems and disappear. List each required capability and test it; do not accept a general claim that an on-premises model is "the same."

Context length on a model card does not prove useful performance at that length. Long contexts consume cache memory, reduce concurrency, and may produce worse task results even when the request fits. Test the real documents and codebase shape. For a million-line repository, the practical question is not whether one prompt can contain it. The system needs a method to analyze the whole tree, preserve relationships, and verify changes against observed behavior.

Updates arrive slower by design. A hosted endpoint can change behind a stable API, for better or worse. Inside the perimeter, every new weight or runtime follows acquisition, review, transfer, evaluation, and promotion. That delay buys control and reproducibility, but it means newly released features and security fixes are not immediate. Decide which delay is acceptable for routine improvements and which flaws trigger the emergency path.

External tools are another dividing line. Web search, hosted code repositories, SaaS ticketing, public package metadata, and provider-managed retrieval are unavailable unless mirrored or exposed through an approved gateway. A workflow built around live tool calls will lose capability when moved inside. Replace each dependency with internal data and APIs, or state plainly that the feature will not exist.

Quality evaluation must be task-specific. Keep a versioned set of representative inputs, expected invariants, prohibited disclosures, latency limits, and human scoring guidance inside the perimeter. Compare candidate releases with the incumbent before promotion. Generic public benchmarks may help shortlist models, but they do not tell you whether a rewrite preserves month-end accounting behavior or whether an extraction pipeline handles your worst forms.

Hosted and isolated costs behave differently

Hosted inference turns much of the platform into variable spend, while an isolated deployment concentrates cost in reserved capacity and operations. Comparing a token price with a GPU purchase price omits most of both systems. Model the service that meets the same availability, latency, context, security, and support requirement.

For the closed side, include GPU nodes, CPU and RAM, fast storage, internal network fabric, rack space, power, cooling, spare capacity, replacement parts, secure transfer equipment, internal registries, monitoring, backup, and staff time. Add acquisition lead time and the cost of capacity that sits idle but exists for a peak or failure. If the environment needs separate development, test, and production zones, count each one. Hardware rented for the project remains dedicated capacity, not hosted token billing.

For the hosted side, include input and output usage, cached-input rules, provisioned throughput, data retention settings, private connectivity, gateway and logging services, evaluation traffic, retries, and engineering work to manage model changes. Add the cost of redaction or data minimization if raw material cannot be sent. A cheap endpoint that legal or security will not approve has no useful unit economics.

A simple comparison is enough to expose assumptions:

closed_annual_cost = annualized_hardware + facilities + licenses + operations + transfer_and_assurance
hosted_annual_cost = request_volume * blended_request_cost + connectivity + operations + assurance
cost_per_success = total_cost / accepted_task_outputs

The last line matters most. Tokens per second and cost per token can reward a fast system that produces work people reject. Define an accepted output for the task, such as a transformation that passes parity tests or an extraction that meets the review threshold, then measure retries and human correction. Use sensitivity ranges for utilization, request growth, hardware life, and staffing instead of pretending that one forecast is exact.

Isolation tends to make economic sense when the workload is steady, the hardware stays busy, the chosen distributable model performs well, and the data constraint already demands local operations. Hosted inference tends to win when demand is bursty, the latest proprietary capability changes the result, global capacity matters, or the organization does not want to operate accelerators. Hybrid designs can route approved low-sensitivity work outside, but they require reliable classification and must fail closed when data sensitivity is uncertain.

Approval should depend on evidence, not the label

Approve an air-gapped model deployment when the team can demonstrate the boundary, reproduce the inference unit, operate every dependency offline, update and roll back through a tested path, and show that the selected model meets the actual task. Reject proposals that offer only a disconnected GPU and a promise to solve updates later. That is an experiment, not a production design.

Ask for evidence you can inspect: the data-flow inventory, threat model, hardware benchmark, capacity calculation, signed artifact manifest, license record, evaluation report, recovery runbook, transfer log, vulnerability intake process, and decommissioning plan. Then observe a release import and rollback. Paper controls often fail at the exact handoff between the connected acquisition team and the operators inside.

Run a failure review before purchase. What happens when one GPU dies, an administrator loses access, the internal registry is corrupt, a certificate expires, a model produces a serious regression, or a severe runtime vulnerability is announced? Name the recovery time the business can tolerate and the people authorized to act. If the answer requires reconnecting the cluster, budget and approve that exception now or redesign the dependency.

Do not demand isolation as a status symbol. If private hosted inference with contractual controls satisfies the threat model, it may deliver better models with less operational risk. If source, records, or policy cannot cross the boundary, accept the capability and staffing costs rather than hiding them. CodeHero completes each legacy rewrite in under 30 days, so the perimeter, hardware, and transfer decisions must be settled early enough to support that delivery window.

The decisive test is operational: disconnect every external path, introduce a signed update, run the representative workload, export an approved result, roll back, and recover from a failed node. If the team can perform that sequence from its own repositories and leave a complete audit trail, the air gap is real enough to carry production work.