A technical overview of the architectural decisions behind deploying large language models in air-gapped and sovereignty-constrained environments.
Abstract. This paper describes the architecture of AIC (Artificial Intelligence Core), a platform for deploying large language models within physically isolated infrastructure. We address three challenges that arise when running inference workloads in environments where no data can transit external networks: model delivery without internet access, knowledge retrieval at scale under strict access controls, and maintaining model quality when fine-tuning on limited domain-specific corpora. We present the design choices that emerged from four years of production deployments across government, financial, and healthcare environments, along with the trade-offs those choices entailed. This is not a product brief — it is an honest account of what worked, what required iteration, and what remains unsolved.
The majority of modern AI infrastructure assumes persistent internet connectivity. Foundation model APIs, cloud-hosted fine-tuning pipelines, and retrieval-augmented generation systems all depend on data flowing freely between client environments and centralized compute. For many organizations, this assumption is untenable.
Government agencies operating on classified networks cannot expose operational data to commercial cloud providers. Financial institutions processing proprietary trading strategies face regulatory and competitive constraints on data egress. Healthcare systems handling protected health information must satisfy jurisdictional data residency requirements that cloud deployments cannot always guarantee.
AIC was designed for these environments. The platform enables organizations to deploy, fine-tune, and operate language models entirely within their own physical infrastructure — including environments with no network connectivity to the public internet (commonly referred to as "air-gapped" deployments).
This paper does not attempt to argue that on-premise AI is universally superior to cloud-based alternatives. For many use cases, cloud AI services offer compelling advantages in cost, scalability, and operational simplicity. Rather, we describe the specific architectural decisions required when those services are not an option, and the engineering trade-offs that result.
AIC's architecture was shaped by five hard constraints derived from early deployment requirements. These are not aspirational goals — they are non-negotiable properties that any viable system in this space must satisfy:
AIC is organized into four logical layers, each independently deployable and horizontally scalable. The separation is not purely organizational — it reflects a security boundary. Each layer runs in its own process space with distinct memory protections, and inter-layer communication occurs exclusively through authenticated, encrypted channels.
┌────────────────────────────────────────────────────────────────┐
│ INTERFACE LAYER │
│ Chat UI · REST API · Python/JS SDK · Admin Console │
│ Session management · Input validation · Rate limiting │
├────────────────────────────────────────────────────────────────┤
│ REASONING LAYER │
│ Query planning · RAG orchestration · Citation generation │
│ Guardrails · Output filtering · Classification enforcement │
├────────────────────────────────────────────────────────────────┤
│ MODEL LAYER │
│ Inference engine · Model registry · Quantization runtime │
│ Batch scheduler · KV-cache management · LoRA adapter loading │
├────────────────────────────────────────────────────────────────┤
│ DATA LAYER │
│ Document ingestion · Vector store · Graph database │
│ SQL bridge · RBAC tag index · Embedding pipeline │
└────────────────────────────────────────────────────────────────┘
│ │ │ │
┌────┴────┐ ┌────┴────┐ ┌────┴────┐ ┌────┴────┐
│ Compute │ │ Storage │ │ HSM │ │ Audit │
│ GPU/CPU│ │ NVMe/SAN│ │Key Mgmt │ │ Ledger │
└─────────┘ └─────────┘ └─────────┘ └─────────┘
The data layer is responsible for ingesting, indexing, and serving organizational knowledge. Documents are processed through a pipeline that extracts text, generates embeddings, and tags each chunk with the access control metadata inherited from the source document.
We use 1024-dimensional embeddings (reduced from an initial 2048-dim design after finding marginal retrieval quality improvement beyond 1024 dimensions — see Section 5.2). Embeddings are stored in a custom vector index optimized for approximate nearest-neighbor search on NVMe storage, achieving p99 retrieval latency of 47ms across indices containing up to 12 billion vectors.
The model layer manages inference workloads across heterogeneous GPU hardware. AIC supports models in SafeTensors, GGUF, and ONNX formats, with automatic selection of quantization strategies based on available VRAM.
The inference engine is built around a paged attention implementation similar to vLLM, with modifications for multi-tenant KV-cache isolation. Each user session maintains a separate KV-cache partition, preventing information leakage between concurrent sessions at different classification levels.
| Model | Parameters | Min. Hardware | Throughput | Context |
|---|---|---|---|---|
| ACX-Core-14B | 14.2B | 1× A100 80GB | ~820K tok/s | 128K |
| ACX-Reason-70B | 69.8B | 4× A100 80GB | ~195K tok/s | 128K |
| ACX-Code-34B | 33.7B | 2× A100 80GB | ~410K tok/s | 64K |
Table 1 — Throughput measured on batch inference with 2048-token outputs. Single-request latency varies. Numbers are approximate and depend on quantization settings.
The reasoning layer sits between the model and the user, orchestrating multi-step inference workflows. When a user submits a query, the reasoning layer:
The hallucination detection module deserves specific mention because it is the component we are least confident in. Our current approach uses a secondary verification pass that cross-references generated claims against the retrieved source material. This catches approximately 89% of hallucinated facts in our internal evaluation — a useful but incomplete solution. We continue to invest in this area.
AIC supports three deployment configurations, each with different operational characteristics:
Connected deployment maintains a one-way data diode connection to ACX for receiving model updates and security patches. Telemetry flows outward through a human-reviewed export process — never automatically.
Disconnected deployment operates with no network connectivity. Updates are delivered via encrypted physical media. This is the standard configuration for classified environments.
Hybrid deployment runs on private cloud infrastructure (AWS GovCloud, Azure Government) with customer-managed encryption keys. Data remains within the customer's cloud tenancy but benefits from elastic scaling.
We spent disproportionate engineering effort on the inference engine early in development. In practice, the more difficult operational challenge is getting model weights into air-gapped environments. A 70B parameter model in FP16 is approximately 140GB. Transferring this via approved physical media, with cryptographic verification at every step, through security review processes that were designed for documents rather than binary blobs, is a non-trivial logistical problem. We now maintain a dedicated team for model delivery operations.
Our initial design used 2048-dimensional embeddings based on published research suggesting higher dimensionality improves retrieval quality. In production, we found that 1024-dimensional embeddings achieved 97.3% of the retrieval accuracy at 50% of the storage cost and 40% lower latency. For most enterprise document corpora, the additional dimensions capture noise rather than meaningful semantic distinctions. We now default to 1024 dimensions with an option to increase for specialized corpora (e.g., multilingual legal documents where the additional capacity provides measurable benefit).
Many clients expect to fine-tune models on relatively small domain-specific datasets (10K-100K documents). The results are mixed. LoRA fine-tuning with fewer than 50K high-quality examples reliably improves domain terminology and formatting compliance. Below that threshold, we observe inconsistent results — sometimes marginal improvement, sometimes degraded general capability. We are transparent with clients about these limitations during scoping.
Several challenges remain partially or fully unsolved:
AIC represents one approach to a real and growing need: deploying capable AI systems in environments where data sovereignty is not optional. The architecture described here reflects four years of production experience, including the mistakes and compromises inherent in building complex systems under hard constraints.
We do not claim this is the only viable architecture for sovereign AI. As the field evolves — particularly as model efficiency improves and inference hardware becomes more accessible — the design space will expand. We publish this document in the hope that it contributes useful data points to that broader conversation.
© 2026 ACX Technologies Inc. · ACX-WP-2026-001 · Distribution: Unlimited · Classification: UNCLASSIFIED