How to Deploy a Local LLM Behind a Corporate Firewall or Air-Gapped Environment

by

How to Deploy a Local LLM Behind a Corporate Firewall or Air-Gapped Environment

Public LLM APIs are a non-starter for a lot of enterprise workloads. If your logs contain hostnames, credentials, customer identifiers, or anything regulated, shipping them to a third-party endpoint is a conversation with your security team you don’t want to have. The alternative — running a local LLM behind a corporate firewall, or in a fully air-gapped environment — is entirely practical in 2026, but it comes with real deployment friction: egress-filtered networks, mandatory proxies, and machines that will never touch the internet.

This guide covers the full path for senior DevOps and platform engineers: why on-prem, how to get Ollama working through corporate proxies (including Docker), how to do a strict offline transfer for air-gapped hosts, and the compliance practices that keep the deployment defensible in an audit.

[IMAGE: Network diagram showing a local LLM behind a corporate firewall]

Why Run an LLM On-Premise with No Internet?

The case for a self-hosted LLM enterprise setup rests on three pillars:

  1. Data sovereignty. Prompts and completions never leave your network. There is no third-party data processing agreement to negotiate, no retention policy to trust, and no cross-border transfer question. The inference happens on hardware you control.
  2. Compliance scope reduction. When the model runs on-premise, the LLM workload inherits your existing controls — network segmentation, access management, logging — instead of creating a new external dependency that has to be assessed, contracted, and continuously monitored.
  3. Availability and cost predictability. No API rate limits, no per-token billing surprises, no outage of a vendor’s endpoint taking down your internal tooling. An on-prem model serving internal ops tasks runs at the fixed cost of the hardware underneath it.

Data Privacy, Compliance, and Enterprise Setup Rules

Before deploying, align with your security organization on the ground rules. A typical on-prem LLM compliance setup needs answers to:

  • Where do model weights come from, and how are they verified? Weights are software artifacts. They should enter your environment through the same vetted channel as any other binary — checksummed, scanned, and recorded.
  • What data classification is the model allowed to process? If the LLM will read production logs, the host it runs on must meet the controls required for that data class.
  • Who can reach the inference API? Ollama and similar servers ship with no authentication. Network policy, a reverse proxy with auth, or mTLS must gate access.
  • What gets logged? Prompts can contain sensitive data. Decide deliberately whether inference requests are logged, where, and with what retention.
  • What is the model’s license? Open-weight models carry licenses (some with use restrictions). Legal review is part of the deployment checklist, same as any vendored dependency.

The payoff for this diligence is significant: use cases like secure log analysis become possible on data you could never send to a public API.

Local LLM Firewall & Proxy Configuration Steps

Most enterprise networks aren’t fully air-gapped — they allow outbound traffic through an authenticated proxy with egress filtering. Here’s the local LLM firewall proxy configuration for that common case.

1. Route Ollama’s outbound traffic through the corporate proxy.

Ollama pulls models over HTTPS. Since it runs as a systemd service, set proxy variables in the unit environment:

sudo systemctl edit ollama

[Service]
Environment="HTTPS_PROXY=http://proxy.corp.example:3128"
Environment="HTTP_PROXY=http://proxy.corp.example:3128"
Environment="NO_PROXY=localhost,127.0.0.1,.corp.example"

sudo systemctl daemon-reload && sudo systemctl restart ollama

2. Request the necessary egress allowlist entries.

Model pulls need HTTPS (port 443) access to Ollama’s registry domains (ollama.com / registry.ollama.ai). If you pull GGUF files directly, add huggingface.co and its CDN endpoints. Submit these to your network team as an explicit, documented allowlist — it’s a short list, which is exactly what firewall teams like to see.

3. Handle TLS interception.

Many corporate proxies re-sign TLS traffic with an internal CA. If pulls fail with certificate errors, install your corporate root CA into the system trust store:

sudo cp corp-root-ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

4. Lock down the inference API itself.

Keep OLLAMA_HOST bound to 127.0.0.1 unless there’s a reason not to. If other hosts need access, front it with a reverse proxy (nginx, Caddy, Traefik) that enforces authentication and TLS, and restrict the port with firewall rules to specific source subnets.

How to Run an Ollama Docker Container Behind a Proxy

Containerized deployment is the norm for fleet rollouts. Running Ollama in Docker behind a proxy involves two separate proxy contexts — the Docker daemon (for pulling images) and the container runtime (for pulling models):

Docker daemon proxy (/etc/systemd/system/docker.service.d/proxy.conf):

[Service]
Environment="HTTPS_PROXY=http://proxy.corp.example:3128"
Environment="NO_PROXY=localhost,127.0.0.1,registry.corp.example"

Container-level proxy for model pulls:

docker run -d \
  --name ollama \
  -e HTTPS_PROXY=http://proxy.corp.example:3128 \
  -e NO_PROXY=localhost,127.0.0.1 \
  -v ollama_models:/root/.ollama \
  -p 127.0.0.1:11434:11434 \
  ollama/ollama

[IMAGE: Docker terminal configuring an Ollama container proxy for firewall compliance]

Notes that save debugging time:

  • Bind the published port to 127.0.0.1 (as above) or an internal interface — not 0.0.0.0 — unless a reverse proxy is in front of it.
  • Persist /root/.ollama in a named volume so model pulls survive container replacement.
  • If your proxy requires authentication, prefer machine credentials injected via your secrets mechanism over embedding them in unit files.
  • Mirror the ollama/ollama image into your internal registry so production hosts never pull from Docker Hub directly.

Deploying in a Strictly Air-Gapped Environment

For a true local LLM air-gapped environment — no route to the internet, ever — the workflow splits into a connected staging host and an offline transfer.

Transferring Weights and Offline Execution

Step 1 — Stage on a connected machine. On a vetted, internet-connected host (ideally in a DMZ used for artifact ingestion), install Ollama and pull the models you need:

ollama pull mistral:7b
ollama pull llama3.2:3b

Step 2 — Package the artifacts. Ollama stores model blobs and manifests under its models directory (/usr/share/ollama/.ollama/models for the system install, ~/.ollama/models otherwise). Archive it, along with the Ollama install binary/package and (if containerized) the Docker image:

tar czf ollama-models.tar.gz -C /usr/share/ollama/.ollama models
docker save ollama/ollama:latest | gzip > ollama-image.tar.gz
sha256sum ollama-models.tar.gz ollama-image.tar.gz > SHA256SUMS

Step 3 — Transfer through your approved channel. Use whatever your organization mandates for moving artifacts across the gap — data diode, scanned removable media, or a one-way transfer service. Verify checksums on the far side before anything else:

sha256sum -c SHA256SUMS

Step 4 — Restore and run offline. On the air-gapped host, install Ollama from the transferred package, unpack the models directory into place, fix ownership, and start the service:

sudo tar xzf ollama-models.tar.gz -C /usr/share/ollama/.ollama/
sudo chown -R ollama:ollama /usr/share/ollama/.ollama
sudo systemctl restart ollama
ollama list   # transferred models should appear here

ollama run and the REST API now work with zero outbound connectivity. An alternative pattern is a Modelfile referencing a transferred GGUF file directly (FROM ./model.gguf followed by ollama create), which is useful when weights come from an internal artifact repository rather than an Ollama registry pull.

Hardware note: air-gapped inference boxes are often repurposed servers without GPUs. That’s workable — see our guide to CPU hardware constraints for sizing RAM and cores for CPU-only inference before you commit to a host.

Best Practices for On-Prem LLM Compliance Setup

Treat the LLM stack like any other production service, plus a few LLM-specific controls:

  • Version and checksum everything. Record model name, quantization, file hashes, and license for every deployed model in your CMDB or artifact registry. “Which model version produced this output?” should always be answerable.
  • Gate the API with authentication and TLS. A reverse proxy with OIDC/mTLS in front of the inference port turns an unauthenticated service into one that fits your access-control story.
  • Segment the inference host. Put it in a zone appropriate for the data it processes, with explicit ingress rules and — in the firewalled (non-air-gapped) case — egress limited to the documented registry endpoints.
  • Decide prompt-logging policy explicitly. Either log prompts into a store with appropriate retention and access controls, or disable request logging entirely. Accidental sensitive-data logging is the most common audit finding in these deployments.
  • Patch cadence for models and runtime. Inference engines get CVEs like any software. In air-gapped setups, fold Ollama/engine updates into your existing offline patch pipeline.
  • Document the tool choice. If auditors ask why you chose your stack, a written evaluation helps — our Ollama vs LM Studio comparison covers the open-source and headless-operation criteria that usually decide it for compliance-conscious teams.

FAQ

Can I run an LLM completely offline with no internet access?
Yes. Once model weights are transferred to the host, tools like Ollama perform inference with zero outbound connectivity. The internet is only needed (on a staging machine) to obtain the weights initially.

How do I pull Ollama models through a corporate proxy?
Set HTTPS_PROXY, HTTP_PROXY, and NO_PROXY in the Ollama systemd unit environment (or as container environment variables in Docker), install your corporate root CA if TLS interception is in place, and allowlist the Ollama registry domains for egress.

Is a local LLM automatically compliant?
No — it removes the third-party data transfer problem, but you still need access controls on the API, appropriate network segmentation, a prompt-logging policy, and license review of the model weights.

What hardware do I need for an on-prem LLM?
For single-team ops workloads, a CPU-only server with 16 GB+ RAM can run 7B–8B quantized models acceptably. See our CPU hardware constraints guide for sizing details.

How do I get model weights into an air-gapped network?
Pull them on a connected staging host, archive Ollama’s models directory (or export GGUF files), verify checksums, move the archive through your approved cross-domain transfer channel, and restore it on the offline host.

Leave a Comment