← Back to blog

Pegas field notes · October 2026

Making a Self-Hosted LLM Easy to Restart Was Harder Than Running It

What RunPod storage, GPU availability, persistent models, HTTP proxying, and cold starts taught us about turning a working AI demo into something practical to operate.

The demo already worked. Restarting it did not.

Pegas already had the part that usually gets the attention: a real browser request could reach a rented cloud GPU, Ollama could run Qwen3 4B, and the frontend could display a structured result.

The harder operational problem appeared when we tried to use the project the way a small portfolio demo is actually used: turn the GPU off when nobody needs it, then bring it back later without rebuilding the whole runtime by hand.

The old restart path was much too manual. After starting a RunPod Pod, we could end up reinstalling Ollama, starting it with the correct CUDA environment, checking that Qwen3 4B existed locally, verifying that inference was really using the GPU, finding the new Direct TCP address, updating MODEL_BASE_URL in Vercel, and redeploying.

It worked, but it was not a good operating model. The goal of this iteration became simple: make a normal Stop → Start cycle boring.

First improvement: make Ollama part of the runtime

The first change was to stop treating Ollama as something we install after the machine starts. We created a RunPod template based on the official ollama/ollama image and exposed Ollama on HTTP port 11434.

The runtime configuration included:

  • OLLAMA_HOST=0.0.0.0:11434
  • OLLAMA_MODELS=/workspace/ollama-models
  • OLLAMA_LLM_LIBRARY=cuda_v13
  • OLLAMA_VULKAN=0

That removed two recurring steps immediately: reinstalling Ollama and manually launching the server after every restart. Starting the Pod now also starts the inference service.

A stable HTTP endpoint changes a lot

The earlier live setup used RunPod Direct TCP. It was enough to expose Ollama, but the externally assigned TCP address could change after a restart. That meant a simple infrastructure restart could cascade into a Vercel environment-variable change and another deploy.

The new template exposed Ollama through RunPod's HTTP proxy instead. Once the service was ready, the root endpoint returned Ollama is running, while /api/tags and /api/generate worked normally.

We then pointed MODEL_BASE_URL at that proxy endpoint. For a normal Stop → Start of the same Pod, the frontend no longer needed a newly discovered TCP address.

There is still an important boundary: if the Pod is terminated and replaced, its identity changes and so can its proxy address. This is a more stable lifecycle for the same Pod, not a claim that the endpoint is permanently immutable.

Persistence is not one problem

Our first persistence test used a RunPod Volume Disk. It did exactly one important thing correctly: the model files survived a Stop → Start cycle.

Then the restart failed for a completely different reason. The Pod was tied to a physical host, and that host no longer had enough free GPU capacity to resume it.

That exposed a distinction that is easy to miss when thinking only about files: persistent local storage can preserve the model while also coupling the workload to one host strongly enough that compute availability becomes the blocker.

We therefore moved the live model store to a RunPod Network Volume in EU-RO-1. The volume size was 10 GB, mounted at /workspace, while Ollama continued to use:

/workspace/ollama-models

Qwen3 4B was pulled once through Ollama's HTTP API. After the pull completed, /api/tags showed qwen3:4b from the persistent model store.

The GPU can become part of the failure mode

The original Pegas proof used an NVIDIA L4, and the existing articles correctly document those experiments.

During this restart work, however, L4 availability in the selected RunPod region was low. RTX 4090 availability was much better, so the new live runtime moved to an RTX 4090.

This was not an attempt to rewrite the historical hardware story or claim that one GPU is universally better. It was an infrastructure decision: Qwen3 4B did not require an L4 specifically, and an on-demand system is only useful if the compute can actually be scheduled when you want to restart it.

The configuration that finally survived Stop → Start

The working combination for this iteration was:

  • RTX 4090 compute in the selected RunPod region
  • official ollama/ollama runtime template
  • 10 GB Network Volume mounted at /workspace
  • model store at /workspace/ollama-models
  • RunPod HTTP proxy on Ollama port 11434
  • Qwen3 4B pulled once through the Ollama HTTP API

We stopped the new Pod and started it again. Ollama started automatically. The HTTP proxy returned. /api/tags still showed qwen3:4b. No new model download was required.

There was a short window during startup when an API request returned HTTP 404. A little later, the same service returned Ollama is running and its API routes worked normally again. The lesson was about readiness timing, not lost persistence or a permanently broken proxy.

The cold-start tax

Persistence solved the model-download problem. It did not mean the model was already loaded into GPU memory.

That difference showed up in two observed live requests after restart:

  • first inference: approximately 21,997 ms
  • following inference: approximately 5,260 ms

This was not a benchmark. It was a real pair of requests that exposed the remaining cold-start cost. The likely explanation is that the first request also had to load the model into GPU memory.

Automatic model warm-up is the obvious next optimization, but it is not implemented yet. The current project still exposes that first-request penalty honestly.

What we would do differently now

  • Design and test restartability before treating the infrastructure as finished.
  • Separate model persistence from container lifetime.
  • Do not couple the architecture to one GPU SKU unless the workload requires it.
  • Prefer stable HTTP service addressing over external TCP mappings that need to be rediscovered.
  • Test a real Stop → Start cycle, not just the first successful deployment.
  • Measure cold and warm inference separately.
  • Keep infrastructure claims limited to what was actually verified.

Current operating model

The normal startup process is now much smaller:

RunPod → Start Pod → wait for Ollama API Ready → Pegas can serve inference

The model remains on persistent storage, Ollama starts with the runtime, and the normal Stop → Start path no longer requires rediscovering a Direct TCP address or downloading Qwen3 4B again.

The next improvement is automatic warm-up so the first user request does not pay the full model-loading cost. That remains future work.