← Back to blog

Pegas field notes · October 2026

When the Demo Worked - and Then Everything Broke

Part 2: what persistent storage, cloud networking, and GPU detection taught us

A practical case study for people who are not ML engineers

We had already crossed the exciting milestone: the model was no longer a mock. A browser request went through the Next.js server, reached the cloud GPU, came back as structured JSON, and appeared in the interface with live metadata. One successful request later measured 1.628 seconds end to end. That number is a single live request, not a benchmark.

The new goal

The next step was not to make the model smarter. It was to make the system less fragile. We wanted three things that sound boring until they fail:

  • Keep the model files between GPU sessions so we did not have to download gigabytes every time.
  • Expose the Ollama API in a way the frontend could reliably reach after a restart.
  • Prove that successful answers were actually generated on the NVIDIA L4, not quietly falling back to CPU.

1. Persistence worked - but it was only one layer

We created a RunPod Global Volume, mounted it at /workspace, and stored the Qwen3 model there. That part worked: after stopping and starting the pod, ollama list could see qwen3:4b immediately without downloading the model again. Persistence was real.

But preserving files is not the same as preserving a working inference stack. Ollama itself contains more than a single executable, model loading behavior matters, and the networking layer is separate again. We had solved storage, not the whole system.

2. A healthy-looking endpoint that returned 404

Next we tried to give Ollama its own exposed HTTP port. The root page said “Ollama is running”, which looked reassuring. But real API paths such as /api/version, /api/tags, and /api/generate repeatedly returned HTTP 404 through the RunPod HTTPS proxy. We reproduced the same behavior on multiple ports, so changing port numbers was not a solution.

3. One Ollama binary was not the whole runtime

To avoid reinstalling Ollama after a reset, we initially copied the ollama executable to persistent storage. The API could start and /api/tags could list the model, so this looked promising. Then the first real generation request failed with a much more useful error: llama-server binary not found.

That error clarified the architecture. Ollama is not just one command-line binary. Its inference runtime includes additional libraries and the llama-server component. Listing models proved that metadata was readable; it did not prove that the full inference engine was present. We stopped trying to preserve partial runtime state and returned to a clean installation on each fresh pod.

4. The most expensive false positive: the GPU existed, but the model was on CPU

The next trap was more subtle. nvidia-smi correctly showed an NVIDIA L4 and available CUDA. The container absolutely had a GPU. Yet ollama ps reported 100% CPU. The model still answered, but cold loading took roughly 95 seconds and generation dropped to only a few tokens per second. A “working” model had quietly become the wrong model path.

Ollama process listing for qwen3:4b showing the model running on 100 percent CPU instead of GPU.
A successful response was not enough. This process listing exposed the wrong execution path.

The fix was to make the backend choice explicit. Starting Ollama with OLLAMA_LLM_LIBRARY=cuda_v13 and OLLAMA_VULKAN=0 produced a clear runtime log: library=CUDA, description=NVIDIA L4. On the clean pod, ollama ps then reported 100% GPU again.

Ollama process listing for qwen3:4b showing the model running on 100 percent GPU.
Same model, corrected runtime path: Ollama now reports 100% GPU.

Same Qwen3 4B model, two execution paths. A successful response is not proof that the GPU is being used.

5. The reset that finally simplified everything

After too many layers of partial fixes, we did the obvious thing: we terminated the messy pod and rebuilt a clean one using what we had learned. We kept the persistent volume as an experiment and backup, but stopped using it as the live model store. The working inference path returned to local pod storage, where Qwen3 4B had behaved well from the beginning.

Direct TCP solved the networking block. A request from a normal Windows PowerShell session reached Ollama and returned PEGAS_OK. We then updated the server-side MODEL_BASE_URL in Vercel, redeployed, and ran the same integration scenario from the public Pegas frontend.

Back online: live structured inference from the public frontend. This request measured 1628 ms and passed schema validation.

Where we ended up

  • The public frontend again reaches Qwen3 4B through the Next.js server route.
  • Ollama is running on a clean RunPod pod with the NVIDIA L4 explicitly selected through CUDA.
  • ollama ps confirmed 100% GPU for the model.
  • External access currently uses RunPod Direct TCP rather than the HTTPS proxy that returned 404 for /api/* in our tests.
  • The Direct TCP endpoint is configuration, not code, and its external port can change after a restart.
  • This demo path is plain HTTP without a dedicated application-level auth layer, so it is not the final production security design.

What we would do differently next time

First, verify each layer separately before combining them. Second, treat GPU usage as something to measure, not assume. Third, keep infrastructure addresses outside the code from day one. Fourth, do not confuse persistence with readiness: preserving model files is useful, but a restarted system still needs its runtime, network path, and accelerator to be healthy.

And finally, do not be afraid to throw away a complicated intermediate setup. After enough debugging, a clean rebuild can be faster than preserving every workaround. The important part is not avoiding failure. It is leaving the experiment with a smaller, better-tested architecture than the one you started with.

The first Pegas article was about proving that one person could run an open model on rented GPU infrastructure. This second chapter was about something less glamorous and more valuable: learning what has to be true for that proof to keep working after the first successful demo.