← Back to blog

Pegas field notes · October 2026

How We Ran an Open-Source LLM on a Cloud GPU

RunPod + Ollama + Qwen3 4B: a practical proof, not a benchmark

Can one person rent a GPU, run an open model, expose it through an API, and get a valid structured response without sending inference to a hosted LLM API?

The short answer

Yes. We launched Qwen3 4B on a rented NVIDIA L4, served it with Ollama, verified GPU inference, exposed it through RunPod, and sent a structured request from a normal Windows machine. The final showcase request completed in 2.602 seconds end to end and passed schema validation.

The setup

  • Cloud GPU: RunPod, 1x NVIDIA L4 (24 GB VRAM)
  • Runtime: Ollama 0.35.0
  • Model: qwen3:4b
  • External test: Windows PowerShell → RunPod HTTPS proxy
  • Output contract: category, priority, summary, nextAction

What we actually did

1. Confirmed the GPU was real and visible

We ran nvidia-smi inside the pod and confirmed the NVIDIA L4, driver 570.195.03, and CUDA 12.8.

Terminal output from nvidia-smi showing an NVIDIA L4 GPU, driver 570.195.03, and CUDA 12.8 inside the RunPod container.
The first layer of proof: the rented NVIDIA L4 is visible inside the container.

2. Installed Ollama and pulled Qwen3 4B

Ollama installed correctly, but the container had no running systemd, so we launched ollama serve manually. The qwen3:4b model download was about 2.5 GB.

3. Verified real GPU inference

ollama ps reported 100% GPU, confirming the model was not quietly falling back to CPU.

4. Learned that “valid JSON” is not enough

A plain JSON response can still break an application contract. We switched to JSON Schema and machine validation and got SCHEMA_VALID.

Terminal output showing a structured Qwen3 4B response followed by SCHEMA_VALID after JSON Schema validation.
A readable JSON response was not enough; the application contract had to be validated too.

5. Exposed the API without resetting the pod

Instead of editing the pod and triggering a reset, we reused the already exposed port 8888, stopped Jupyter Lab, and temporarily moved Ollama there.

6. Sent the final request from Windows

The final path was:

Windows → RunPod HTTPS proxy → Ollama → Qwen3 4B → structured JSON

The Showcase Proof Run reported:

  • Model: qwen3:4b
  • Runtime: Ollama 0.35.0
  • External HTTP: OK
  • Schema valid: True
  • Measured latency: 2602 ms
Windows PowerShell output from the Pegas showcase proof run showing the request path, qwen3:4b, Ollama 0.35.0, 2602 ms latency, and a schema-valid structured result.

What surprised us

  • Containers do not always behave like normal servers.
  • JSON is not the same as a stable API contract.
  • Infrastructure changes can be destructive.
  • A public model endpoint is useful for a smoke test, not for the final app.

What we proved

  • NVIDIA L4 was available inside the pod.
  • Ollama reported 100% GPU inference.
  • qwen3:4b returned structured JSON matching our schema.
  • The API was reachable externally through RunPod HTTPS proxy.
  • One end-to-end Showcase Proof Run measured 2602 ms.

Cost and next step

We stopped the GPU immediately after the proof run, so compute billing returned to $0.00/hour. RunPod billing data was still delayed, so we are waiting for the official billing record before publishing the final experiment cost.

The next application path is:

Browser → Next.js /api/analyze → RunPod → Ollama → Qwen3 4B → validated response → UI

After that comes the frozen 25-case benchmark for accuracy, valid-output rate, latency distribution, real misses, and actual cost.

For Pegas, this was the moment the project stopped being a polished frontend preview and became a real self-hosted AI proof.