Pegas field notes · October 2026
The Day Pegas Stopped Needing a GPU Babysitter
Moving Pegas from a fragile manually operated GPU workflow to request-driven serverless inference changed the project more than any model upgrade.
For most of this project, getting an open model to answer a request was not the hard part.
Keeping it operational was.
Pegas already had a real open-weight model, a live frontend, structured output, a fixed benchmark, runtime evidence, and a public code trail. But the compute layer still behaved like a fragile demo. Turning the GPU off to save money could mean manually recovering the runtime, checking model files, finding a new endpoint, updating Vercel, and testing the whole path again.
That is not a usable operating model. It is a ritual.
Today that changed.
The biggest result of this session was not a better prompt or a higher benchmark score. It was a different relationship between the website and the GPU:
No traffic
→ zero live GPU containers
→ no active GPU compute
New request from the website
→ Modal starts a container automatically
→ Ollama starts
→ qwen3:4b loads
→ the request completes
Next requests
→ the same warm runtime answers in about a secondtextFor the first time, Pegas started to feel less like a machine I have to keep alive and more like a system that can operate on its own.
The problem was never just inference
The previous RunPod setup proved something useful: a self-hosted open model could serve real requests from a normal web product. It also exposed the operational cost of doing that with a manually managed Pod lifecycle.
Across restart experiments, the same classes of problems kept returning:
- GPU inventory could disappear after a stop;
- storage persistence did not guarantee compute availability;
- migration could change Pod identity and endpoint;
- a new endpoint could require a Vercel environment-variable change and redeploy;
- cold model loading could take tens of seconds;
- recovery still depended on the operator knowing exactly what to fix.
That meant the real bottleneck had moved.
The model worked. The website worked. The missing piece was orchestration.
The acceptance criterion became much stricter than “one inference succeeded.” The new backend had to survive scale-down and recovery repeatedly through one stable endpoint, without a human restarting infrastructure.
A deliberately narrow migration
The first decision was to avoid changing everything at once.
Pegas already used Ollama and qwen3:4b. So the first Modal proof kept both:
Modal
→ NVIDIA L4
→ Ollama
→ qwen3:4btextNo vLLM migration. No model swap. No frontend redesign.
That mattered because if the experiment failed, there would be only one major new variable: the compute/orchestration layer.
The model was baked into the Modal image so a fresh container would not need a manual ollama pull. The deployed app used min_containers=0, allowing it to scale completely to zero when idle.
The first deploy did fail, but for a useful reason: the current Ollama installer required zstd, which was missing from the image. Adding one package fixed the build. It was a small reminder that reproducibility is often decided by boring dependencies rather than dramatic architecture.
After that, the Modal endpoint deployed successfully.
The first cold request was ugly
The first isolated lifecycle test was not fast.
A cold request took roughly 85–96 seconds in the early tests. When the same path was connected to the real Pegas frontend, the first production-path request took:
91,070 mstextThat is objectively too slow for a polished public demo.
But it was also the first time the long delay had a very different meaning.
With the old operating model, a stopped GPU could mean manual recovery work.
With the new one, the long request itself was doing the recovery.
No Pod start. No terminal session on the GPU host. No model reinstall. No endpoint replacement. No Vercel edit.
The request waited, but the system recovered by itself.
That distinction is the main architectural result of the day.
Then the same model became fast
The second request changed the picture completely.
With the container already alive and the model resident, the same real frontend path returned in:
1,328 mstextAdditional warm requests in the isolated test landed around 0.95–1.4 seconds. The Modal function history showed the same split very clearly: one cold call had startup plus a long first execution, while the following calls executed around one second.
This made the next engineering problem much more precise.
The steady-state runtime is already good enough for the portfolio demo. The remaining performance problem is almost entirely cold start.
That is a much better problem to have.
The benchmark became operationally cheap
Once the runtime was warm, the full 25-case benchmark completed quickly through the same live inference path used by the demo.
The current run showed:
Category accuracy: 56% (14 / 25)
Priority accuracy: 68% (17 / 25)
Valid output: 100% (25 / 25)
Median latency: 1264 ms
P95 latency: 1464 mstextThe accuracy numbers are not presented as a victory. They are evidence.
The model is imperfect at this task. Category accuracy is only 56%, and priority accuracy is 68%. That is exactly why the benchmark exists. Pegas is more credible when it exposes where the model fails instead of hiding behind a few hand-picked examples.
What changed today is that running the benchmark no longer feels operationally expensive once the runtime is warm.
The frontend did not need to be rebuilt
Another important result was architectural restraint.
The production frontend already had the right responsibilities:
- build the classification prompt;
- define the structured schema;
- validate the model response;
- expose latency and proof metadata;
- keep the benchmark logic stable.
So the Modal integration did not move those responsibilities into a new service.
Instead, the Modal app added an Ollama-compatible /api/generate route and acted as an authenticated transport to the local Ollama runtime.
The real path became:
Browser
→ Vercel / Next.js
→ /api/analyze
→ existing production prompt + schema
→ Modal stable endpoint
→ Ollama
→ qwen3:4b
→ structured result
→ frontendtextThis preserved the existing application contract and reduced migration risk.
The proof panel continued to show the deployed model, configured compute, runtime, schema validity, latency, and the absence of a hosted LLM API in the target inference path.
Scale-to-zero changed the economics and the operating model
The most important test happened after the website was already working.
We stopped sending requests and waited.
Then:
modal container listtextreturned no active containers.
The Modal dashboard showed the same state:
Containers: 0 live
Calls: 0 runningtextThis is the state Pegas should spend most of its life in.
The website remains deployed on Vercel. The Modal app remains deployed behind a stable endpoint. But no GPU container needs to sit running just in case somebody visits.
Then we sent another request from the website.
Without running a start command, a new container appeared automatically:
That is the operational behavior I wanted from the beginning:
I do not "start the model."
A request starts the compute it needs.textThat single change removes an entire category of failure from the demo workflow.
The cost evidence was finally understandable
During the session, Modal's billing report showed the actual resource-level costs generated by the experiments: CPU, memory, network egress, and L4 GPU usage.
At the point captured in the billing summary, the session had generated approximately:
Metered cost: $0.35
Credits: -$0.35
Billed cost: $0.00textThat screenshot is not a promise that the project is free. It is useful because it makes the cost model concrete.
Pegas can now sit at zero live containers when unused and consume GPU compute when somebody actually asks the model to do work.
That is dramatically easier to reason about than paying for a long-lived GPU Pod and manually deciding when it is safe to stop it.
What this did not solve
Today was not the end of the infrastructure work.
The remaining cold-start latency is still the biggest weakness. A first request taking around 90 seconds is too long for a polished public experience.
The next optimization work should focus specifically on that path rather than changing the whole architecture again. Possible experiments include:
- reducing the amount of model/runtime initialization on a fresh container;
- evaluating Modal GPU or memory snapshots;
- testing whether another inference runtime such as vLLM improves cold loading without compromising the portfolio story;
- deciding whether a small warm window is worth the cost before demos or interviews;
- improving the UI so a real cold start is explained rather than feeling like a frozen request.
There is also a proof limitation still visible in the current runtime panel: the backend endpoint itself is not exposed to the frontend proof UI. That is intentional from a security perspective, but the proof mechanism can probably be improved so the runtime can attest to the Modal deployment more directly without revealing credentials.
And the benchmark still says the model itself needs work. Faster infrastructure does not turn 56% category accuracy into a good classifier.
Those are now separate, tractable problems.
The real result of the day
The most important outcome was psychological as much as technical.
Before this session, the project had a recurring contradiction: Pegas was supposed to demonstrate a reproducible, understandable AI system, but keeping the demo alive still required too much operator knowledge.
That made every infrastructure problem feel existential. If stopping a GPU could break the demo, then every cost-saving action carried risk.
The new operating model is different:
Do nothing
→ GPU disappears
Open the site later
→ request arrives
→ compute appears
→ model answers
Keep using it
→ warm requests are fast
Stop using it
→ compute disappears againtextThe first request is still too slow. The model is still imperfect. There is still optimization work ahead.
But the architecture finally points in the right direction.
Today Pegas stopped feeling like a fragile GPU experiment that happened to have a frontend.
It started feeling like a small product with an infrastructure lifecycle of its own.
And that changed my confidence in the project more than any benchmark score could.
---