Pegas field notes · October 2026
Why Does Running an Open Model Cost So Much?
What we learned about GPU economics, CPU hosting, hosted inference, and why RunPod was a reasonable first choice for Pegas — even though it was not the final infrastructure answer.
When we started Pegas, one thing looked strange almost immediately.
A normal web project can live on inexpensive hosting. A small VPS may cost tens of dollars per month. But the moment we rented a GPU for a self-hosted LLM, the price suddenly became an hourly number.
During our Pegas experiments, an NVIDIA L4 was shown at roughly $0.49 per hour, while an RTX 4090 configuration was around $0.74 per hour.
Run that continuously and the monthly cost becomes significant very quickly.
So the obvious question is: why does this cost so much, and why not just put the model on an ordinary server?
A GPU server is not really “normal hosting”
For a normal web application, the server mostly needs a CPU, RAM, storage, and networking.
That is enough for things like:
- serving web pages;
- running APIs;
- handling authentication;
- talking to a database;
- processing ordinary background jobs.
Large-language-model inference is a different kind of workload.
Generating text requires a huge number of matrix operations. GPUs are designed to execute many of these operations in parallel, while ordinary CPUs are general-purpose processors optimized for a much broader mix of tasks.
That is the reason the infrastructure economics change.
When we rent a GPU instance, we are not just renting “a server with more power.” We are renting access to a specialized accelerator with its own high-speed memory, power requirements, cooling, and limited availability in a data center.
That hardware is expensive to buy and expensive to keep online.
The hourly price reflects more than the card itself. The provider also has to pay for the host machine, electricity, cooling, networking, maintenance, idle capacity, and the fact that a GPU sitting unused still represents expensive hardware.
Could Pegas run on an ordinary CPU server?
Yes.
This is important.
Qwen3 4B is small enough, especially in a quantized format, that it can run on a normal server with sufficient system RAM.
In fact, during an earlier Pegas debugging session, we accidentally proved this.
The NVIDIA GPU was visible to the operating system, but Ollama did not select the CUDA backend correctly. The model fell back to CPU execution.
The model still worked.
The problem was performance.
Inference slowed dramatically compared with the GPU path. The observed generation speed dropped to roughly 6–7 tokens per second, cold starts became painful, and requests could become slow enough to hit timeouts.
So the real question was never:
Can the model run on a CPU?
It can.
The useful question was:
Can it run fast enough for a public interactive demo?
For Pegas, the GPU made that experience much more practical.
RAM and VRAM are not the same thing
A CPU server stores model data in ordinary system RAM.
A GPU uses its own high-bandwidth memory, usually called VRAM.
The difference matters because LLM inference repeatedly moves large amounts of model data through the compute pipeline.
System RAM can hold the model, but GPU VRAM is designed to feed massively parallel compute much more efficiently.
That is why a model that technically fits into an ordinary server can still feel much slower there.
For a background batch job, that may be acceptable.
For a live demo where someone clicks a button and waits for an answer, it may not be.
Why hosted LLM APIs can look much cheaper
There is another apparent contradiction.
Using a commercial LLM API often feels cheaper than renting an entire GPU.
The reason is utilization.
With a hosted API, one user does not rent a whole accelerator. The provider operates a large shared cluster and spreads the cost across many customers and many requests.
The user normally pays for usage, such as processed tokens, rather than for possession of a specific GPU for an hour.
That model is economically efficient because the infrastructure can keep expensive hardware busy across thousands of workloads.
But Pegas had a different goal.
We specifically wanted to demonstrate:
- an open model;
- our own inference runtime;
- a rented GPU;
- our own server-side integration;
- no third-party hosted LLM API in the inference path.
If we had replaced the backend with OpenAI, Anthropic, Together, Groq, Fireworks, or another hosted model API, the demo would still work — but it would no longer prove the same thing.
The point of Pegas was not merely to call a model.
The point was to show the infrastructure behind the inference.
Why we chose RunPod
For the original proof of concept, RunPod was a very reasonable choice.
We needed something much simpler than building the same experiment directly on AWS, Google Cloud, or Azure.
Traditional cloud GPU infrastructure can involve a large amount of setup before the model even starts:
- instance types;
- networking;
- firewall rules;
- IAM;
- storage;
- CUDA compatibility;
- drivers;
- ports;
- billing configuration.
RunPod reduced that initial barrier substantially.
The workflow looked closer to:
- choose a GPU;
- deploy a Pod;
- open a terminal;
- start the runtime.
It also offered several things that matched Pegas well:
- hourly GPU rental;
- multiple GPU types;
- prebuilt CUDA and PyTorch environments;
- browser terminal access;
- persistent storage options;
- exposed HTTP and TCP ports;
- reusable templates;
- a relatively approachable UI.
For the original question —
Can one person without an ML-engineering background rent a GPU, run Qwen3 4B, expose it through Ollama, and connect it to a normal web frontend?
— RunPod did its job.
We proved the full path.
The problem appeared after the proof worked
The difficulty came later.
Running a model once is not the same thing as operating it cleanly.
For a portfolio project, we did not want to pay for a GPU 24/7.
The ideal lifecycle looked simple:
Stop the GPU when we are done.
Start it later.
The demo comes back.
That is where the infrastructure became much more complicated than expected.
Across repeated tests we encountered several different lifecycle problems:
- a stopped Pod could lose access to the physical GPU it had used;
- local persistent storage could tie the workload to a specific host;
- migration could create a new Pod identity;
- a new Pod identity meant a new HTTP proxy hostname;
- changing the hostname meant changing
MODEL_BASE_URLin Vercel; - Direct TCP endpoints could change after restarts;
- Global Volume persistence worked, but model loading from it performed poorly in our experiment;
- the model could unload from GPU memory after inactivity;
- the next request would then pay the cold-start cost again.
None of these problems meant that RunPod was “bad.”
They meant that we had crossed from a proof-of-concept problem into an operations problem.
GPU availability is part of the system
One of the clearest lessons came from GPU availability.
The original Pegas proof used an NVIDIA L4.
Later, when we tried to make the environment restartable, L4 availability in the selected region was low.
An RTX 4090 was much easier to obtain.
That forced an architectural realization:
If a demo can only restart when one specific GPU SKU happens to be available on one specific host, then the demo is not really restartable.
The infrastructure should be able to treat compatible GPU capacity as a pool, not as a machine that we hope is still waiting for us.
The economics explain the architecture
At this point the relationship between cost and architecture became much clearer.
A dedicated GPU Pod is easy to understand:
This GPU is mine while it is running.
But that also means:
I pay while it is running, and I become responsible for its lifecycle.
A serverless GPU model changes the relationship:
Here is my runtime. Give it GPU capacity when a request needs it.
The provider can then handle the scheduling, reassignment, and scaling.
That is much closer to what Pegas actually needs.
The project does not require one physical RTX 4090 or one physical L4.
It requires:
- a compatible NVIDIA GPU;
- our open model;
- our runtime;
- a stable HTTPS endpoint;
- persistence for model assets;
- predictable cold-start behavior.
The trade-off we discovered
Our later restartable RunPod configuration became much easier to operate than the original setup, but it also showed a performance trade-off.
Warm requests that had previously been around 2–4 seconds were now more commonly around 4–6 seconds.
After roughly five minutes of inactivity, the next request could again take around 20 seconds, because the model had to become resident in GPU memory again.
That distinction matters.
There are really two latencies:
- cold latency — infrastructure or model state has to be initialized;
- warm latency — the model is already loaded and ready.
A useful AI demo should measure both.
Why this was still a useful result
It would have been easy to stop Pegas as soon as the first successful inference appeared.
That would have produced a cleaner story, but a less useful one.
Instead, we tested what happens when the GPU is actually stopped, restarted, migrated, and left idle.
That exposed the difference between:
“The model runs.”
and:
“The system is practical to operate.”
Those are not the same achievement.
What we would choose differently now
For a similar project, we would make infrastructure lifecycle a first-class requirement from the beginning.
The requirements would be:
- one stable HTTPS endpoint;
- no manual terminal recovery;
- no model re-download after ordinary scale-down;
- no Vercel configuration changes after GPU reassignment;
- model persistence independent of a physical host;
- automatic access to compatible GPU capacity;
- explicit measurement of cold and warm latency;
- the ability to scale to zero when the demo is idle.
That points away from a manually managed GPU Pod and toward a serverless GPU runtime.
For Pegas, the next infrastructure experiment should be evaluated against those criteria before the frontend is connected to it.
The main lesson
Running an open-source model was not the hardest part.
Making the system economical, restartable, and boring to operate was harder.
The cost of GPU hosting is not just the price of faster hardware.
It changes the architecture.
And that is probably the most useful thing the Pegas experiment has taught us so far.