For the past 3 years, graphics processing units (GPUs) have dominated the large language model (LLM) conversation. In traditional chatbot applications, central processing units (CPUs) provide a fraction of the total compute per request, while GPUs do the heavy lifting. However, inference isn't a single model answering a single question. A growing reliance on tool calls, multistep reasoning, and orchestration across small, specialized models changes the math on where compute should live. Intel has called out this shift noting that the CPU-to-GPU ratio is moving from 1:8 in training workloads to 1:1, and in some cases 4:1 in agentic deployments.

Here we'll examine why the assumptions making GPUs the obvious choice for LLM inference are being renegotiated, what's driving renewed demand for CPU-based serving, and what the data says about where the industry is heading.

What CPUs are good at

At their core (no pun intended), CPUs and GPUs solve fundamentally different problems.

A modern GPU contains tens of thousands of cores designed to execute the same operation on thousands of data elements simultaneously. This makes them extraordinarily fast at the dense matrix multiplications that dominate the forward pass of a transformer for inference. During training, and during high-concurrency batched inference, that parallelism translates directly into throughput: greater tokens per second (TPS), and many more requests served per dollar of compute.

Modern CPUs, by contrast, have anywhere from one to hundreds of cores optimized for sequential, conditional, and branching logic. They're particularly fast at a single operation moving through a complex decision tree. They have direct access to main system memory, and they're the natural execution environment for the orchestration layer wrapping any model. Tool dispatch, code execution, Python runtimes, sandboxes, input/output (I/O), and the agent loop control flow all sit in the CPU.

These architectures, while different in nature, aren't competitors; they work best in tandem. The real question isn't which architecture is better, but which workload belongs where.

This division of labor comes down to how we measure their work: FLOPS versus instruction latency.

GPUs live and die by floating-point operations per second (FLOPS). Because AI models are massive webs of decimal numbers being multiplied and added, a GPU's job is to brute-force trillions of these matrix calculations simultaneously. It's built entirely for raw mathematical throughput.

CPUs, by contrast, specialize in instruction latency. Instruction latency measures how fast a single core can execute an unpredictable chain of diverse commands. A CPU core excels at the rapid logic shuffling needed to parse JavaScript Object Notation (JSON), handle network I/O, or check security permissions.

If you force a GPU to run a chaotic Python runtime, its massive FLOP capacity sits idle, choked by constant task-switching. If you force a CPU to crunch an LLM's math, it works perfectly, but takes ages because it lacks the massive parallel pipelines. The GPU is the mathematical muscle; the CPU is the logic engine directing the show.

The traditional inference stack: Where CPUs were passengers

When we think of the traditional inference stack, we think about the chatbot serving application. Here, the CPU often plays a supporting role.

A request arrives at the API server, where the CPU tokenizes and schedules it. The CPU handles all these initial tasks. It then hands the request to the GPU for the forward pass, which dominates the compute budget. The GPU runs attention, feeds forward layers, and samples the next token with a brief CPU sync to retrieve it and update the scheduler. It then repeats until an end-of-sequence token is emitted. Finally, the CPU collects the output and returns it to end the session.

In this model, the CPU acts as a receptionist, doing the coordination and organizational work, while the GPU does the computational heavy lifting. The ratio of CPUs to GPUs in AI data centers during the training era reflected this: roughly 1 to 2 CPUs for every 8 GPUs, a ratio dictated more by GPUs throughput than by CPU demand. Teams provisioned CPUs to keep GPUs fed. This setup is slowly shifting away from the de facto standard, as advancements in CPU inference increase global efficiency.

The first driver: How agents are changing the workload

With the rise of agentic AI comes a new inference profile magnifying the need for CPU-specific compute.

"Agentic AI" conjures up images of integrated agent harnesses, like Hermes and Openclaw, and developers with hosts of agents tied into their coding assistants. While these use cases are becoming more popular, agentic AI isn't as niche as it may seem at first glance. In fact, many people aren't using the "traditional" process of querying a chatbot. Popular AI assistants like Claude and ChatGPT are backed by enormous models that often incorporate "reasoning" stages into their more complex responses. During reasoning, the "chatbot" becomes a full agentic system. A single user-visible task may decompose into dozens of individual model calls, each of which is short, context-dependent, and conditionally branching.

In an agentic system, the model doesn't issue a single prompt and wait for a response. It generates a plan of action, executes tool calls, compiles all the returned data, and culminates in completing an action or returning an answer to the user. The CPU is in charge of parsing that output, figuring out which tool to invoke, making the API calls or running the code, collecting the result, and feeding it back. Then the loop repeats—sometimes dozens of times per user request, with parallel sub-agents adding more CPU coordination work on top. The model quickly becomes one component of a larger reasoning loop. The ratio of organizational work to pure computation shifts, making the CPU a much bigger player, and potential bottleneck, in the full process.

In a collaboration between researchers at Georgia Tech and Intel, researchers found that in agentic workloads, CPU-side tool processing accounts for 50–90% of total end-to-end latency. Intel's CEO Lip-Bu Tan, at Computex 2026, framed the shift precisely: "for reinforcement learning, orchestration, and agents, the CPU is a much better fit."

The numbers from Intel's Q1 2026 earnings call quantify how far this has already moved:

  • Training workloads: ~1 CPU per 8 GPUs
  • Inference workloads: Already shifted to ~1 CPU per 4 GPUs
  • Agentic workloads: Converging toward 1:1, with some customers reporting 4 CPUs deployed per GPU

Intel's Q1 2026 Data Center and AI segment revenue came in at $5.1B, up 22% year-over-year, with demand running ahead of supply. Intel has deprioritized consumer chip production to redirect fab capacity to server Xeon parts.

Arm's own analysis forecasts the shift more starkly. Traditional AI data centers require approximately 30 million CPU cores per gigawatt (GW) of capacity. Arm CEO Rene Haas estimates in the AI agent era, that figure rises to 120 million CPU cores per GW. The orchestration demands of agentic workloads drive this proposed 4× increase.

The second driver: Smaller, localized models

Separate from the shift agentic AI is cultivating, a second structural change is pushing inference toward CPUs. This change is the move toward smaller, domain-specific models deployed close to where data is generated.

We can attribute this change in tides to 4 main pressures on providers:

  1. Latency: A round-trip to a GPU cluster in a cloud data center takes time. For real-time applications like voice interfaces, autonomous systems, and industrial monitoring, that latency budget is measured in milliseconds. Local computing removes the network roundtrip.
  2. Privacy: Sending data off-device or off-premises to a cloud provider means data leaves the organization's control. Healthcare, finance, legal, and many other enterprise workflows increasingly cannot tolerate this. While this doesn't guarantee your data is being picked apart and auctioned off to the highest bidder or used for training, a local or CPU-resident model running on existing on-premises server infrastructure processes data without it ever leaving the facility, completely mitigating that risk.
  3. Cost: A single GPU can cost upwards of $45,000. For workloads that don't require the throughput of a GPU, that cost is unjustifiable. CPU inference on already-provisioned server infrastructure has a near-zero marginal hardware cost.
  4. Offline availability: Edge deployments frequently operate without reliable internet connectivity. Cloud GPU inference isn't an option in these scenarios. Local CPU inference is.

Small language models (SLMs) have matured to the point where this is viable. Hugging Face SmolLM2, ranging from 135M to 1.7B parameters, is designed for maximum efficiency on resource-constrained devices. The 135M variant fits entirely in CPU cache on modern smartphones. The performance gap between SLMs and large models narrows drastically when SLMs are augmented with retrieval-augmented generation (RAG), and reverses entirely on domain-specific tasks after fine-tuning.

A model trained on carefully curated data can achieve domain-specific performance with 100× less compute than training a general-purpose LLM. That's the caveat though. When it comes to fine-tuning and training models on larger GPUs, you have a lot more headroom to throw things at the model. On these smaller deployments, users need to be much more aware of the actions they're taking to produce performant and competitive models.

What this looks like in the industry today

These aren't theoretical workloads. The CPU era is fast approaching, if not already here.

OpenAI and AWS signed a $38B, 7-year infrastructure partnership in November 2025. The press release states access to hundreds of thousands of NVIDIA GPUs, with the ability to expand to tens of millions of CPUs to rapidly scale agentic workloads.

In March, Arm launched the AGI CPU, marking the company's first production silicon in its 35-year history. NVIDIA launched the Vera CPU at GTC San Jose in March 2026, calling it "the world's first processor purpose-built for agentic AI." Jensen Huang described it as NVIDIA's "next multibillion dollar business." The Vera CPU delivers:

  • 1.8× faster sandbox performance compared to leading x86 CPUs for agentic inner-loop work (code compilation, Python tool chains, software code analysis)
  • 2× memory bandwidth and 3× bandwidth per core compared to leading x86 CPUs with DDR5, via LPDDR5X memory with SOCAMM

First deliveries went to Anthropic, OpenAI, SpaceXAI, and Oracle Cloud Infrastructure, who announced plans to deploy hundreds of thousands of Vera CPUs beginning in 2026.

NVIDIA's flagship agentic inference platform, the Vera Rubin NVL72 rack, combines 72 Rubin GPUs with 36 Vera CPUs per rack, illustrating a pivot from the traditional 1:8 CPU:GPU ratio to a 1:2 ratio. The architecture is explicitly optimized to use GPUs for token generation and CPUs for everything wrapping it.

Morgan Stanley estimates the agentic CPU shift represents $32.5–$60B in incremental CPU market growth by 2030.

Figure 1: The industry shift toward CPU-heavy AI infrastructure, highlighting NVIDIA Vera CPU's 1.8× performance advantage, the NVL72 rack's 1:8 to 1:2 CPU-to-GPU ratio change, and key developments from Arm, OpenAI/AWS, and Morgan Stanley projecting $32.5–$60B in CPU market growth by 2030.

Figure 1: The industry shift toward CPU-heavy AI infrastructure, highlighting NVIDIA Vera CPU's 1.8× performance advantage, the NVL72 rack's 1:8 to 1:2 CPU-to-GPU ratio change, and key developments from Arm, OpenAI/AWS, and Morgan Stanley projecting $32.5–$60B in CPU market growth by 2030.

Measuring and deploying LLMs on CPUs

For GPU-dominated inference workloads, the GPU advantage is substantial and well-documented. A single H200 serving Llama 3.1 8B hits approximately 7,464 generation tokens/sec on chat workloads. The best-performing CPU virtual machine (VM) in the same Azure benchmark falls significantly short.

But peak throughput under high concurrency isn't the benchmark that matters most for CPU inference. It's more relevant to measure the workload profile corresponding to CPU-appropriate deployments, which can be more complicated to test properly.

GPU benchmarking has established norms, but CPU inference benchmarking is currently a mess of vendor-specific or not-so-reproducible numbers. The key difference is that CPUs share cores, memory bandwidth, and cache with everything else on the machine, so isolation requires careful socket-level or node-level separation GPU setups don't need. Red Hat's response is an open source 3-phase framework (vllm-cpu-perf-eval) using Red Hat Ansible Automation Platform and GuideLLM testing baseline capacity, realistic traffic variance, and production optimizations in a reproducible, standardized way.

This can all sound pretty daunting, especially if your mental model of LLM deployment is still built around GPU clusters. In practice, the ecosystem has matured considerably. Red Hat has a wide collection of validated models on Hugging Face. Additionally, Intel's AI Software Catalog lists a broad range of production-ready models validated and optimized for Xeon Gen 4–6 processors with vLLM support.

For a hands-on look at what deployment looks like, our recent vLLM office hours session walks through end-to-end inference deployment on Intel CPUs.

The architecture that makes CPU inference work

Several developments in the software stack have made CPU inference viable at a level it was not 2 years ago.

vLLM's CPU backend supports continuous batching, PagedAttention, prefix caching, chunked prefill, and tensor parallelism. These are the same scheduling and memory management primitives driving high-throughput GPU serving, but now also on:

  • x86 AVX-512
  • Arm AArch64
  • Apple Silicon (currently experimental)
  • IBM Z

The OpenAI-compatible API is identical across backends, which means a deployment starting on CPU infrastructure can migrate to GPU without application-layer changes.

On Arm, the oneDNN and Arm Compute Library build path, combined with INT4 quantization via llmcompressor, is a production path tested on AWS Graviton4.

On Intel, Intel Extension for PyTorch (IPEX) enables Advanced Matrix Extensions (AMX)-backed BF16 operations automatically when installed. The key environment variables give operators direct control over the memory and threading tradeoffs that matter most for CPU serving:

  • VLLM_CPU_KVCACHE_SPACE(KV cache allocation in GB)
  • VLLM_CPU_OMP_THREADS_BIND(OpenMP thread pinning)
  • The experimental VLLM_CPU_SGL_KERNEL (AMX + BF16 small-batch kernels)

Conclusion

The GPU isn't going anywhere. For high-concurrency production serving of large models, GPU is the right answer and will remain so.

What's changed is the perimeter of that statement. Agentic AI introduced a class of workloads—orchestration, tool execution, code sandboxing, and multi-agent coordination—where CPUs are the correct tool for the job. The ratio shift from 1:8 to 1:1 in Intel's data center projections isn't a temporary anomaly. This momentous change is the consequence of inference becoming iterative rather than single-shot.

At the same time, the economics and constraints of edge deployment—latency, privacy, cost, and offline availability—drive a parallel wave of CPU-based inference for small, localized models where GPU infrastructure was never the right fit.

NVIDIA building Vera. Arm jumping back into the game. Intel's Xeon demand running ahead of supply. OpenAI contracting for tens of millions of CPUs. The CPU is back.

What next?

  • Deploy models on CPUs with vLLM on OpenShift AI: Red Hat is a leading contributor to vLLM, and Red Hat OpenShift AI provides a supported, production-grade platform for serving models on both CPU and GPU infrastructure, with the same OpenAI-compatible API regardless of backend.
  • Benchmark your CPU inference performance: Use the open source vllm-cpu-perf-eval framework, along with GuideLLM, to establish reproducible baselines for your hardware and workloads.