GPU vs. CPU for AI: What's the Difference

1. Concept Overview & Systemic Problem

To understand why graphics accelerators have become the primary infrastructural currency of the 21st century, one must look at the fundamental differences in the design philosophies of silicon chips:

• CPU (Central Processing Unit) — optimized for minimizing latency (Latency-Oriented): This is a small ensemble of 8–32 extremely powerful general-purpose cores with large L1/L2/L3 caches and complex branch prediction units. The CPU is designed to execute a single sequential thread of complex operating system instructions as quickly as possible.
• GPU (Graphics Processing Unit) — optimized for maximizing throughput (Throughput-Oriented): This is an array of 5,000–18,000 simple parallel computing cores (ALU), grouped into Streaming Multiprocessors (SM). They do not waste transistors on branch prediction but use the SIMT (Single Instruction, Multiple Threads) paradigm: the same mathematical operation (e.g., vector multiplication) is simultaneously applied to thousands of data streams.

Mental model: The CPU is a speedy courier in a sports car delivering one urgent package instantly. The GPU is a giant freight train carrying 10,000 tons of gravel in a single trip.

┌─────────────────────────────────────────────────────────────┐
│                   ARCHITECTURE COMPARISON                   │
├─────────────────────────────────────────────────────────────┤
│ 1. CPU: Few Fast Cores + Gigantic L3 Cache                   │
│    [ CORE 1 ]  [ CORE 2 ]  [ CORE 3 ]  [ CORE 4 ]          │
│    └───────────────── L3 CACHE (64MB) ─────────────────┘    │
│    Memory Bus: DDR5 (60 - 90 GB/s)                           │
├─────────────────────────────────────────────────────────────┤
│ 2. GPU: Thousands of Simple Cores + Wide Memory Bus          │
│    [SM][SM][SM][SM][SM][SM][SM][SM][SM][SM][SM][SM] ...     │
│    (up to 18,000 CUDA Cores + Tensor Cores)                  │
│    Memory Bus: GDDR6X / HBM3e (1,000 - 3,350 GB/s)          │
└─────────────────────────────────────────────────────────────┘

2. Practical Benchmarking: Measuring Speed via CLI

You can measure the speed of matrix multiplication on CPU versus GPU using a simple one-liner test in Python:

# Matrix multiplication speed test 8192x8192 on available devices
python3 -c "
import torch, time
n = 8192

# 1. CPU Test
a_cpu, b_cpu = torch.randn(n, n), torch.randn(n, n)
t0 = time.time()
torch.mm(a_cpu, b_cpu)
print(f'CPU Time: {time.time() - t0:.3f} s')

# 2. GPU Test (if CUDA or Apple MPS is available)
device = 'cuda' if torch.cuda.is_available() else ('mps' if torch.backends.mps.is_available() else None)
if device:
    a_gpu, b_gpu = a_cpu.to(device), b_cpu.to(device)
    torch.mm(a_gpu, b_gpu) # warmup
    t0 = time.time()
    torch.mm(a_gpu, b_gpu)
    if device == 'cuda': torch.cuda.synchronize()
    print(f'GPU ({device}) Time: {time.time() - t0:.4f} s')
"

On a modern system, the GPU completes the computation in 0.015 seconds, while a flagship 16-core CPU takes over 0.8–1.2 seconds for the same operation (a difference of 50–80 times!).

3. Why LLM Inference is Memory-Bound

During the text generation phase, the model operates under the rule: to output one token, it must read from memory all model weights once:

• If the Llama 3.1 8B model in 4-bit format occupies 5 GB:
  • On a system with a DDR5 bus (read speed 60 GB/s):
    Theoretical speed limit: $\frac{60 \text{ GB/s}}{5 \text{ GB}} \approx 12 \text{ tokens/sec}$.
  • On the RTX 4090 (memory speed 1,000 GB/s):
    Theoretical speed limit: $\frac{1000 \text{ GB/s}}{5 \text{ GB}} \approx 200 \text{ tokens/sec}$.

This is why the GPU outperforms the CPU in generative AI not so much due to clock speed or cores, but because of the colossal data transfer speed between the memory chip and computational units.

4. Production Engineering Scenarios

• For agent orchestration, code parsing, HTTP request routing, and database interactions, the CPU remains the irreplaceable master of the system.
• For weight matrix computations in neural networks, embedding vectorization, and media generation, the GPU with high memory bandwidth (VRAM) is the unequivocal standard.