The GIL — threading vs multiprocessing vs asyncio

TL;DR

CPython has a Global Interpreter Lock (GIL) — a mutex that ensures only one OS thread executes Python bytecode at a time. This single fact decides which concurrency primitive you reach for:

Picking the wrong tool: 10–100× speed left on the table. Picking threading for CPU-bound pure-Python: zero speedup, sometimes slower due to GIL contention. Picking multiprocessing for I/O-bound: heavy, slow startup, painful inter-process communication.

Python 3.13 ships an experimental free-threaded build (no GIL) under PEP 703. Production adoption will take years. For now, plan around the GIL.

The picture in your head

The GIL is a single lock owned by the interpreter. Whichever Python thread is currently executing bytecode holds the lock. Every ~100 bytecodes (or 5 ms, configurable), the holder is preempted and the lock is released, and another thread gets a turn. Only one thread is running Python at a time, ever.

Implications:

• Two threads doing pure-Python addition do not run in parallel. They take turns, with overhead from the lock contention. Often slower than one thread.
• A thread blocked in socket.recv() or time.sleep() releases the GIL while it waits. Another thread can run. → Threading helps for I/O.
• A thread inside numpy.matmul or torch.matmul releases the GIL while the C/CUDA code runs. Another thread can run pure Python in parallel. → Threading can help for “calls into C” work.
• Multiple processes have multiple GILs (one each, in their own interpreter). They run in true parallel. → Multiprocessing for CPU-bound pure Python.

When to use what — the decision table

Question 1: Is your bottleneck I/O (network, disk)?
  YES → asyncio (preferred for many concurrent calls)
        or threading (simpler if you have sync libraries)

  NO  → continue

Question 2: Is your inner loop already in NumPy/PyTorch/scikit-learn/C?
  YES → no concurrency needed; the C library is parallel
        (or use threading for multiple parallel calls)

  NO  → continue

Question 3: Is the workload CPU-bound, pure Python, on multiple inputs?
  YES → multiprocessing (or rewrite in NumPy/Numba)

  NO  → you don't need concurrency

Threading — the concurrency that often disappoints

from concurrent.futures import ThreadPoolExecutor
import time

def cpu_work(n):
    # Pure-Python loop — GIL bound
    total = 0
    for i in range(n):
        total += i * i
    return total

def io_work(n):
    time.sleep(1)               # simulates I/O
    return n

# CPU-bound — threading does NOT help
with ThreadPoolExecutor(max_workers=8) as ex:
    list(ex.map(cpu_work, [10**7] * 8))    # ~8s, same as serial

# I/O-bound — threading absolutely helps
with ThreadPoolExecutor(max_workers=8) as ex:
    list(ex.map(io_work, [None] * 8))      # ~1s, not 8

Two takeaways:

1. CPU + threads = no speedup. Sometimes a tiny slowdown from contention.
2. I/O + threads = parallel waiting. The threads spend almost all their time blocked on I/O, releasing the GIL, so they overlap.

For the I/O case, asyncio is usually a better fit — no per-thread memory cost, scales to thousands. For sync-only legacy code, threads remain a valid option.

Multiprocessing — true parallel CPU

from concurrent.futures import ProcessPoolExecutor
from multiprocessing import freeze_support

def cpu_work(n):
    total = 0
    for i in range(n):
        total += i * i
    return total

if __name__ == "__main__":
    freeze_support()                          # required on Windows
    with ProcessPoolExecutor(max_workers=8) as ex:
        results = list(ex.map(cpu_work, [10**7] * 8))

Each worker is a separate Python process with its own interpreter, its own GIL, its own memory. They run in true parallel.

Costs to know:

• Startup is expensive. Each process forks (or spawns) and imports the script. ~50–200ms per worker on first call.
• Argument passing is by pickle. Anything you pass to a worker gets serialised, sent over a pipe, deserialised. Don’t pass huge arrays; pass file paths and let the worker load them.
• Return values get pickled too. Same concern in reverse.
• No shared memory by default. Workers can’t see your main process’s variables.

For shared large data (model weights, big arrays), use multiprocessing.shared_memory or multiprocessing.Manager, or memory-map the file from disk in each worker (and rely on the OS page cache).

When threading actually shines for ML — wrapped C code

If you’re calling numpy.matmul, torch.matmul, sklearn.fit, or any function whose heavy work is in C/CUDA, that function releases the GIL while it runs. Other threads can execute Python in parallel during that time.

from concurrent.futures import ThreadPoolExecutor
import numpy as np

A = np.random.standard_normal((2000, 2000))

def matmul_chunk(_):
    return A @ A    # heavy lifting in BLAS, releases GIL

with ThreadPoolExecutor(max_workers=8) as ex:
    list(ex.map(matmul_chunk, range(8)))    # actually parallel!

This is why scikit-learn’s n_jobs=-1 parameter (using joblib) often defaults to threading: the heavy bits (fit, predict) are in Cython/C, not pure Python.

If your CPU work is mixed — some Python overhead, some C work — you typically still get a meaningful speedup from threading because the C parts run in parallel.

The 3.13 free-threaded build

PEP 703 made the GIL optional in CPython 3.13. With a free-threaded build (python3.13t), threads run in true parallel — including for pure-Python CPU work.

Caveats today:

• Most C extensions are not yet thread-safe under no-GIL. NumPy and PyTorch are working on it; many smaller packages aren’t.
• Single-threaded performance is ~10% slower than the GIL build (the no-GIL build pays for finer-grained locking).
• Production adoption needs the ecosystem to catch up — likely a multi-year transition.

For new code, plan for the GIL today and follow the no-GIL transition as it matures.

A worked decision — embedding 1M strings

The problem: 1 million strings, embed each with a sentence-transformer model running on GPU.

The lesson: when the bottleneck is the GPU, more processes don’t help. The win is batching and feeding it efficiently. Don’t reach for multiprocessing reflexively — figure out the actual bottleneck.

When you do have multiple GPUs, then multi-process (or torchrun’s DDP) is the right answer — one process per GPU.

Common gotchas

• max_workers set too high. Threads or processes both have overhead. For threading, ~10× CPU count is usually the practical ceiling. For multiprocessing, ~CPU count is right.
• Heavy pickle in ProcessPoolExecutor.map. Sending 100MB arrays per task pickles, sends, unpickles every time. Slow. Use shared memory or pass references (paths) instead.
• Importing huge libraries on every worker. Expensive imports happen per process. Use initializer= to import once per worker.
• fork vs spawn. On Linux, default is fork (cheap, but copies all state including open file handles, hard with CUDA). On macOS / Windows, default is spawn (full re-import, safer with CUDA). Set explicitly with multiprocessing.set_start_method("spawn") if unsure.
• CUDA + fork = chaos. PyTorch documents this clearly: don’t use fork start method with CUDA initialised; use spawn.
• Shared mutable state between threads. Threads in the same process share memory. Use threading.Lock, Queue, or concurrent.futures to avoid races. The GIL prevents some races (atomic bytecodes) but not all (compound operations).

Quick reference

Where this shows up in real ML codebases

• PyTorch DataLoader(num_workers=N) — multiprocessing for parallel data loading, sidestepping the GIL.
• scikit-learn n_jobs=-1 — threading via joblib (most ops release the GIL).
• Hugging Face Dataset.map(num_proc=8) — multiprocessing for parallel preprocessing.
• torchrun / accelerate — multiprocessing for distributed training (one process per GPU).
• FastAPI workers — uvicorn --workers N runs N processes (each serving async coroutines internally).
• Airflow / Prefect — orchestrate independent tasks across processes / containers.

The defensive habit: before reaching for any concurrency primitive, profile (see PY 404) to identify the bottleneck. CPU-bound? Memory-bound? GPU-bound? I/O-bound? Each has a different right answer. Adding concurrency to the wrong one wastes your time and slows the code.

Resources

• Python docs — threading — docs.python.org.
• Python docs — multiprocessing — docs.python.org.
• Python docs — asyncio — docs.python.org.
• PEP 703 — Making the GIL Optional — peps.python.org — the no-GIL plan.
• David Beazley — Understanding the Python GIL — dabeaz.com — the canonical talk; old but still accurate.
• PyTorch — multiprocessing best practices — pytorch.org — CUDA + fork warnings, shared tensors.