Skip to content
Ephizen logo ephizen / wiki
CS 502

async / await Fundamentals

Coroutines, the event loop, and what `await` actually does — with no hand-waving.

Software Engineering Practices intermediate #python #async #concurrency

TL;DR

async / await is concurrency for I/O-bound work. The runtime — an event loop — runs many coroutines on a single thread, switching between them whenever one awaits (i.e. asks to wait for I/O). While one coroutine is parked waiting for the network or disk, the loop runs other coroutines that have work to do.

This is not parallelism. There is one OS thread, executing one piece of Python at a time. The win is that I/O latency is overlapped — 1000 LLM API calls don’t take 1000× longer than one, because most of the time is spent waiting for responses, and the loop fills those gaps with other calls.

The four words you need: coroutine (an async function or its return value), await (give control back to the loop until this operation completes), task (a coroutine scheduled to run by the loop), event loop (the thing that runs everything).

When to use async: making N concurrent network calls (LLM APIs, HTTP fetches, DB queries). When not to use async: CPU-bound work (matmul, parsing, image processing) — the GIL means you don’t get parallelism, and async adds complexity for no benefit. See PY 303.

The picture in your head

A regular function runs from start to finish, blocking on every I/O call. While it’s blocked waiting for a response, the CPU sits idle.

A coroutine is the same code, except every await is a “I’m going to wait — go do something else, wake me up when ready.” The event loop maintains a queue of coroutines and switches between them at every await point. When the OS tells the loop “the response for coroutine A is ready”, it picks A back up where it left off.

sync (1 worker, 5 sequential calls, 100ms each):
[call 1: 100ms][call 2: 100ms][call 3: 100ms][call 4: 100ms][call 5: 100ms]
                                                                Total: 500ms

async (1 worker, 5 concurrent calls, 100ms each):
[call 1 sent...]
 [call 2 sent...]
  [call 3 sent...]
   [call 4 sent...]
    [call 5 sent...]
     [...all wait for ~100ms...]
                              [responses arrive ~simultaneously]
                                                                Total: ~110ms

Same single thread. The wall-clock difference is the gap that the event loop fills with overlapping work.

A first coroutine

import asyncio
import time

async def fetch(url):
    print(f"start {url}")
    await asyncio.sleep(1)         # pretends to be a network call
    print(f"done {url}")
    return f"<html for {url}>"

async def main():
    t0 = time.perf_counter()
    results = await asyncio.gather(
        fetch("https://a"),
        fetch("https://b"),
        fetch("https://c"),
    )
    print(f"all done in {time.perf_counter() - t0:.2f}s")
    return results

asyncio.run(main())

Output:

start https://a
start https://b
start https://c
done https://a
done https://b
done https://c
all done in 1.00s

Three “1-second” calls completed in 1 second total because they ran concurrently. The serial version would take 3.

The vocabulary, exactly

Coroutine function: a function defined with async def. Calling it doesn’t run it — it returns a coroutine object.

async def f():
    return 42

c = f()                  # coroutine object; nothing executed yet
print(c)                  # <coroutine object f at ...>

Coroutine object: the thing you get from calling an async def. It must be awaited or scheduled, or Python warns “coroutine was never awaited.”

await: only valid inside async def. Expects a awaitable — another coroutine, a Task, or a Future. Yields control to the event loop until the awaitable is done.

Task: a coroutine wrapped to be scheduled by the event loop. Created by asyncio.create_task(coro). Tasks start running immediately; unlike a bare coroutine, they don’t need to be awaited to start.

Event loop: the runtime. asyncio.run(main()) creates one, runs main until it completes, then closes it.

asyncio.run — the entry point

async def main():
    ...

asyncio.run(main())

This creates a new event loop, runs main() to completion, and shuts down. Use it once at the top of your program. Don’t call it from inside a coroutine — it would create a nested loop, which is invalid.

If you’re inside Jupyter, the loop is already running, so use await main() directly (Jupyter cells run in an event loop).

gather vs create_task vs sequential await

Three ways to run multiple coroutines:

# 1) Sequential — slow, one at a time
async def slow():
    a = await fetch("a")
    b = await fetch("b")
    c = await fetch("c")
    return a, b, c
# Total: 3 * fetch_time

# 2) gather — fan out, fan in
async def fast_gather():
    return await asyncio.gather(fetch("a"), fetch("b"), fetch("c"))
# Total: max(fetch_times)

# 3) create_task — start now, await later
async def fast_tasks():
    ta = asyncio.create_task(fetch("a"))
    tb = asyncio.create_task(fetch("b"))
    tc = asyncio.create_task(fetch("c"))
    # ... do other work here ...
    return await ta, await tb, await tc
# Total: max(fetch_times)

gather is the canonical fan-out pattern. create_task is for when you want to start work in the background and gather later, possibly mixed with other operations. The sequential form is the trap — looks async, runs serial.

gather with error handling

By default, gather cancels other tasks and re-raises the first exception:

results = await asyncio.gather(t1, t2, t3)   # raises if any fails

To collect all results including exceptions:

results = await asyncio.gather(t1, t2, t3, return_exceptions=True)
# results contains either values or exception instances

For batches where partial failure is expected (LLM calls!), this is the right form. Filter out the exceptions afterwards:

ok = [r for r in results if not isinstance(r, Exception)]
errs = [r for r in results if isinstance(r, Exception)]

TaskGroup — the modern fan-out (3.11+)

async def main():
    async with asyncio.TaskGroup() as tg:
        t1 = tg.create_task(fetch("a"))
        t2 = tg.create_task(fetch("b"))
        t3 = tg.create_task(fetch("c"))
    # All tasks done here. Any exception cancels the others and re-raises.
    print(t1.result(), t2.result(), t3.result())

Cleaner than gather for structured concurrency. Use it on Python 3.11+.

Awaiting a sync function — to_thread

What if you need to call a blocking function inside an async program? A requests.get(...) would block the entire event loop, freezing every other coroutine. Use asyncio.to_thread:

async def fetch_with_blocking_lib(url):
    return await asyncio.to_thread(requests.get, url)

to_thread runs the function in a thread-pool executor and awaits the result, freeing the event loop to run other coroutines while the thread handles the blocking call.

For network code, prefer the async-native library (httpx instead of requests, aiofiles instead of open, async DB drivers) — fewer threads, less overhead. to_thread is the bridge for code you can’t rewrite.

Common gotchas

  • Calling an async function without await. fetch(url) returns a coroutine object that does nothing. You’ll get a runtime warning.
  • Calling sync I/O from a coroutine. requests.get blocks the event loop; every other coroutine waits. Use httpx, aiohttp, or asyncio.to_thread.
  • time.sleep in a coroutine. Same problem — blocks the loop. Use asyncio.sleep.
  • Forgetting to await gather. asyncio.gather(a, b) returns a future; you have to await it.
  • Mixing event loops. Don’t call asyncio.run from inside a coroutine. Don’t keep references to coroutines across loop closes.
  • Unbounded concurrency. await asyncio.gather(*[fetch(u) for u in urls]) with 100,000 URLs spawns 100,000 concurrent connections, hitting rate limits and OOM. Use asyncio.Semaphore to bound concurrency. See PY 302.
  • CPU-bound work in async. Doesn’t speed up; just adds overhead. Use multiprocessing for parallel CPU.

When async is wrong

  • CPU-bound code. np.fft.fft, model.forward(x), parsing JSON. No speedup; the GIL holds everything to one thread anyway. Use multiprocessing.
  • A small number of calls. 5 sequential network calls done serially might be 500ms instead of 100ms async. If that’s acceptable, sync is simpler.
  • Existing sync codebase. Async is contagious — once you go async at the leaf, every caller has to be async too. Mixing requires to_thread / run_in_executor and gets ugly.

Where async shows up in real ML codebases

  • LLM API fan-out. The canonical use case — see PY 302.
  • Web servers. FastAPI, Litestar, Sanic — all async. Every endpoint is async def.
  • Data fetching. Pull from N S3 buckets / N database shards concurrently while training.
  • Streaming responses. SSE / chunked transfer for incremental LLM output.
  • Concurrent eval. Run N evaluations in parallel against a shared prompt.
  • Webhook handlers. Inbound events fire concurrent downstream calls.

The defensive habit: when you reach for async, ask “is the bottleneck network/disk wait, or is it computation?” Async helps the first; it doesn’t help the second.

Resources