How to Profile and Fix Performance Issues in Asyncio Apps: A Practical Guide for Developers

Asyncio apps are powerful: they let you handle thousands of concurrent connections on a single process. But when latency, p95/p99 spikes, or memory growth appear, debugging feels like chasing ghosts — tasks are suspended and resumed across await points, the event loop is invisible, and traditional profilers can mislead.

This guide gives you a production-safe, step-by-step playbook to profile async Python, use low-overhead tools like py-spy, combine sampling with tracing, capture coroutine stacks, and apply practical fixes. You'll get checklists, copy-paste commands, code snippets, and real case studies so you can quickly diagnose async performance and prevent regressions.

Why async performance problems are different: common causes and symptoms

Event loop fundamentals that matter for performance

In asyncio, a single event loop schedules coroutines and callbacks. If any piece of code blocks the loop (e.g., CPU-bound work, time.sleep, or synchronous I/O), everything stalls until that work completes. Conversely, too many concurrent tasks or excessive context switching can raise latency and memory pressure. Understanding where time is spent — running code vs awaiting I/O — is the core of asyncio profiling.

Symptoms: latency, p99 spikes, CPU vs I/O vs blocking

High overall CPU usage and degraded throughput — likely CPU-bound work inside the process.
Increased tail latency (p95/p99) while p50 is normal — could be scheduling contention, blocking calls, or head-of-line blocking.
Steady memory growth — possible task leak or retained objects; use tracemalloc.
Sudden latency spikes correlated with external systems — network I/O or threadpool starvation.

Quick checklist: what to observe before profiling

Before you start heavy instrumentation, collect low-overhead evidence and create a reproducible scenario.

Metrics: request latency (p50/p95/p99), throughput, CPU, memory, file descriptors, and threadpool queue size.
Logs: correlate timestamps, trace IDs, and unusual stack traces or timeouts.
Reproducible scenario: can you reproduce under load locally or in a staging environment?
Minimal repro: isolate the subsystem (HTTP handler, background worker) to profile small scope.

Collect these quick artifacts for the incident ticket: top or htop, ps, process PID, a short stack dump, and a sampling profile (30s). These are cheap to capture and often enough to point you in the right direction.

Tools and setup: profilers, memory tools and Docker/CI tips

Here’s a concise toolset and when to use each:

py-spy — low-overhead sampling profiler that can attach to running Python processes. Great for production snapshots and flame graphs. (keyword: py-spy asyncio)
cProfile — deterministic tracer for focused runs (higher overhead). Good for local targeted profiling.
yappi — supports CPU and wall-clock profiling, threads, and can save pstats for analysis.
pyinstrument / pyflame — alternative sampling tracers for quick stacks.
tracemalloc — track memory allocations and snapshots to find leaks.
System tools: perf, eBPF (bcc/bpftrace), and ss/netstat for kernel-level correlation.
Tracing: OpenTelemetry / Jaeger / Zipkin for distributed traces and async span propagation.

Integrating with Docker and CI

To attach py-spy to a containerized process:

docker exec -it  sh -c 'py-spy record -o /tmp/profile.svg --pid 1 --duration 30'

For CI: add lightweight microbenchmarks and smoke benchmarks that run under a --profile flag to capture cProfile dumps; store artifacts in CI for retention and comparison.

Step-by-step asyncio profiling workflow (sampling vs tracing)

Use sampling first (low risk). If you need exact call counts or micro-timings, use tracing on a limited scope.

1) Quick, production-safe sampling (py-spy)

Attach py-spy for a short period (10–60s) during the problematic window to capture a flame graph and call hotspots.

Command (on host):

py-spy record -o profile.svg --pid  --duration 30 --rate 100

Or interactive top:
```
py-spy top --pid 
```

py-spy is non-intrusive and safe in production — it reads process memory without modifying the target. Look for heavy Python functions, or unexpected blocking functions (e.g., time.sleep, sync DB client calls).

2) Correlate with asyncio task stacks

Sampling shows which functions are hot, but you also want to see where tasks are suspended vs running. Use a programmatic task dump:

import asyncio, traceback, sys

async def dump_all_tasks():
    loop = asyncio.get_running_loop()
    tasks = asyncio.all_tasks(loop)
    print(f"Total tasks: {len(tasks)}")
    for t in tasks:
        print('Task:', t)
        for frame in t.get_stack(limit=10):
            traceback.print_stack(frame, file=sys.stdout)

# Run this as an admin/debug endpoint or attach via a debug shell

This reveals coroutines stuck at await boundaries or repeatedly re-scheduling.

3) Targeted tracing with cProfile or yappi

When you have a small reproducible scenario, run a deterministic profiler to measure function-level timings:

python -m cProfile -o out.prof -c 'run_my_smoke_test()'
# Then analyze with snakeviz or pstats

Use cProfile only on scoped tests — avoid running it on production steady state due to overhead.

4) Memory investigation with tracemalloc

import tracemalloc
tracemalloc.start()
# ... run workload
snapshot1 = tracemalloc.take_snapshot()
# after some time
snapshot2 = tracemalloc.take_snapshot()
for stat in snapshot2.compare_to(snapshot1, 'lineno')[:20]:
    print(stat)

This shows allocations by file/line to find leaks. Combine with task dumps to find orphaned tasks retaining objects.

5) Correlate with system metrics (epoll, threadpool)

If your threadpool is saturated, you may see latency even when CPU is low. Monitor ThreadPoolExecutor queue lengths (instrument), OS-level socket backlogs, and epoll wait counts. Use ss -s and perf or eBPF traces to find kernel-level bottlenecks.

Practical examples & recipes

Example: event-loop blocking and the fix

Problematic code (blocks event loop):

import time

async def handle(req):
    # BAD: blocks event loop
    time.sleep(0.5)
    return 'ok'

How it shows up: p99 latency spikes, py-spy shows time.sleep on the main thread, and task dump shows many tasks waiting behind the blocker.

Fix: run blocking call in an executor:

import asyncio, time

async def handle(req):
    loop = asyncio.get_running_loop()
    await loop.run_in_executor(None, time.sleep, 0.5)
    return 'ok'

Example: hidden CPU-bound work discovered by sampling

Symptom: CPU high, throughput low. py-spy shows heavy Python time inside json.dumps on large payloads.

Fixes:

Move serialization to a process pool / separate worker.
Stream responses instead of building large objects in memory.
Use faster serializer (e.g., orjson) if compatible.

Example: fixing excessive task creation

Problem: incoming stream spawns a task per item without bounds -> memory growth and scheduling churn.

Diagnosis: asyncio.all_tasks() shows thousands of pending tasks; p95 latency increases as concurrency grows.

Fix: bound concurrency with a semaphore or worker queue:

sem = asyncio.Semaphore(100)

async def process_item(item):
    async with sem:
        await do_io(item)

# Or use a queue with a fixed number of worker coroutines

Real-world case studies

Here are three short postmortems you can reproduce locally and learn from.

1) Event-loop blocking from a sync DNS client

Symptom: p99 latency spikes after a library upgrade. Sampling shows synchronous socket.getaddrinfo calls on the main thread. Fix: switch to an async DNS resolver or run the DNS lookup in an executor. Post-fix: p99 dropped from 1.2s to 60ms.

2) Hidden CPU-bound JSON serialization

Symptom: throughput drop with many large responses. Flame graph shows json.dumps consuming most CPU cycles. Fix: stream JSON via generator and use orjson for speed. Post-fix: throughput doubled and CPU normalized.

3) Runaway task creation in a websocket consumer

Symptom: memory growth and GC pressure. Investigation found per-message coroutines spawned but not awaited on error paths. Fix: add proper task tracking, cancel unneeded tasks, and enforce concurrency limits with a queue. Post-fix: memory stabilized and latency dropped.

Optimization patterns and best practices

Use executors for blocking or CPU-bound work — run_in_executor for short blocking calls; process pools for heavy CPU tasks.
Bound concurrency — semaphores or worker queues prevent overload and preserve tail latency.
Batch I/O when possible — fewer syscalls and less scheduling overhead.
Prefer non-blocking libraries — async DB drivers, async HTTP clients, and async-friendly SDKs.
Correct await usage — don’t forget to await coroutines, and avoid accidentally creating tasks without tracking them.
Backpressure — propagate rate limits upstream and reject early when the service is saturated.
Use uvloop when appropriate — uvloop often improves throughput and latency, but re-run your profiling because behavior can change.

Verification, monitoring and preventing regressions

After applying fixes, verify with targeted tests and add continuous observability to detect regressions early.

Automated benchmarks: add synthetic load tests (locust, wrk, or a custom async runner) in CI gating changes that affect latency.
Continuous profiling: use low-overhead continuous profilers (pyroscope, parca) or periodic py-spy snapshots to trend hotspots over time.
Tracing and metrics: instrument handlers with OpenTelemetry to capture spans and correlate with external calls and p99 traces.
Task and threadpool metrics: expose the number of in-flight tasks, task creation rate, and ThreadPoolExecutor queue size to Prometheus.
On-call cheat sheet: keep a one-page with minimal triage commands (below).

On-call quick triage commands (cheat sheet)

# PID and top processes
ps aux | grep myapp
top -p 

# py-spy flamegraph for 30s
py-spy record -o /tmp/profile.svg --pid  --duration 30

# interactive py-spy top
py-spy top --pid 

# dump asyncio tasks (run inside app or via debug endpoint)
# see code snippet earlier (dump_all_tasks)

# quick memory snapshot (if enabled in-app with tracemalloc)
# use saved snapshots to compare

Keep these commands in an incident playbook so responders can act quickly without guessing.

Conclusion

Profiling and diagnosing performance issues in asyncio apps require a different mindset than synchronous programs. Start with low-risk sampling (py-spy), then combine task dumps, targeted tracing (cProfile / yappi), and memory snapshots (tracemalloc) to form and validate hypotheses. Use bounded concurrency, executors, and non-blocking libraries to remove common performance pitfalls, and instrument your service with continuous profiling and tracing so regressions are caught early.

If your team wants a ready-to-run toolkit, build a small diagnostics container with py-spy, a debug endpoint for task dumps, and CI performance tests. For step-by-step reproducible examples and a starter repo, see related topic.

Found this helpful? Try the sample snippets in a small sandbox, run a short py-spy capture on a staging instance, and share your toughest async performance problem — I’ll help turn it into a reproducible case study.

Call to action: Bookmark this guide, add the on-call cheat sheet to your incident runbook, and consider adding continuous profiling to your observability stack. Questions or want a walkthrough on a specific incident? Reply below or follow related topic for deeper tutorials.

Search This Blog

Simplify Your Day As Developer