Confirming a hunch

I landed two small performance fixes in Quinn, a pure-Rust, async-compatible implementation of the IETF QUIC transport protocol. Neither was a heroic rewrite; what made them worth writing up is how they got confirmed: a user had a hunch backed by a profile, and I was able to help turn that into something a maintainer could act on.

Issue #2720

EileenGalvin was running Quinn on an edge device and their heap profile showed around 75% of allocation blocks going through a BTreeMap during a 1000 Mbps transfer; they had prototyped a replacement and explained why it wasn’t quite right.

Plausible is not confirmed

Ralith asked to see the stack traces extended far enough to actually confirm the source and djc asked whether the profile had traces into Quinn’s own code at all: the hypothesis was reasonable, but the backtraces stopped inside the standard library’s BTreeMap internals and never reached Quinn.

So I re-ran the profile with deeper backtraces, using dhat with enough frames to resolve past std and into Quinn, on a release build over a loopback transfer. That pinned it to one specific map: PacketSpace::sent_packets. I posted the method so others could reproduce it. djc’s reply was short: happy to review a PR.

Verifying before believing

I never trusted a number in isolation. For allocations I used dhat, and then a counting allocator inside the real benchmark client to check that the microbenchmark result held in the actual binary. For throughput I ran interleaved A/B tests on loopback to cancel drift, then repeated across a netem veth pair at 10 and 40 ms of delay with a little loss. Later I added CPU profiling with perf and frame pointers to see where the cycles actually went.

The results were not uniform: the send-path fix genuinely moved loopback throughput, around 8% on download, while the receive-path fix did not move throughput on my x86 box. An allocation and free of these small buffers costs about 25 nanoseconds, so cutting tens of thousands of them per second is well under 1% of CPU on my server. The benefit only shows up where allocation is a bigger slice of the budget, which is exactly the edge device the report came from.

Two traps I walked into:

• My first receive-path attempt was a zero-copy arena, and it ballooned total allocated bytes from about 225 MB to 17.6 GB, because under sparse packet arrival each batch grabbed the whole worst-case buffer for a handful of real datagrams. I threw it out and kept a simpler design.
• One netem run showed a 20~30% improvement that evaporated the moment I re-ran it with more rounds. It was noise. The rule I took away: interleave your runs, use enough of them, and check that the knob you turned actually did something. I also found that netem’s packet reordering did nothing at all on my machine, which quietly invalidated a comparison I had almost trusted.

The fixes

The first PR, #2730, replaced the BTreeMap with a sparse ring buffer (suggested by Ralith) indexed by packet number. Packet numbers are monotonic, so insertion becomes an amortized constant-time append and removal is constant time. I kept the old map’s API so the call sites barely changed and the existing simulation tests still exercised the ordering logic. Allocations for that map dropped from about 24,000 to 18, and total allocations fell by roughly 40%.

The second PR, #2735, kept one reusable receive buffer, reserved space in it each poll, and split each datagram off as an owned slice sharing that one allocation. Empty polls stop allocating entirely. This was the shape EileenGalvin had suggested. Receive-path allocations fell from about 38,000 blocks to 6,700, and whole-client allocations per gigabyte dropped 72%.

The best part came after. EileenGalvin reported that the two changes together took their edge device from 100% to 60% peak CPU at the same throughput. The person who filed the report validated the fix on the hardware that motivated it.

What I took from it

• A profile is a hypothesis until the backtraces reach your own code.
• Allocation count, CPU, and throughput are different quantities, and a win in one does not imply a win in another.
• Being honest that a metric didn’t move, rather than hunting for one that did, is what let the numbers line up with someone else’s experience that motivated the entire exercise.

The maintainers were quick and constructive throughout, and a bug report turned into a productive collaboration. That’s the version of open source I keep coming back for.