GoKawiilTokio Gives Progress, Not Ordering: Scheduling 1M Tasks
This is a prequel to my previous article, Your Rust Service Isn't Leaking — It Could Be the Allocator.
In that post, I wrote about how a few memory allocators behaved differently under our workload. Before we figured out that the memory behavior was tied to allocators, we first tried to reduce the memory usage from the application side.
Our service was event-driven:
Read events from a message queue (Kafka/Redis Streams/NATS) For each event, spawn a Tokio task to process it
Our Task Pattern
In our workload, each event had a maximum of 1000 user tokens. For every user token, we had to make an outbound call, wait on I/O and collect all responses belonging to the event.
Here is a simplified version of the code where we fan-out tokio tasks for user token, fan-in the responses, and generate a response event.
struct Event { payload : Bytes , user_tokens : Vec < String > , } ... loop { let event : Event = fetch_next_event ( ) . await ; tokio :: spawn ( async move { let data = event . payload . clone ( ) ; let mut tasks = JoinSet :: new ( ) ; for token in & event . user_tokens { let token = token . clone ( ) ; let data = data . clone ( ) ; tasks . spawn ( async move { process ( token , data ) . await } ) ; } let mut responses = Vec :: with_capacity ( event . user_tokens . len ( ) ) ; while let Some ( res ) = tasks . join_next ( ) . await { responses . push ( res ) } generate_response_event ( event , responses ) ; } ) ; }
Our primary focus here was throughput. While the fan-out above is unbounded, I assumed it practically wouldn't be a problem. These Tokio tasks are short-lived. They finish and drop once they receive responses, usually within a few milliseconds. So I assumed tasks spawned early would also finish early. Though individual outbound calls could finish out of order, I expected the earlier events overall to finish first, even while newer events were still coming in.
... continue reading