Log by Time, not by Count
July 20, 2025
"How to Log" is a surprisingly deep topic in software engineering with many different viewpoints, and they're almost all valid in different situations. I'm going to argue that when processing lots of events, it's best to log every X seconds, rather than every X messages. This is a simple concept, but I've never seen it written down before.
Let's quickly look at some pseudocode to understand what I mean.
Count-based logging
num_events_processed = 0 while True : event = read_event_from_queue() process_event(event) num_events_processed += 1 if num_events_processed % 1_000 == 0 : module_logger.info( "Processed 1000 events." )
Time-based logging
last_time_logged = time.time() num_events_processed_since_last_log = 0 while True : event = read_event_from_queue() process_event(event) num_events_processed_since_last_log += 1 current_time = time.time() if current_time - last_time_logged >= 1.0 : module_logger.info( f"Processed {num_events_processed_since_last_log} events." ) num_events_processed_since_last_log = 0 last_time_logged = current_time
Default to writing time based logging.
Log rate should be consistent
... continue reading