At Heartbeat, we have a lot of different tasks that need to run at a particular time. Users can create draft posts or events that get published at a certain time. Event reminders need to be sent at a certain number of hours before an event. Automated workflows can be set up that send emails or direct messages after a delay.
For the longest time, all of these tasks were managed by a variety of cron scripts. We had createScheduledPosts.ts that would run every 15 minutes, scan our table of scheduled posts and create any that needed to be published. sendEventReminders.ts would run every single minute, scan our table of events, and send any notifications that needed to be sent out. And so on.
Each of these cron jobs would need to be managed independently. Whenever a new feature was added that involved running tasks in the future, a new cron job would be created. If one of the scripts started erroring, I’d need to figure out why, fix it and then figure out a way to retroactively run the tasks that were missed while the script was broken. Sometimes, we’d get reports from customers that a certain task that was supposed to run did not. I’d painstakingly dig into the logs & code, trying to figure out why a particular event reminder did not get sent on time. The first couple times this happened, I’d usually discover that we lacked the logs to even properly diagnose the issue. All I would be able to do is add some more logs and hope that I’d find the problem the next time. Once the logs were in place, I’d uncover some bug caused by timezones, improper error handling or who knows what else.
Eventually, I came to my senses and realized that all of these various cron jobs were doing the same thing. And rather than have 10 different cron jobs each implementing their own half-baked version of a task scheduler, we should just have a robust, centralized system for scheduling tasks.
The way it works is we have a single database table called ScheduledTasks with the following schema:
enum ScheduledTaskStatus { QUEUED EXECUTING COMPLETED } model ScheduledTask { id String @id communityID String createdAt DateTime lastStatusUpdate DateTime timestamp DateTime status ScheduledTaskStatus expectedExecutionTimeInMinutes Int expirationInMinutes Int? priority Int payload Json message String? @@index([status, timestamp]) }
payload is a discriminated union that contains each type of task we have. For example:
type ScheduledTaskPayload = | { type : " PUBLISH_EVENT " ; eventID : EventID ; } | { type : " PUBLISH_SCHEDULED_POST " ; scheduledPostID : ScheduledPostID ; } | { type : " SEND_EVENT_REMINDER " ; eventID : EventID ; } | { type : " SEND_EMAIL " ; email : string ; subject : string ; body : string ; };
Now, whenever we have a task that needs to be scheduled for the future, all we need to do is insert a new ScheduledTask into the database. We have a single cron job responsible for executing scheduled tasks that runs once every minute.
The cron job works as follows:
... continue reading