Degraded performance of background tasks

Summary

Following a recent incident impacting task performance, we conducted a thorough review of our task management infrastructure. This analysis surfaced several key areas of improvement that have since been addressed to ensure better system reliability and performance.

‌

What Happened

The incident revealed that certain tasks within the Offstreet system were unnecessarily interconnected, leading to performance bottlenecks. In some cases, a single intensive task - in this case, the generation of the new reports - could degrade the performance of unrelated tasks running in parallel.

‌

What We Learned

• We identified architectural bottlenecks that allowed tasks to impact one another.
• Some reports being generated were significantly larger than anticipated — reaching millions of rows and several megabytes in size.

‌

Actions Taken

• Removed interdependencies between tasks to isolate workloads and prevent cascading performance issues.
• Initiated efforts to better handle large reports, improving both performance and user experience when working with large datasets.
• Temporarily rolled back to the old Report functionality until we are confident that the new reports can handle production load

‌

Next Steps

We are continuing to monitor and optimize report generation, including:

• Better handling of extremely large reports
• Investigating further overall efficiency of our task execution engine that have been applied to the report generation, that we believe can be applied to other tasks as well
• Enhancing user feedback and progress visibility for long-running tasks
• Internal postmortem to better understand how to prevent impact on our users (small or large) in the future

‌

Conclusion

This incident led to meaningful improvements in the robustness and performance of our platform. We’re committed to continuously refining our infrastructure to ensure a reliable and efficient experience for all Offstreet users. The new reporting feature is highly anticipated, and we’re excited to bring it back online soon - better than ever - and continue gathering feedback to make it even more useful.