Degraded performance of background tasks

Incident Report for Offstreet

Postmortem

We deployed a system update this morning at 8:45am that unexpectedly caused our task manager to stop processing jobs. Between 8:45am and 12:54pm, the following functionality was impacted:

  • Reports were unavailable
  • Email and SMS notifications were delayed until systems were back fully operational
  • Parkers registering for event permits during the window were sent delayed to integration partners when systems were back fully operational

    • Non event permits were not impacted
  • Bulk jobs (e.g. permits, locations, tenants) were stuck in pending but not lost

We rolled back the update at 12:50pm, and by 12:56pm the task manager was fully operational and processing new jobs. No data was lost, anything that was not processed was processed by 12:56pm. All systems are stable again, and no further impact is expected.

Posted Sep 18, 2025 - 14:41 CST

Resolved

A fix has been implemented and all systems are back operational.
Posted Sep 18, 2025 - 12:58 CST

Identified

The issue has been identified and our team is working on a solution.
Posted Sep 18, 2025 - 11:10 CST

Investigating

We are currently investigating an issue with our background task job. This impacts Bulk upload tasks, emails, and report generation.
Posted Sep 18, 2025 - 10:35 CST
This incident affected: Offstreet Dashboard, Offstreet Core, and Offstreet Reporting.