Website not loading for some users
Incident Report for Mobilize
Postmortem

Summary

On Monday 11/14/2022 from 2:04 PM to 3:36 PM Eastern Standard Time, Mobilize.us experienced an outage of our Domain Name Service provider.

Severity

Impacted users could not reach and therefore use any Mobilize-associated sites, custom domains, or navigate to Mobilize.us based URLs (such as virtual join links).
This affected:

  • The Mobilize web app (including all dashboards, feeds, and supporter surfaces for all organizations)
  • Any Mobilize-affiliated custom domains
  • Any mobilize related links were unusable

Impact

Based on traffic patterns during the outage window, 70% of total site users were impacted and could not reach and therefore use any mobilize.us associated sites, custom domains, or navigate to mobilize.us based URLs (such as virtual join links)

The duration was 1h 34m, from around 2:04 PM to 3:36 PM Eastern (19:04 - 20:36 UTC) The users affected were not a predictable set of users, as this depended on: DNS caching on user’s browsers or other web agents (such as the Zoom app for Mobilize virtual join links).

30% of our users did not experience a disruption, for example if their browser cached and maintained Mobilize A records (IP addresses) before the incident began. Some users experienced a disruption for only certain domains or subdomains —The duration of effect for each user varied depending on client browser DNS caching policies.

Root cause

Root cause was due to an outage in the Mobilize DNS servers hosted by a third party vendor resulting in DNS resolution failure. This DNS resolution failure made Mobilize.us (sites and URLs) inaccessible for a large percentage of our users.

What steps are we taking to prevent this from happening again?

To prevent this type of outage from recurring in the future, the Mobilize engineering team is working with the DNS provider to expand our failover and backup systems to strengthen the resiliency of Mobilize infrastructure.

Posted Nov 18, 2022 - 13:44 EST

Resolved
A Domain Name Service (DNS) outage with our upstream provider prevented ~70% of users from accessing www.mobilize.us from ~1:40PM Eastern to 3:40PM Eastern when service was restored. We have continued to closely monitor the site and have not seen a repeat of the issue. Once a root cause for the outage is determined we will publish a post mortem.
Posted Nov 14, 2022 - 17:55 EST
Update
This issue is resolved and the site is fully operational. We are continuing to investigate the root cause.
Posted Nov 14, 2022 - 15:39 EST
Update
We are continuing to investigate.
Posted Nov 14, 2022 - 15:08 EST
Investigating
We are currently investigating this issue. The issue seems to be affecting some but not all users.
Posted Nov 14, 2022 - 14:35 EST
This incident affected: Website.