On Monday 11/14/2022 from 2:04 PM to 3:36 PM Eastern Standard Time, Mobilize.us experienced an outage of our Domain Name Service provider.
Impacted users could not reach and therefore use any Mobilize-associated sites, custom domains, or navigate to Mobilize.us based URLs (such as virtual join links).
This affected:
Based on traffic patterns during the outage window, 70% of total site users were impacted and could not reach and therefore use any mobilize.us associated sites, custom domains, or navigate to mobilize.us based URLs (such as virtual join links)
The duration was 1h 34m, from around 2:04 PM to 3:36 PM Eastern (19:04 - 20:36 UTC) The users affected were not a predictable set of users, as this depended on: DNS caching on user’s browsers or other web agents (such as the Zoom app for Mobilize virtual join links).
30% of our users did not experience a disruption, for example if their browser cached and maintained Mobilize A records (IP addresses) before the incident began. Some users experienced a disruption for only certain domains or subdomains —The duration of effect for each user varied depending on client browser DNS caching policies.
Root cause was due to an outage in the Mobilize DNS servers hosted by a third party vendor resulting in DNS resolution failure. This DNS resolution failure made Mobilize.us (sites and URLs) inaccessible for a large percentage of our users.
To prevent this type of outage from recurring in the future, the Mobilize engineering team is working with the DNS provider to expand our failover and backup systems to strengthen the resiliency of Mobilize infrastructure.