Official Blog
Built in the cloud. Engineered for your enterprise.
What Happened to Google Docs on Wednesday
Friday, September 9, 2011
Posted by Alan Warren, Engineering Director
(Cross-posted from the
Google Docs Blog
.)
Not our best week. On Wednesday we had an outage that lasted one hour and meant that document lists, documents, drawings and Apps Scripts were inaccessible for the majority of our users. We use Google Docs ourselves every day, so we feel your pain and are very sorry.
So what happened? The outage was caused by a change designed to improve real time collaboration within the document list. Unfortunately this change exposed a memory management bug which was only evident under heavy usage.
Every time a Google Doc is modified, a machine looks up the servers that need to be updated. Due to the memory management bug, the lookup machines didn’t recycle their memory properly after each lookup, causing them to eventually run out of memory and restart. While they restarted, their load was picked up by the remaining lookup machines - making them run out of memory even faster. This meant that eventually the servers couldn’t properly process a large fraction of the requests to access document lists, documents, drawings, and scripts which led to the outage you saw on Wednesday.
Our automated monitoring noticed that attempts to access documents were failing at an increased rate, and alerted us 60 seconds later after the failure rate increased sharply. The engineering teams diagnosed the problem, determined that it was correlated with the feature change, and started rolling it back 23 minutes after the first alert. In parallel, we doubled the capacity of the lookup service to mitigate the impact of the memory management bug. The rollback completed 24 minutes later, and 5 minutes after that the outage was effectively over as the additional capacity restored normal function.
Since resolution, we have been assembling and scrutinizing the timeline of this event, and have assembled a list of steps which will both reduce the chance of a future event, decrease the time required to notice and resolve a problem, and limit the scope which any single problem can affect. We intend to take all these steps; some are not easy, but we're committed to keeping Google's services exceptionally reliable. In the meantime, rest assured that we take every outage very very seriously, and as always we'll post a full incident report of what happened to the
Apps Dashboard
once our investigation is complete. Again, we apologize for the inconvenience and frustration which the outage has caused.
Labels
#innovationupgrade
#InspireGirls
#moregoogleapps
#SysAdminDay
#tbt
#throwbackthursday
#top10trust
100% web
50states
ad contest
add-ons
admin
Admin console
admin sdk
AirPlay
Android
Android for Work
Android for Work Live
Android Marshmellow
Android Nougat
Android security
Android security tips
Apps Adventures
apps script
apptuesday
Armed Forces Day
Asia Pacific
Atmosphere Live
Audi
Audi Connect
audit
Australia
big data
Big Query
bigquery
Boston
browser
Chomebox for Meetings
Chrome
Chrome Device Management
Chrome digital signage
Chrome for Business
Chrome for Work
Chrome Frame
Chrome OS
Chromebit
Chromebooks
Chromebooks for Business
Chromebooks for Education
Chromebooks for Work
Chromebox for digital signage
Chromebox for meetings
Chromebox for signage
Chromeboxes
Chromecast
City 24/7
Classroom
Clearing Kosovo
Cloud
cloud computing
cloud computing gonegoogle
cloud computing gonegoogle Google Apps
cloud computing gonegoogle Google Apps google docs small business success story
cloud computing gonegoogle Google Apps google docs small business success story switch
cloud datastore
cloud platform
Cloud Platform Live
cloud print
cloud series
cloud services
cloud sql
collaboration
Colorado
Connectors
contacts
Control Panel
customer
customer love
Customer story
Customer support
Customer testimonial
data centers
data processing amendment
data protection
Developer
developers
Digital Learning Day
Docs
documents
DPA
Drawings
Drive for Education
drive sharing
Earth
earth and maps
EC
education
Education on Air
EMC
EMM
Energy
enterprise
EU
events
FedEx
Fedex.com
Finance
Firebase
Forms
franchises
GAFE
Gartner
GE
geo
Global Partner Summit
gmail
Gone Google
gonegoogle
Google AdWords
Google App Engine
Google Apps
Google Apps Blog
Google Apps for Business
Google Apps for Education
Google Apps for Government
Google Apps for Work
Google Apps Marketplace
Google Apps Reseller
Google Apps Script
Google Apps Vault
Google BigQuery
Google Calendar
Google Calendar app
Google Certified Teachers
Google Chrome
Google Chromebases
Google Classroom
Google Cloud Datastore
Google Cloud DNS
Google Cloud Platform
google cloud storage
Google Cloud Vision API
google commerce search
Google Compute Engine
Google Doc
Google Docs
Google Domains
Google Draw
Google Drive
Google Drive for Work
Google Earth
Google Earth Engine
Google Earth Enterprise
Google Earth Images
Google Earth Pro
Google Email Security and Archiving
Google Enterprise
Google Enterprise Search
Google Expeditions
Google for Education
Google for Education Partner Program
Google for Education Training Center
Google for Entrepreneurs
Google for Work
Google for Work and Google for Education Partner Program
Google for Work partner program
Google Forms
Google Green
google groups
Google Hangout
Google Hangouts
Google I/O
Google Keep
Google Maps
Google Maps API
Google Maps APIs
Google Maps Coordinate
Google Maps Engine
Google Maps Engine Pro
Google Maps Engine public data program
Google Maps for Business
Google Maps for Work
Google Maps Gallery
Google Maps Tracks API
Google Message Continuity
google message security
Google Mobile Device Management
Google My Maps
Google My Maps Pro
Google Places API
Google Play
google play for education
Google Prediction API
Google Research tool
Google Science Fair
Google Search Appliance
Google Security Key
Google Sheets
Google Site Search
google sites
Google Slides API
Google Smart Lock
Google spreadsheets
Google Springboard
google storage
Google Storage for Developers
Google Translate
Google Vault
Google Video
Google Wave
google+
Google+ api
Google+ Communities
googlenew
government
GSA
GSA 7.0
GSA for Commerce
guest post
HALO Trust
Hangout on Air
Hangouts on Air
HEAT
hints and tips
HIPAA
Inbox
Inbox by Gmail
innovation
international trade
Internet Explorer
intranet
io2011
iOS
iPad
IT
K-12
Kubernetes
large business
MAM
manufacturing
Mapping a better world
marketplace
marketplace highlights
mashups
MCCs
MDM
medium business
migration
mobile
mobile management
model contract clauses
moms
Mother's Day
mpstaffpick
MyHEAT
NAVMAN
new features
news
Niagara International Transportation Technology Coalition
non-profit
noteworthy
offline
OpenID Connect
Parters
partner
Partner Showcase
partners
Place Summaries
Postini
privacy
product ideas
productivity
Quickoffice
Receptionist's Day
reports
Reseller
retail
RSA
Safer Internet Day
SBW2013
SBW2014
sbweek
SCCs
Search
Security
Security Key
small business
Small Business Week
Small businesses
SMB
spam and security trends
Startups
success story
support
switch
System Admin
T Dispatch
Teamwork 2015
Thanksgiving
Transport and Logistics
Trust
UK
university
University of Calgary
Updates
utilities
Veteran Owned Businesses
Veterans Day
Veterans Day 2013
Veterans Day 2014
viewpoint
VNX
wallet
webinar
webmaster
Winter
women in tech
Women's History Month
Work Resolutions
World Bank
Archive
2016
October
September
August
July
June
May
April
March
February
January
2015
December
November
October
September
August
July
June
May
April
March
February
January
2014
December
November
October
September
August
July
June
May
April
March
February
January
2013
December
November
October
September
August
July
June
May
April
March
February
January
2012
December
November
October
September
August
July
June
May
April
March
February
January
2011
December
November
October
September
August
July
June
May
April
March
February
January
2010
December
November
October
September
August
July
June
May
April
March
February
January
2009
December
November
October
September
August
July
June
May
April
March
February
January
2008
December
November
October
September
August
July
June
May
April
March
February
January
2007
December
November
October
September
August
July
June
May
April
March
February
January
2006
December
November
October
September
August
July
June
May
April
March
February
January
Feed
Google
on
Follow @gsuite
Useful Links
G Suite
Gmail
Hangouts
Calendar
Google+
Google Drive
Google Maps
Google Cloud Platform
