We all experience system outages from time to time. That's technology. The key is to not waste time pointing the finger or denying the issues but instead drive to root cause and put the right process or technology in place to make sure you never hit the same problem twice.
Thank you for the transparency into the issue. It's refreshing.
Nice transparency. Bummer though. No-one's immune to issues like this. The best you can hope for is a rapid rollback scenario after a bad deploy. Sounds like you've got that in place too.
I appreciate the openness, however this illuminates the achilles heel with services such as this. When they fail, they fail for everyone, globally. While some may be able to get on without proper document building tools or the ability to research something held in a google doc for an hour, email is akin to dialtone and users expect it to work, period. Any mechanism to segment users and implement some A B testing would go a long way towards isolating and reducing the impact of such changes, as well as reducing the roll back time needed should that be required. If the SAR team that I worked with was on a mission where they rely on Google Docs, it would have impaired the ability to communicate with the teams and reduce the effectiveness of the search effort. In a live search scenario, that could make the difference between someone surviving or not.
Thanks Google, for explaining what happened. It feels like we are the part of the solution when such transparency is maintained.
ReplyDeleteWe all experience system outages from time to time. That's technology. The key is to not waste time pointing the finger or denying the issues but instead drive to root cause and put the right process or technology in place to make sure you never hit the same problem twice.
ReplyDeleteThank you for the transparency into the issue. It's refreshing.
Why can't you just find the bug and squash it?
ReplyDeleteNice transparency. Bummer though. No-one's immune to issues like this. The best you can hope for is a rapid rollback scenario after a bad deploy. Sounds like you've got that in place too.
ReplyDeleteThanks for this update! The 30 minutes of down time is nothing when compared to the hours and weeks of time that I've saved by Going Google.
ReplyDeleteAll i can say is thank u for begin transparent, it feels good.
ReplyDeleteAll i can say is thank u goggle . It feels good when an organization is this transparent whom we relay on.
ReplyDeleteI appreciate the openness, however this illuminates the achilles heel with services such as this. When they fail, they fail for everyone, globally. While some may be able to get on without proper document building tools or the ability to research something held in a google doc for an hour, email is akin to dialtone and users expect it to work, period. Any mechanism to segment users and implement some A B testing would go a long way towards isolating and reducing the impact of such changes, as well as reducing the roll back time needed should that be required.
ReplyDeleteIf the SAR team that I worked with was on a mission where they rely on Google Docs, it would have impaired the ability to communicate with the teams and reduce the effectiveness of the search effort. In a live search scenario, that could make the difference between someone surviving or not.
Loving the transparency, keep up the good reporting!
ReplyDelete