Very interesting that you post this DR article touting your capabilities while you are 3+ days into push apps not functioning for a very large number of your users.
It's great to have back-end DR, but how are we trying to accommodate for internet connection failure between customer facility and Google Apps cloud?
Statistics still show that network failures, especially on last mile etc are still more frequent than actual DR situation coming up in a operational environment.
Redundant Internet connection is generally the first solution, but reality is, due to nature of business of how ISPs work, the last mile connection mostly share media from same Telco leading to SPOF and yes that's generally failure prone as well.
Once this piece of the puzzle has been resolved, it can be considered as a complete solution with adequate redundancy to avoid any outages.
I'd love to be able to take advantage of this. Unfortunately, doing actual business with Google Apps has been extremely frustrating for our small company. They incorrectly charged our credit card twice during the free trial period, and since then, we can't seem to contact a single live human. The data backup and replication is probably fabulous, but the lack of customer service makes us wary about doing business with you.
What would be nice is if you could back up your entire Google Apps (Mail, Docs, etc..) to a third party or some type of appliance. I know that Google has a highly redundant system but placing all your email eggs in one basket is scary. And being able to backup Google Apps and restore it to other email environments (Exchange) is attractive.
@mdfi13, Google has a group "The Data Liberation Front" whose mission is to ensure that all data stored in Google can be extracted by users easily and efficiently.
You talk about keeping data safe in the cloud, for recovery purposes, but you don't talk about how you keep it secure, from theft for example, in the cloud. Thoughts?
Charlie - application errors are not a DR situation. Someone corrupting data doesn't help for DR (or RAID). That's why you need backup tapes...or you make sure that the applications don't have mistakes.
However I'd say that loosing access to push email isn't a disaster (and the people of Chile would agree).
Either way - choose a tool that's right for you, and it sounds like you didn't.
Reading the others I find a mixed bag. Surely google is trying hard. The customer service issues seem classic. People need contact with movers and shakers[folks that have real power]. I realize the liability when someone [a customer service rep.] says something wrong but it's worth the risk when considering the over-all'bottom line'. T'would be a novel and noble company that treats each customer like a V.I.P. Good luck with customer service applications that are supposed to answer all the questions with no words from a real person that matters. edeichingerTechsupport10yrsTx
I hope the core of Rajen's reasoning still gets heard despite clear customer service and billing issues being experienced. The fact is that a huge number of companies are making do with 1999 level disaster recovery technologies based around syndication (copying data with a view to rebuilding systems later if they're destroyed) instead of replication, the live mirroring of all enterprise data onto one or more geographically load balanced remote sites, for instant switchover. This is a very important DR concept for businesses of all sizes to grasp. It's the difference between rebuilding and live failover, and the latter is now affordable to smaller businesses, not just the banks.
Last mile fail is a huge issue, as Prayag says, and many telcos share infrastructure which means there may not be resilience just because you have connectivity from two different suppliers. Get around it by using something like a licensed wireless connection with fibre; the former is getting strong & fast enough to be either primary or secondary.
All these being said of the super DR which Google has put in place for the cloud, I will also buy the option to allow company Administrators to do local backup of all their users in case the DR for Google them self fails.
Ebenezer, how do you allow your admins to do local backups? I would love to know how this can be implemented. I need to be able to backup all emails to local SAN or else I won't be able to sell Google Apps to my IT Manager. :-(
@Ebenezer, how do you perform local backup of Google Apps (especially all emails)? I would love to know how it is possible to implement this! Thanks in advance to your input!
As also pointed out by mdfi13, Google's effort into this area, although laudable, only address a part of the recovery problem. One still needs to be able to do snapshot and (selective) restore, and to do is all at once, and efficiently. The reason that you need this is to deal with the problem of errors, as opposed to disasters, for example if a user accidentally deletes (and I mean REALLY deletes!) something. (In our case it was an administrator who accidentally deleted an entire user!) In this case you need to have at least weekly, and preferably daily selectively restorable backups to rollback to. Being able to POP pull all the email and separately all of the documents, and who knows how to pull all the user profiles (maybe that's doable too), and what-all else defines our enterprise on the gCloud that needs to be saved. BrianB points out that there are ways to do this, but there is no (afaikt) button I can push as a site admin to dump an image, and then be able to pull pieces out and selectively restore them. (The best model of this that I know of is Apple's Time Machine.)
@Charlie I would say that DR and rolling out a new app is two very different processes. While discussing any problems with new apps is a great topic, I personally things it's a completely different subject.
@Thomas Sanders This is a great question that goes unanswered. I would like for Google to give a clear answer to this question. I would personally like to know if this also applies to standard Google accounts. I would like to know how Google protects my personal-non Google Apps account.
@Prayag I'm not sure about Google Apps, but at one time many of the standard non-apps Google accounts had offline sync options (using Google Gears). I know this was disabled for some of the services (such as Docs) but will come back once Gears is updated with HTML5 support. I would assume that in time all of Google Apps will be supported by Gears, allowing for you to continue to work, even when you're offline.
@Dr. M. L. Grim I like this question. I'm trying for find the original video that talks about security, but in the mean time I'll put what Google said into my own words.The first line of protection is at the hard-drive level it's self (I'm of course talking about the HDDs at the Google data centers). They said they take all of your files, break them into multiple parts, and store only a few parts on a single hard drive. Also, every file stored on their hard-drives have an obscure nonsense file name. So if someone got hold of a HDD from a Google data center, they would not be able to extract any usable information.
You can also use a secure connection (https) for some (if not all) of your apps. This allows you to use Gmail on a public wifi without the fear of someone trying to stealing the data being sent over the network. There are many of systems in place too. For example, the admin can force all users to change their password every month. Here is an other example (http://bit.ly/cdCaNy). I'm not sure if this is implemented on Google Apps yet, but since it's for stand non-apps Google accounts I'm sure it's in place for Google Apps too.
@Jeff I agree that administrators should have the ability to restore data much like a user can view a Google Docs spreadsheet history. Being able to restore any action such as restoring a deleted user or restoring a user's deleted document would make me feel much better about using Google Apps. I have basically no knowledge of Google Apps APIs but someone many want to look into it an see if there is an API for pulling and pushing data between Google Apps and your internal computer system.
Little late to the discussion- Any thoughts on Google Mail Continuity? We are running Exchange, but inbound is already routing through Postini so this looks like a great fit. They need to release more details-
thans you Google Apps APIs but someone many want to look into it an see if there is an API for pulling and pushing data between Google Apps and your internal computer system.
If my gmail is down and I don't know if its the server or something I did, that's 100% of users. To tell us it's only .02% in a backhanded way of indicating a small number of users, is ridiculous.
Out of curiosity, a business buys into Google Apps, intrepid into the cloud.
It fails, loses its web services; it is merged into an ongoing successful venture, or is marketed from a bankruptcy chapter proceeding.
The successor wants to continue business.
Will the data be there, as with local storage?
Will it have been blitzed into the Ozone, by Google, if it has had such an interim gap in services - and in paying Google fees.
Bottom line of the scenario - ongoing data maintenance in the event of an account stoppage - has Google any policy or planning that way and if so, what's the story? Sorry Charlie, or something more reassuring?
Regarding the March incident, I think it indicates that Google has adequate cloud backup provisions. If they accidentally released software that wiped people's accounts (a very rare incident, unlikely to happen again soon) and were able to recover everything from tape, then they've proven that there is enough protection in place. The next time you will see anything close to user data loss at Google it will be 3-4 years from now, it will affect 0.02% of users, and Google will recover the data.
You can always make your own backups of the Google Apps data if you are that paranoid. It's slow, but you should be able to do it once a month, especially if you do it incrementally.
Anyone who is planning to or has already migrated to Google Apps, can obtain service from a Google Apps Reseller. Often this service is free, as Google pays resellers a commission.
Thanks for posting this great article on Disaster Recovery. If you want to know more about Backup and Disaster Recovery Services then, visit NetSoft where business continuity solutions facilitate cost-effective backup plans at your disposal. Please contact us for more information on our services!
Very interesting that you post this DR article touting your capabilities while you are 3+ days into push apps not functioning for a very large number of your users.
ReplyDeleteI wish that my company could take advantage of this service. It really seems like a no brainer from the IT perspective.
ReplyDeleteThis is great but what about giving admins the ability to restore accidentally deleted data from these multiple data centers?
ReplyDeleteThis is really fascinating stuff. Google allows Small & Medium business users to concentrate on business, than worry about disaster strategies.
ReplyDelete~Dhananjay Rokde
Would this also apply to the Standard Google Apps?
ReplyDeleteUntil I can recover from "user error" in a 4 hour or less window, we'll be using our systems. (Plus, echo what Charlie said above)
ReplyDeleteIt's great to have back-end DR, but how are we trying to accommodate for internet connection failure between customer facility and Google Apps cloud?
ReplyDeleteStatistics still show that network failures, especially on last mile etc are still more frequent than actual DR situation coming up in a operational environment.
Redundant Internet connection is generally the first solution, but reality is, due to nature of business of how ISPs work, the last mile connection mostly share media from same Telco leading to SPOF and yes that's generally failure prone as well.
Once this piece of the puzzle has been resolved, it can be considered as a complete solution with adequate redundancy to avoid any outages.
we are implementing Google Apps!
ReplyDeleteI'd love to be able to take advantage of this. Unfortunately, doing actual business with Google Apps has been extremely frustrating for our small company. They incorrectly charged our credit card twice during the free trial period, and since then, we can't seem to contact a single live human. The data backup and replication is probably fabulous, but the lack of customer service makes us wary about doing business with you.
ReplyDeleteWhat would be nice is if you could back up your entire Google Apps (Mail, Docs, etc..) to a third party or some type of appliance. I know that Google has a highly redundant system but placing all your email eggs in one basket is scary. And being able to backup Google Apps and restore it to other email environments (Exchange) is attractive.
ReplyDelete@mdfi13, Google has a group "The Data Liberation Front" whose mission is to ensure that all data stored in Google can be extracted by users easily and efficiently.
ReplyDeleteYou talk about keeping data safe in the cloud, for recovery purposes, but you don't talk about how you keep it secure, from theft for example, in the cloud. Thoughts?
ReplyDelete@mdfi13, use POP/IMAP to dl all emails regularly. I use "getmail" every 15 mins.
ReplyDeleteCharlie - application errors are not a DR situation. Someone corrupting data doesn't help for DR (or RAID).
ReplyDeleteThat's why you need backup tapes...or you make sure that the applications don't have mistakes.
However I'd say that loosing access to push email isn't a disaster (and the people of Chile would agree).
Either way - choose a tool that's right for you, and it sounds like you didn't.
Reading the others I find a mixed bag. Surely google is trying hard. The customer service issues seem classic. People need contact with movers and shakers[folks that have real power].
ReplyDeleteI realize the liability when someone [a customer service rep.] says something wrong but it's worth the risk when considering the over-all'bottom line'. T'would be a novel and noble company that treats each customer like a V.I.P. Good luck with customer service applications that are supposed to answer all the questions with no words from a real person that matters. edeichingerTechsupport10yrsTx
I hope the core of Rajen's reasoning still gets heard despite clear customer service and billing issues being experienced. The fact is that a huge number of companies are making do with 1999 level disaster recovery technologies based around syndication (copying data with a view to rebuilding systems later if they're destroyed) instead of replication, the live mirroring of all enterprise data onto one or more geographically load balanced remote sites, for instant switchover. This is a very important DR concept for businesses of all sizes to grasp. It's the difference between rebuilding and live failover, and the latter is now affordable to smaller businesses, not just the banks.
ReplyDeleteLast mile fail is a huge issue, as Prayag says, and many telcos share infrastructure which means there may not be resilience just because you have connectivity from two different suppliers. Get around it by using something like a licensed wireless connection with fibre; the former is getting strong & fast enough to be either primary or secondary.
Sheila Averbuch -- ENN
All these being said of the super DR which Google has put in place for the cloud, I will also buy the option to allow company Administrators to do local backup of all their users in case the DR for Google them self fails.
ReplyDeleteEbenezer, how do you allow your admins to do local backups? I would love to know how this can be implemented. I need to be able to backup all emails to local SAN or else I won't be able to sell Google Apps to my IT Manager. :-(
ReplyDelete@Ebenezer, how do you perform local backup of Google Apps (especially all emails)? I would love to know how it is possible to implement this! Thanks in advance to your input!
ReplyDeleteAs also pointed out by mdfi13, Google's effort into this area, although laudable, only address a part of the recovery problem. One still needs to be able to do snapshot and (selective) restore, and to do is all at once, and efficiently. The reason that you need this is to deal with the problem of errors, as opposed to disasters, for example if a user accidentally deletes (and I mean REALLY deletes!) something. (In our case it was an administrator who accidentally deleted an entire user!) In this case you need to have at least weekly, and preferably daily selectively restorable backups to rollback to. Being able to POP pull all the email and separately all of the documents, and who knows how to pull all the user profiles (maybe that's doable too), and what-all else defines our enterprise on the gCloud that needs to be saved. BrianB points out that there are ways to do this, but there is no (afaikt) button I can push as a site admin to dump an image, and then be able to pull pieces out and selectively restore them. (The best model of this that I know of is Apple's Time Machine.)
ReplyDelete@Charlie
ReplyDeleteI would say that DR and rolling out a new app is two very different processes. While discussing any problems with new apps is a great topic, I personally things it's a completely different subject.
@Thomas Sanders
This is a great question that goes unanswered. I would like for Google to give a clear answer to this question. I would personally like to know if this also applies to standard Google accounts. I would like to know how Google protects my personal-non Google Apps account.
@Prayag
I'm not sure about Google Apps, but at one time many of the standard non-apps Google accounts had offline sync options (using Google Gears). I know this was disabled for some of the services (such as Docs) but will come back once Gears is updated with HTML5 support. I would assume that in time all of Google Apps will be supported by Gears, allowing for you to continue to work, even when you're offline.
@Dr. M. L. Grim
I like this question. I'm trying for find the original video that talks about security, but in the mean time I'll put what Google said into my own words.The first line of protection is at the hard-drive level it's self (I'm of course talking about the HDDs at the Google data centers). They said they take all of your files, break them into multiple parts, and store only a few parts on a single hard drive. Also, every file stored on their hard-drives have an obscure nonsense file name. So if someone got hold of a HDD from a Google data center, they would not be able to extract any usable information.
You can also use a secure connection (https) for some (if not all) of your apps. This allows you to use Gmail on a public wifi without the fear of someone trying to stealing the data being sent over the network. There are many of systems in place too. For example, the admin can force all users to change their password every month. Here is an other example (http://bit.ly/cdCaNy). I'm not sure if this is implemented on Google Apps yet, but since it's for stand non-apps Google accounts I'm sure it's in place for Google Apps too.
@Jeff
I agree that administrators should have the ability to restore data much like a user can view a Google Docs spreadsheet history. Being able to restore any action such as restoring a deleted user or restoring a user's deleted document would make me feel much better about using Google Apps. I have basically no knowledge of Google Apps APIs but someone many want to look into it an see if there is an API for pulling and pushing data between Google Apps and your internal computer system.
great post!
ReplyDeleteOh, thank you Sesli sohbet for you good.
ReplyDeleteLittle late to the discussion- Any thoughts on Google Mail Continuity? We are running Exchange, but inbound is already routing through Postini so this looks like a great fit. They need to release more details-
ReplyDeletethans you Google Apps APIs but someone many want to look into it an see if there is an API for pulling and pushing data between Google Apps and your internal computer system.
ReplyDeletearkadaşlık
great post nice thanks.
ReplyDeleteI'm so impressed with the how reliable GMail has proven to be - beginning Sunday morning. (/sarcasm)
ReplyDeleteIf my gmail is down and I don't know if its the server or something I did, that's 100% of users. To tell us it's only .02% in a backhanded way of indicating a small number of users, is ridiculous.
ReplyDeleteOut of curiosity, a business buys into Google Apps, intrepid into the cloud.
ReplyDeleteIt fails, loses its web services; it is merged into an ongoing successful venture, or is marketed from a bankruptcy chapter proceeding.
The successor wants to continue business.
Will the data be there, as with local storage?
Will it have been blitzed into the Ozone, by Google, if it has had such an interim gap in services - and in paying Google fees.
Bottom line of the scenario - ongoing data maintenance in the event of an account stoppage - has Google any policy or planning that way and if so, what's the story? Sorry Charlie, or something more reassuring?
Regarding the March incident, I think it indicates that Google has adequate cloud backup provisions. If they accidentally released software that wiped people's accounts (a very rare incident, unlikely to happen again soon) and were able to recover everything from tape, then they've proven that there is enough protection in place. The next time you will see anything close to user data loss at Google it will be 3-4 years from now, it will affect 0.02% of users, and Google will recover the data.
ReplyDeleteYou can always make your own backups of the Google Apps data if you are that paranoid. It's slow, but you should be able to do it once a month, especially if you do it incrementally.
Creating backup is not different any where Google is not special about it
ReplyDeleteThnxs for advice by urs
ReplyDelete@charlie - Totally relevant! Gmail was down for 3+ days for many users not too long ago.
ReplyDeleteAnyone who is planning to or has already migrated to Google Apps, can obtain service from a Google Apps Reseller. Often this service is free, as Google pays resellers a commission.
ReplyDeleteEnterprise Trucking Software
ReplyDeleteExspeedite MOBILE TRUCKING SOFTWARE provides everything that our trucking customers need so they can do business in an organized and efficient way...
Thanks for posting this great article on Disaster Recovery. If you want to know more about Backup and Disaster Recovery Services then, visit NetSoft where business continuity solutions facilitate cost-effective backup plans at your disposal. Please contact us for more information on our services!
ReplyDelete