Woefully Unprepared for a Hardware Failure

As I mentioned on Monday, I suffered a hard drive failure over the weekend as a result of a pretty severe storm here in Atlanta. There is a happy ending to the story, in that I was able to recover all of the data I cared about and I was operational immediately and back to full speed within 48 hours.

So why do I say I was woefully unprepared for a hardware failure? Basically, restoring my work environment took way too long and I made some amateurish mistakes that cost me time. Thankfully not any money, but losing time is the same thing and it really annoys me. I get that I only have to support one person, so that makes me different than most of my readership, BUT there are some things I picked up and some new processes that I'm implementing that will streamline the process the next time I have a failure.

First, let me describe my work environment a bit to give some background. I use a desktop PC when I'm in the office. I have a Mac ibook notebook computer for when I travel or need to get out of the house and work at Panera, etc. I also have a family PC that my oldest daughter uses to explore the web, play games, etc. My backup system is pretty simple, but designed to make sure I have no single point of failure and that I can get up and running very quickly. In concept anyway.

I have a folder on my desktop that is all of my Security Incite files. Since I use a Mac laptop, I needed some way to keep all of those files current and synchronized on both platforms. I use a great product called FolderShare (who was acquired by Microsoft last year). FolderShare allows me to replicate files between multiple devices and automatically syncs whenever there are any changes. And it transparently supports both PC and Mac. I have current versions of all my business files on both the desktop and Mac at all times. So if I decide to go work remotely, I just grab the Mac and I'm good to go.

I also keep my music and photos replicated between my desktop and the family PC, so I always have two copies. I can't replace the digital pictures, so I like the idea of having those instantly replicated to two places. This product kicks butt and if you are multi-platform it's a necessity. Best of all it's free. You can't beat that.

I use a hosted Exchange provider for email (I can connect to the same mailbox from Outlook on my PC and Entourage on my Mac), so all of my data was up on their site. I can get to it via OWA (outlook web access) or can pull down the data very quickly once I reinstall Outlook.

But I'm not satisfied with that because if something happens at my house (God forbid) I am SOL. So I also backup most of the relevant files to an online back-up service using Connected Corp, which was acquired by Iron Mountain. So I have access to that backup as well.

So on most days I feel pretty good that my data is protected. I'd actually say I got a bit complacent. Complacency is a BAD place to be. That's when people get hurt.

When I found my PC unable to boot last weekend, I just pulled out my Mac and I was instantly productive. No problem there. FolderShare worked like a charm. All my files were exactly where they were supposed to be. Then I ran into a number of problems. First, it took me some time to run the diagnostics on my machine. And when I did, I ran the wrong diagnostics (when it took about 30 hours I should have suspected something). When I finally got the diagnostics right, I learned that my drive seemed structurally intact, but I still couldn't boot - so I'd have to rebuild the machine.

The PC I bought came from Systemax, and they have a very cool capability to restore the machine to factory settings. I think the software to do this comes from PowerQuest (which was bought by Symantec). This process took about 20 minutes and was painless. I was back in Windows XP and set up my user account, printer shares, etc. I started the FolderShare restore and within a hour, I had all of my work files back on my PC. So far so good.

I took me a while, but I found all of my software (you know, Microsoft Office, Quicken) and downloaded whatever I didn't have (iTunes, Firefox, etc.). This took another couple of hours to replace all of that software - but that went pretty smoothly as well.

But then I realized some of my personal files were out of date on my family PC. I had a blown fan so the computer sounded like a 747, so most of the time I had that machine turned off. I actually had another machine that I was going to replace it with, but I was too lazy to move everything over. Thus, FolderShare wasn't syncing my personal files, including Quicken - which I use for both personal and business accounting. It wasn't Foldershare's fault, it was mine. But nonetheless I needed to get my Quicken file back ASAP. I had invoices to send out. In a start-up cash is king and no invoice means no cash.

No worries, right? I had the Quicken files on the online service, so I'll just reinstall that software and pull down the files and I'm good to go. Ah, not so much. It seems that Connected doesn't want folks to just download their software, so they require an active account number to download. Uh oh. My account number was on the old, re-imaged drive, so I didn't have it. No worries, I can have them send it to me via email. Uh oh. None of my email addresses seemed to work. Hmm. OK so I'll call. They don't have 24/7 support.

WHAT? A backup service that does not have 24/7 support! Now I was pretty pissed. I had burned most of my Sunday trying to get back and now I can't get over the finish line because my backup provider thinks failures only happen between 8 AM and 9 PM Mon-Fri. That's a problem. But, I had most of my files and could be productive on the Mac - so it wasn't a total waste.

On Monday AM, I called Connected again, but they require an account number to even talk to a rep. This is ridiculous. I don't have the account number. So I called the main number at Iron Mountain and it seemed like I was from Mars. They had no idea what the Connected offering was. I got sent to a customer service rep who also needed an account number to arrange for my backup tapes to be delivered. WHAT? I don't have backup tapes. Suffice it to say, I was pretty frustrated.

Finally, the rep understood what I needed and then told me to wait a minute. He then came back and said he needed an account number to forward my call to the Connected people. No kidding. He did say he found a prompt on another number that asked for a credit card. Well I had the credit card they were billing, so I gave that a try. The customer service rep was nice and he tried to be helpful, but he just didn't have the information about the Connected offering. So my customer experience was less than stellar.

So I finally got through to the right rep (using my credit card number) and he gave me my account number within 5 minutes. I was restoring my data and had everything I needed within an hour. Again, why they don't have 24/7 is beyond me. They also should offer the ability to talk to a rep with or without an account number. When you lose a machine, it's pretty traumatic. To have to navigate through a bunch of crap to talk to someone did not help the delicate state I was in. The poor guy who did finally answer the phone got an earful, which could have been easily avoided.

All's well that ends well, I guess. But it was painful at the time and I lost a lot of time. Time I should have been chasing my kids around. I did learn a lot from the experience, so here are a few tips:

1. Practice - This was my biggest mistake. I had gone over the details in a theoretical fashion, but I hadn't actually called or tried to reinstall the Connected software in 2 years. I didn't know they required an account number, which is now stored in my Blackberry and on paper. I was also unfamiliar with my system diagnostics, so I wasted time figuring that out.
   
   
2. Document - I found that through most of Monday I was downloading and installing software that I needed (like Acrobat), but forgot to do over the weekend. That costs time, so now I have a list of software that I need on the machine. So if this happens again, I can just work through the list and get thing back up and running. I guess I could burn an image (using Ghost or something like that) of my machine, but the applications I use change pretty frequently. So I'm not sure that would save me a lot of time. But if you support hundreds (or thousands) of PCs, having standard images and separating out the data is key.
   
   
3. Consider virtualization - Virtualization is all the rage now, in that it allows you to image a PC (or Linux box) and get it running very quickly. You keep your data on another partition on the drive (maybe even another drive altogether). If something blows up, you just install another image and you are good to go. I'm going to look at this, but it may be overkill for my single person shop.
   
   
4. Have multiple backups - I'm just more comfortable with most of my data protected by Connected, but also replicated to another PC through Foldershare. That way I've got multiple contingency plans. And in this case, I seemed to have needed every single one.
   
   
5. Practice - Did I mention that one already? Well I'll say it again. You don't know what doesn't work until you actually try it.

So, I know this was a bit of a long post, but hopefully you can learn something from my pain. Anyway, now I have it documented, so I can refer back to it the next time I screw something up.