One of my companies uses an ISP a few time zones away. I created all of the pages for our site, while the business guys set up the e-commerce portion of the site, all done about three or four years ago. It was a good value at the time. But this weekend the ISP had a massive crash that supposedly corrupted several of their servers. Of course, it hit our site and took it offline.
Besides being down for a few days now, our e-commerce data was wiped. You'd think that the ISP would have a failover and backup and would be working hard to restore everything. Not so. I spent a few hours uploading the pages and putting things back into place, but the e-commerce area was not something I had worked with... the ISP didn't provide access to it. Hell, it took me a day to get them to even put a placeholder up so I could start repairing things.
So now the business guys are scambling, trying to recover the rest of the site. I'm away for the weekend to go see my son graduate from high school, so I can't help. They're on their own. The fact that the business guys see this all as magic scares the crap out of me. They just hadn't ever really considered what was going on behind the curtain. Nobody's fault... the magic just went bad.
It just galls me that in an age of technology, disaster recovery has been so ill-conceived. Lesson learned: don't let those that are mystified by technology make important technology choices. I'd laugh, but it's all just too sad.