Last night, at midnight, the first night in about two weeks that I'd gotten to sleep BEFORE midnight, my phone started buzzing that FeedBurner's servers couldn't be contacted. It turns out there was a big power failure in downtown chicago, and the facility where FeedBurner's servers are located, Equinix, didn't cut their customers over to backup power successfully (at least not any of the customers we know of that we checked). Now, backup power at a Tier 1 hosting facility like Equinix is essentially the one service they provide that you'll ever need. Especially in Chicago, where major downtown fires are starting to become a hallmark of the city's urban ecosystem. We should have a tourist campaign around them, they're getting to be so frequent. Or at least one of those construction site signs "Welcome to Chicago, no major urban fires in 14 days".
Anyway, as I say, backup power at a Tier 1 hosting provider is really all you're paying for. It's the only thing you can't gaurantee on your own. So, when Equinix failed last night, for more than 3 hours, I knew that they would spin it in their systems reports. I knew it, and I knew it because 99 out of 100 major companies suck at customer service. Instead of taking responsibility, they rationalize, justify, and equivocate. Equinix's explanation to our bandwidth provider Internap this morning was hilarious, except for the part where we were freaking out for three hours last night. "[our] generator came
on, but due to heavy loads it appears [we] are still having issues with [our] back up generator". Issues? Power is one of those great things that is either YES or NO, ON or OFF. You don't have working power that has issues. I'll remember to tell that to my daughter the next time she asks about a burned out lightbulb, "Well honey, the lightbulb is on but it has issues". Secondly, the "but due to heavy loads" comment is repulsive. Are you kidding me? What did you think was going to happen when you needed the backup generator? That only one or two companies might need it and that the other servers would quickly figure out how to create their own energy, converting the material from the surrounding cages into usable power? WTF? "due to heavy loads"??? Shouldn't the facility know what it's current total energy demands are? They were surprised that ALL the servers in the facility needed power when the power went out?
Just once I would like to see a large company come out and say "We failed. The one time you needed us, we failed." It never happens. Too many companies view customers as "people that bother us when something goes wrong". We had four people up from 1am to 4am on a friday night / saturday morning. Morningstar's servers are near ours, they were down, as were other companies we checked, and I'm sure hundreds of other companies had people up all night waiting for power to be restored so they could bring sites back up. What do you get from Equinix on status? 'Everything's working as designed but there are issues'. How about "The entire philisophical foundation of why you use a hosting facility failed in a massive way. We realize we're going to have to triple our efforts to regain ANY customer confidence in Equinix. We will be communicating with you regularly over the coming weeks as we test and retest our backup systems." How about trying that next time.
Posted by Dick at June 25, 2005 10:28 AM | TrackBack