Ah, the '03 Blackout. Those were the days.
We knew there was a problem. The few server checks, environmental monitors, and UPS alerts properly employed by my employer then pointed to a problem. The early hours knew no scope but there was fear.
I was already home with my grade school children when the power went out.
You can read my personal recollections at harmonypirate.com
I had a land line Ameritech (or SBC, now AT&T) phone, a Sprint cell phone, and a work-provided Ameritech/SBC/AT&T T1 line into my house. Between the three I was able to stay on-line through most of the blackout as long as I could find power. At one point when there was an issue (in Europe, I think) I ran an extension cord from my running Pontiac Vibe to a power strip powering my laptop, CSU/DSU, and router. This was short-sighted as I lacked extra gas but worked out.
I remember driving to the colocation facility. The alarms enthusiastically triggered on Friday. The Colocation (colo) facility near me was up but some servers were in an odd state of up and completely down for some services at the same time. I packed the kids into the car and drove out.
The facility was doing all the things they needed to do: they were overseeing topping off the deisel tanks (great service contract) and supervising checking the UPS batteries (great service contract) and making sure there was air flow and ordering some extra UPS units just in case (again, great service contracts) and performing security walk-throughs. My immediate problem was that we were tripping one of our circuits over and over again. Some dual power supply devices terminated into the same circuit and other underutilized circuits were full from an outlet perspective.
The kids and I ran up to the local Best Buy and picked up some power strips and fans. My employer was lucky that I had enough cash as Best Buy couldn't process credit or debit card transactions. When we got back to the colo, the facilities folks conveniently weren't around while I re-ran 120 vac power on to the consumer grade power strips. I set up the fans to provide additional air flow. My new co-worker at the time … I can't remember if he helped out or not. I seem to recall he found himself stuck but I could be combining different events in my head.
As my kids and I headed home on 8 Mile Road we saw people outside talking. We returned to the colo for the next two days until the power came back on. We kept seeing people outside, talking, having a good time.
Oh, and my former employer's servers stayed up and functioned within normal parameters through the entire outage once the power distribution error was fixed.
These are the times that try a company's providers. Having a provider with a deisel truck topping off the tanks in the early hours of a massive blackout is a good sign.
p.s. My colleagues at the time might recall some of the details better than I do. I reserve the right to amend and extend my comments.