Resurrection

We’ve all done it — that fatal keystroke that wipes out the wrong partition on the disk, vanishes the critical directory, destroys the mission-critical database. Thank god, this time it wasn’t me :)

Got a call late last night from a contact of a contact (not a Birdhouse customer) who had been running a search/replace operation at a very high-powered Movable Type installation. Apparently watching TV while working (lesson #1), the staffer had left the Find field blank and put the string “sponsor” in the replace field. MT 3.1 puts a weighty warning on the search/replace page, but no further confirmation dialog stops you from sawing off your toes (3.2 throws a confirmation dialog). Result: the string “sponsor” between every letter of every Title field, affecting 11,000 entries. Basically, they were screwed and pulled me in.

The client had been sticking href links directly into their Title fields, (naughty! — lesson #2), so the messed up titles were extremely long. So long, apparently, that they exceeded max length in the database for that field. So when I ran the same replace operation in reverse, they ended up with truncated titles with unclosed href tags — not helpful. Time to reach for backups. The system’s automated daily backup had already run, and so included the same error. The weekly/complete backup, including far, far more stuff than we needed, was gargantuan. Allowed it to download overnight, only to find the tarball corrupt in the morning.

Eventually reached a contact at the web host, who was able to dig just the database we needed out of the weekly backup on his end — complete up to five days ago. Got that hooked back up to their MT install, then appended with a tweaked MT import covering the intervening period. All that remained was to replace the bogus titles on the last 68 records, which they were able to do by copy/pasting from saved HTML pages.

Glad they’re up and running again, and that I was able to help. Once again, face to face with the critical nature of solid backups (a backup isn’t a backup until you’ve tried to restore it!). Exhausted, and it wasn’t even my fire to put out for a change.

Music: Sufjan Stevens :: They Are Night Zombies!! They Are Neighbors!! They Have Come Back From the Dead!! Ahhhh!

Leave a Reply

Your email address will not be published. Required fields are marked *