Send News. Want a reply? Read this. More in the FAQ.   News Forum - All Forums - Mobile - PDA - RSS Headlines  RSS Headlines   Twitter  Twitter
Customize
User Settings
Styles:
LAN Parties
Upcoming one-time events:
Greenbelt, MD 08/22

Regularly scheduled events

Out of the Blue

Well at some point when it seemed like it was going to be short story rather than a novel, I promised a full accounting of the events surrounding our server crash and subsequent down-time. At this point I don't think it's possible to marshal all the necessary details into a concise (or even rambling) explanation, but in an effort to fulfill that promise and satisfy curious minds, I will attempt to point out some of the highlights of the situation, without getting too far into the blame game. That said, I have to say up front that the main culprit in the downtime is myself, as noted in this first point below:

Things that went particularly wrong:
After our previous server crash the mail I sent off to the appropriate IT folk to make sure our back-ups were being properly conducted had a typo in the address line, and was never received, much less acted upon. This single point of failure was really the downfall of the whole process, and I have to accept full responsibility for that (and I have scolded others in the past for introducing typos into things like email addresses and URLs when it would be far safer to use an address book entry or copy and paste them from known good locations). There was also an unfortunate communication breakdown (for which the fault lies elsewhere) that prevented all the data from the old server from being transferred to the new one when it could have prevented all this from being a problem. The actual server crash was heat related, apparently the cooling failed in the box that was Frankensteined together, and things sort of melted down. It took a while to determine the drives needed data recovery, and the recovered data was returned on a drive that was not useable in any systems we had on hand (we specifically asked to get the data back on DVD but that was apparently impractical, don't ask me how we then ended up with incompatible media). Eventually a machine had to be built to accommodate the drive and this server was then placed in the data center for transfer. Naturally a host of disk errors ensued, because what fun would disaster recovery be without further disasters?

Things that went particularly right:
The folks at UGO stepped up to the plate big time as far as providing resources, coming up with a brand new server to move the site onto temporarily (and now permanently), in coordinating and paying for the data recovery, and then springing for the parts to build a new machine just to read the recovery drive. Additionally, their IT people, in particular Chris, devoted several man-days to this whole process. Bagpuss also deserves thanks for some timely technical assistance, as well as jumping in and setting up the Chatbear forum we used during the recovery. Furn, Blammo programmer who is now wrapped up in his studies, also sacrificed a good deal of his valuable spare time to work on various aspects of this. Last, but far from least, is Frans, who is always a great friend to this site. Frans put in a yeoman effort on this project: helping sort through the aftermath of the crash, devising a plan going forward, providing moral support when things looked bleak, and even at the very end, sympathetic over the prospect that I would have to manually enter the some 1200(!) stories from the last seven weeks into the database, he wrote a script to automate most of the database update, which he did while fighting the effects of pneumonia! Friends don't get any better than that, and I can only hope he knows how much that's appreciated. So huzzah and kudos to Frans, apologies for the downtime, and thanks for hanging in there with us.

Oh yeah, we will be implementing a disaster recovery plan that will hopefully be less of a disaster going forward.

R.I.P.: Prince Rainier of Monaco Dies at Age 81 and Saul Bellow, Author and Nobel Winner, Dead at 89.

Play Time: Gyroball.
Links of the Day: SIN CITY Comparisons. Thanks Chris.
GGG-GMAIL!. Language warning.
The Baby Sitter Outsource. Thanks Brent.
Stories of the Day: Florida Law Expands Right to Kill in Self-Defense.
Web Postcards Hide Trojan Horse Programs.
Science!: Honda Robot Helps Teach in Classrooms.
Windmills in the Sky.
Food Companies Test Flavorings That Can Mimic Sugar, Salt or MSG (registration required).
Image of the Day: Gibby's Game Room.
Media of the Day: Dance, White Boy, Dance. Thanks Mallig.
Get Perpendicular. Thanks Ant and Shacknews.
Teh Funnies: Dr. Fun. Thanks Philip.
Auction of the Day: Natural shaped wood knot penis. Thanks advocate.
Follow-ups: Thompson's ashes to be shot from cannon. Thanks nin.
Fabien Cousteau.
Mars rovers enjoy a new lease of life.
Thanks Mike Martinez and Ant.

View
3. Outstanding! Apr 6, 2005, 11:47 Tigger
 
I was beginning to wonder if there was going to be a recovery!

--
Tigger
"It's not the years, it's the mileage" ~Indiana Jones
 
Avatar 7252
 
--
Tigger
Vic Fontaine for President
Previous Post Next Post Reply Quote Edit Delete Report
 
    Date Subject Author
  1. Apr 6, 11:37 No subject Sixis
  2. Apr 6, 11:42  Re: No subject k1llb0y
  4. Apr 6, 11:49  Re: No subject nin
  5. Apr 6, 11:52   Re: No subject nin
>> 3. Apr 6, 11:47 Outstanding! Tigger
  6. Apr 6, 12:03 Re: No subject All4naught
  7. Apr 6, 12:13  Re: No subject Warhawk
  8. Apr 6, 12:20   Re: No subject Zathrus
  9. Apr 6, 12:25    Re: No subject Warhawk
  10. Apr 6, 12:26     Re: No subject  Blue 
  12. Apr 6, 12:28      Re: No subject Parallax Abstraction
  14. Apr 6, 13:01      Re: No subject Warhawk
  15. Apr 6, 13:04       Re: No subject  Blue 
  16. Apr 6, 13:12        Re: No subject MeatForce
  19. Apr 6, 13:30        Re: No subject Warhawk
  21. Apr 6, 13:36        Re: No subject Cynips
  18. Apr 6, 13:30      Re: No subject Tigger
  20. Apr 6, 13:32       Re: No subject Warhawk
  11. Apr 6, 12:28    Re: No subject Zathrus
  13. Apr 6, 12:31 No subject Sho
  17. Apr 6, 13:20  Re: No subject Bludd
  22. Apr 6, 13:46 Great to see ya back CrimsonPaw
  23. Apr 6, 13:52  Re: Great to see ya back Valcor
  24. Apr 6, 14:03   Re: Great to see ya back sauron
  25. Apr 6, 14:09    Re: Great to see ya back Heretic_QPF
  26. Apr 6, 14:21     Re: Great to see ya back bjbgraphics
  27. Apr 6, 14:39      Re: Great to see ya back Warhawk
  28. Apr 6, 14:58       Re: Great to see ya back Tango
  29. Apr 6, 15:19 Re: No subject ExcessDan
  40. Apr 6, 16:42  Re: No subject Hump
  50. Apr 6, 23:33   Re: No subject ExcessDan
  51. Apr 7, 01:58    Re: No subject Bearkyn
  52. Apr 7, 13:29     Re: No subject MMORPGHoD
  53. Apr 8, 01:32      Re: No subject Communist
  55. Apr 8, 08:51       Re: No subject Tango
  30. Apr 6, 15:24 Re: No subject WarPig
  31. Apr 6, 15:27  Re: No subject snide
  32. Apr 6, 15:28   Re: No subject Stolk
  34. Apr 6, 15:31   Re: No subject The Truth
  35. Apr 6, 15:50    Re: No subject Ihya
  36. Apr 6, 15:55     Re: No subject Warhawk
  37. Apr 6, 16:02     Re: No subject nin
  38. Apr 6, 16:06      Re: No subject Warhawk
  43. Apr 6, 17:44      Re: No subject Bronco
  44. Apr 6, 18:12       Re: No subject Warhawk
  33. Apr 6, 15:29 Re: Too little... WarPig
  39. Apr 6, 16:34 BluesNews Lives! Cholma
  41. Apr 6, 17:18 Good to see it back! Tim1_2
  42. Apr 6, 17:25 "You meddling kids" Killer Kane
  45. Apr 6, 20:15 The new old site Jim
  46. Apr 6, 21:25  Re: The new old site BDFD
  47. Apr 6, 21:41 Re: they finally said it Michael Bolton's hair dresser
  48. Apr 6, 22:08 Re: Why this is and who we think we are chiisu81
  49. Apr 6, 22:36 No subject xXBatmanXx
  54. Apr 8, 07:29 No subject theyarecomingforyou


footer

Blue's News logo