Send News. Want a reply? Read this. More in the FAQ.   News Forum - All Forums - Mobile - PDA - RSS Headlines  RSS Headlines   Twitter  Twitter
Customize
User Settings
Styles:
LAN Parties
Upcoming one-time events:

Regularly scheduled events

Out of the Blue

Well at some point when it seemed like it was going to be short story rather than a novel, I promised a full accounting of the events surrounding our server crash and subsequent down-time. At this point I don't think it's possible to marshal all the necessary details into a concise (or even rambling) explanation, but in an effort to fulfill that promise and satisfy curious minds, I will attempt to point out some of the highlights of the situation, without getting too far into the blame game. That said, I have to say up front that the main culprit in the downtime is myself, as noted in this first point below:

Things that went particularly wrong:
After our previous server crash the mail I sent off to the appropriate IT folk to make sure our back-ups were being properly conducted had a typo in the address line, and was never received, much less acted upon. This single point of failure was really the downfall of the whole process, and I have to accept full responsibility for that (and I have scolded others in the past for introducing typos into things like email addresses and URLs when it would be far safer to use an address book entry or copy and paste them from known good locations). There was also an unfortunate communication breakdown (for which the fault lies elsewhere) that prevented all the data from the old server from being transferred to the new one when it could have prevented all this from being a problem. The actual server crash was heat related, apparently the cooling failed in the box that was Frankensteined together, and things sort of melted down. It took a while to determine the drives needed data recovery, and the recovered data was returned on a drive that was not useable in any systems we had on hand (we specifically asked to get the data back on DVD but that was apparently impractical, don't ask me how we then ended up with incompatible media). Eventually a machine had to be built to accommodate the drive and this server was then placed in the data center for transfer. Naturally a host of disk errors ensued, because what fun would disaster recovery be without further disasters?

Things that went particularly right:
The folks at UGO stepped up to the plate big time as far as providing resources, coming up with a brand new server to move the site onto temporarily (and now permanently), in coordinating and paying for the data recovery, and then springing for the parts to build a new machine just to read the recovery drive. Additionally, their IT people, in particular Chris, devoted several man-days to this whole process. Bagpuss also deserves thanks for some timely technical assistance, as well as jumping in and setting up the Chatbear forum we used during the recovery. Furn, Blammo programmer who is now wrapped up in his studies, also sacrificed a good deal of his valuable spare time to work on various aspects of this. Last, but far from least, is Frans, who is always a great friend to this site. Frans put in a yeoman effort on this project: helping sort through the aftermath of the crash, devising a plan going forward, providing moral support when things looked bleak, and even at the very end, sympathetic over the prospect that I would have to manually enter the some 1200(!) stories from the last seven weeks into the database, he wrote a script to automate most of the database update, which he did while fighting the effects of pneumonia! Friends don't get any better than that, and I can only hope he knows how much that's appreciated. So huzzah and kudos to Frans, apologies for the downtime, and thanks for hanging in there with us.

Oh yeah, we will be implementing a disaster recovery plan that will hopefully be less of a disaster going forward.

R.I.P.: Prince Rainier of Monaco Dies at Age 81 and Saul Bellow, Author and Nobel Winner, Dead at 89.

Play Time: Gyroball.
Links of the Day: SIN CITY Comparisons. Thanks Chris.
GGG-GMAIL!. Language warning.
The Baby Sitter Outsource. Thanks Brent.
Stories of the Day: Florida Law Expands Right to Kill in Self-Defense.
Web Postcards Hide Trojan Horse Programs.
Science!: Honda Robot Helps Teach in Classrooms.
Windmills in the Sky.
Food Companies Test Flavorings That Can Mimic Sugar, Salt or MSG (registration required).
Image of the Day: Gibby's Game Room.
Media of the Day: Dance, White Boy, Dance. Thanks Mallig.
Get Perpendicular. Thanks Ant and Shacknews.
Teh Funnies: Dr. Fun. Thanks Philip.
Auction of the Day: Natural shaped wood knot penis. Thanks advocate.
Follow-ups: Thompson's ashes to be shot from cannon. Thanks nin.
Fabien Cousteau.
Mars rovers enjoy a new lease of life.
Thanks Mike Martinez and Ant.

View
55 Replies. 3 pages. Viewing page 1.
< Newer [ 1 2 3 ] Older >

55. Re: No subject Apr 8, 2005, 08:51 Tango
 
is the shade of yellow pissing anyone else off?
Yeah I noticed that... it's not as... yellow as I remember

 
Avatar 18712
 
Reply Quote Edit Delete Report
 
54. No subject Apr 8, 2005, 07:29 theyarecomingforyou
 
I'm glad that the site is back to usual... these colours are so much more pleasant and readable than the darker colours; I had forgotten how pretty the site looked.

Only sad thing was that BluesNews forgot me... I had to resign up (not that I had an especially high post count).

 
Avatar 22891
 
SteamID: theyarecomingforyou
Reply Quote Edit Delete Report
 
53. Re: No subject Apr 8, 2005, 01:32 Communist
 
is the shade of yellow pissing anyone else off?

 
Reply Quote Edit Delete Report
 
52. Re: No subject Apr 7, 2005, 13:29 MMORPGHoD
 
Testing 1.. 2.. 3..

 
Reply Quote Edit Delete Report
 
51. Re: No subject Apr 7, 2005, 01:58 Bearkyn
 
Great to have Blues back up!

 
Reply Quote Edit Delete Report
 
50. Re: No subject Apr 6, 2005, 23:33 ExcessDan
 
I've got my copy of Blood 1 but I have no idea whatsoever on how to get it online with it.

You don't even need that you just need to download the Transfusion Project that I linked to in the original post. The interface has built in TCP/IP support. Get my attention if you grab it and want to try.

Mayor Dan
------------
"Take me Jesus, take me Allah, rape me in my room
Torch our days in paranoia while we gorge ourselves on gloom"
 
ExcessDan
Reply Quote Edit Delete Report
 
49. No subject Apr 6, 2005, 22:36 xXBatmanXx
 
wow.

I didn't think I was going to be correct on so many accounts of what happened.

That is spooky.

xXBatmanXx - fka BatmanX & Batman

Just say NO to drugs propaganda.
 
Avatar 10714
 
In this present crisis, government is not the solution to our problem; government is the problem. / Few men have virtue enough to withstand the highest bidder.
Playing: New dad
Reply Quote Edit Delete Report
 
48. Re: Why this is and who we think we are Apr 6, 2005, 22:08 chiisu81
 
it's good to be back

huzzah to Frans and others!!!!!!!

"I find your lack of faith disturbing." - Darth Vader
 
Smile Like You Mean It
Reply Quote Edit Delete Report
 
47. Re: they finally said it Apr 6, 2005, 21:41 Michael Bolton's hair dresser
 
Woohoo it's back!

 
Dr. Michael Bolton's Hairdreser PhD, MD, LLP, DDS
Reply Quote Edit Delete Report
 
46. Re: The new old site Apr 6, 2005, 21:25 BDFD
 
I like it better the way it was previously....

 
Reply Quote Edit Delete Report
 
45. The new old site Apr 6, 2005, 20:15 Jim
 
Some here refer to the "outage" and "Blues News is back", but I really don't think we ever "lost" it, except for a few hours on the first day the server melted down. Blue and crew did an exemplary job of keeping the site going, and the community managed to keep itself together, that it had become barely noticeable that anyting was "missing" save for the archives and individual news forums.

Good job to everyone involved, and thanks again!

 
Jim
Reply Quote Edit Delete Report
 
44. Re: No subject Apr 6, 2005, 18:12 Warhawk
 
Oh, what is the official way to shorten Gunnar's name? Assuming he is a wonder dog too of course.

GTWD is the obvious choice, but maybe something completely different is in order?


*** Warhawk ***

We came here to rob them and that's what we're gonna do - beat their heads in, gouge their eyes out, slash their throats. Soon as we wash the dishes.
 

Have I lied to you? I mean, in this room? Trust me, leave that thing alone. - GLaDOS

Did IQs just drop sharply while I was away? - Ripley
Reply Quote Edit Delete Report
 
43. Re: No subject Apr 6, 2005, 17:44 Bronco
 

May HTWD bite your crotch...

This is sooo nice to see.

Glad you are back!

Oh, what is the official way to shorten Gunnar's name? Assuming he is a wonder dog too of course.

-TPFKAS2S
http://www.braglio.com
teh m4y0r ownz a GabeN b4ckp4k
http://www.azurebrigade.com
- WoW Guild / Board
 
Avatar 10139
 
-TPFKAS2S
Reply Quote Edit Delete Report
 
42. "You meddling kids" Apr 6, 2005, 17:25 Killer Kane
 
I miss the sig pics tho.....any chance ?

 
Avatar 14092
 
Reply Quote Edit Delete Report
 
41. Good to see it back! Apr 6, 2005, 17:18 Tim1_2
 
And I hope that Gibby guy has insurance...yikes.  
Currently playing: DayZ
Reply Quote Edit Delete Report
 
40. Re: No subject Apr 6, 2005, 16:42 Hump
 
anyone up to play some Blood 1 or Shadow Warrior tonight some time after 1030pm EDT?

I've got my copy of Blood 1 but I have no idea whatsoever on how to get it online with it.


----------------------------------------------------------------------
"Sometimes I dream about....cheese. Here, have a medkit"
- Half Life 2 NPC
 
Avatar 10137
 
----------------------------------------------------------------------
"Both the “left” and the “right” pretend they have the answer, but they are mere flippers on the same thalidomide baby, and the truth is that neither side has a clue."

- Jim Goad
Reply Quote Edit Delete Report
 
39. BluesNews Lives! Apr 6, 2005, 16:34 Cholma
 
Huzzah! It's alive! Alive!

Sounds like you took Mr. Toad's Wild Ride getting things back together.


 
Avatar 3753
 
Reply Quote Edit Delete Report
 
38. Re: No subject Apr 6, 2005, 16:06 Warhawk
 
Dude, that is SUCH an insult to our favorite web master!

May HTWD bite your crotch...

Where is Gunnar - the cheering section? Sitting on his chest? Getting HTWD a beer from the fridge?


*** Warhawk ***

We came here to rob them and that's what we're gonna do - beat their heads in, gouge their eyes out, slash their throats. Soon as we wash the dishes.
 

Have I lied to you? I mean, in this room? Trust me, leave that thing alone. - GLaDOS

Did IQs just drop sharply while I was away? - Ripley
Reply Quote Edit Delete Report
 
37. Re: No subject Apr 6, 2005, 16:02 nin
 
*Grabs rubber mask off Blue's face*

Everyone: DEREK SMART?!?!?!


Dude, that is SUCH an insult to our favorite web master!

May HTWD bite your crotch...




May 3rd, 2005 - "We are preparing to destroy you." http://www.nin.com
 
http://www.nin.com/pub/tension/
Reply Quote Edit Delete Report
 
36. Re: No subject Apr 6, 2005, 15:55 Warhawk
 
I prefer a more plausable, Scooby Doo ending to 'The Case of the Burned out Server'...




*** Warhawk ***

Why don't you knock it off with them negative waves? Why don't you dig how beautiful it is out here? Why don't you say something righteous and hopeful for a change?

Crap!
 

Have I lied to you? I mean, in this room? Trust me, leave that thing alone. - GLaDOS

Did IQs just drop sharply while I was away? - Ripley
Reply Quote Edit Delete Report
 
55 Replies. 3 pages. Viewing page 1.
< Newer [ 1 2 3 ] Older >


footer

.. .. ..

Blue's News logo