Fucking servers are down AGAIN

Post by **Martrae** » Sat Dec 18, 2004 7:15 am

SoE Job Opening

Hey, wasn't WoW just advertising for a DBA, too? Wonder if the position was filled Thursday.

Diekan · Post by **Diekan** » Sat Dec 18, 2004 7:58 am

If it was, that was a pretty stupid thing to do - he could land his nerd ass in jail for something on that scale....

LOL I'd like to see something like that on HBO... Nerds in Prison.

Post by **Lyion** » Sat Dec 18, 2004 8:15 am

Martrae wrote:SoE Job Opening

Hey, wasn't WoW just advertising for a DBA, too? Wonder if the position was filled Thursday.

Woot, another game company that wants me. Unfortunately, I'm perfectly happy where I am.

kaharthemad · Post by **kaharthemad** » Sat Dec 18, 2004 8:28 am

Geez Lyion...drag your sorry ass over to Sony and do something that would make the nerd kingdom you help reside over happy.

Post by **Lyion** » Sat Dec 18, 2004 8:46 am

I'm really curious what caused this downtime. I seriously doubt it was database. It appears they run Oracle RAC on Linux, which is pretty bullet proof and very easy to restore, even for larger instances.

My guess is they rolled out new code, accidently deleted the old code from the CVS, and then introduced an assload of new problems. So, they're debugging and fixing on the fly.

Either that or there is a bug they found that rolling back isn't fixing, so they're trying to fix it in real time.

This should never happen, unless they have catastrophic hardware failure, and even that should be replaceable in a few hours.

Most enterprize shops run a Development -> QA -> Production configuration.
Smaller shops run a Dev/QA Combo to production, which is riskier. It appears SOE <and WOW> both do not use test servers.

SOE needs to run a test server, give people INCENTIVES to use it <i.e. 1/2 exp cost to level, or whatever> and test this crap before rolling it to production

Post by **Martrae** » Sat Dec 18, 2004 8:48 am

They did roll out a test server last week. Don't think many people know it's there tho.

Post by **Lyion** » Sat Dec 18, 2004 8:50 am

Apparently they aren't testing their code on the test server, or this wouldn't of happened.

Also, they need to give incentives for people to play there.

kaharthemad · Post by **kaharthemad** » Sat Dec 18, 2004 9:03 am

I liked what Dark Age did for their test server. They allowed you once a week or every time (past 20) you leveled to make a copy of your main on the test
you could change race, sex, and abilities to test out features of the game. DAOC had a very well used server due to this.

As for this problem, yah total QA problem. Correct me if I am wrong Lyion but a ORACLE does not take this long to restore and load.

Post by **Lyion** » Sat Dec 18, 2004 9:06 am

Not to restore and load, but if they have massive indices to recreate, it can be very time consuming. Without an index, certain things would be slow as molasses.

A few weeks ago I did an Index build that took 12 hours to create.

kaharthemad · Post by **kaharthemad** » Sat Dec 18, 2004 10:03 am

I see the damn forum is down too...as the guiness ad says "BRILLIANT"

kaharthemad · Post by **kaharthemad** » Sat Dec 18, 2004 1:19 pm

Guk, Najena, Blackburrow, Everfrost, Oggok, and Lavastorm are currently online and available for play. We will continue to bring game worlds up as we are able to do so. Please monitor this thread for the latest updates.

We are working with a hardware vendor to address some remaining issues with the following servers, which will be experiencing additional downtime: Befallen, Permafrost, Crushbone, Runnyeye, Mistmoore, Oasis, Innothule, Steamfont, Faydark, and Innovation.

We will continue to work around the clock until all worlds are online. Thank you for being patient while we get all the servers up and running!

Ryan "Blackguard" Shwayder

in IT terms this is a system fuckup.

sounds like a system incompatability issue if you have that many servers toasty.

This sounds so much like a whoops....Brenlo split his egg nog on the server rack.

Captain Insano · Post by **Captain Insano** » Sat Dec 18, 2004 2:00 pm

Diekan wrote: 10am EST

OH NOEZ 10am EST?!!!111

How bout this.. get a fucking job and you won't need to play at 10am.

Harrison · Post by **Harrison** » Sat Dec 18, 2004 2:35 pm

You didn't see the part where he said he works nights now did you?

And I know you did that just to piss him off, but I figured I'd nip it at the bud

labbats · Post by **labbats** » Sat Dec 18, 2004 2:40 pm

I have no sympathy for anyone who buys and plays a Sony online game within the first 3 months of release.

Diekan · Post by **Diekan** » Sat Dec 18, 2004 6:36 pm

Aren't YOU the one who's /cough self employed LOL. I have a REAL job try again thx.

Post by **Martrae** » Sun Dec 19, 2004 6:57 am

Hello Everyone,

We sincerely apologize for the downtime this weekend. We had some hardware issues that caused us to have to tread very carefully in bringing the game back up. We wanted to triple check everything to make sure there wouldn’t be any major problems, and it’s taken a while to both fix the problem (we had to fly out engineers from one of our hardware vendors) and to bring each of the servers back online because of the careful validation process we need to go through. We have to do this very carefully and we hope you can understand our desire to do this right. We're trying to balance that with getting the game back up as quickly as possible. So far most servers are back up and the rest should be up in the next 4-6 hours... hopefully less. I realize that ETA's that come and go can frustrate people, so we do our best to give you the most realistic estimates possible.

Out of respect for our players, we’re doing the following:

1) Every player will receive 3 free days.

2) Early this week (Monday or Tuesday) we will have a 24 hour period where we will have accelerated XP gain. Details are still being worked out and we’ll communicate this to you as we firm up the plans.

We are committed to providing you with excellent service. Our team has been here for over 40 hours straight, and I can tell you our dedication to doing right by our players is the thing that’s foremost on everyone’s mind here at SOE.

Thank You,

John Smedley

President, Sony Online Entertainment

We wanted to thank you for your patience with the issues that we've needed to deal with over the last several hours. As a small token of our appreciation, we have increased experience gain for both adventuring and tradeskilling. This additional bonus will continue for 48 hours, starting with our normal update on Sunday morning at 7AM PST, and last until our update of Tuesday morning. Once again, thank you for your patience, and incredible support.

The EverQuest II Team

Bruce A. Ferguson
Live Producer, EverQuest II

Honestly, I'm amazed. I really expected some token gesture like a free day or something.

Jei · Post by **Jei** » Sun Dec 19, 2004 9:05 am

yeah, that is pretty amazing. makes the downtime worth it lol =p

Post by **Martrae** » Sun Dec 19, 2004 9:06 am

Hey, I'm used to old school EQ and the 'it's our servers and you can kiss our ass' attitude. This is nothing short of a miracle.

Diekan · Post by **Diekan** » Sun Dec 19, 2004 9:27 am

I agree - I clearly remember those days.... TN would be down for hours at a time lol at one point it seemed like it was on a weekly basis - with ZERO compensation.

Kaeerwen · Post by **Kaeerwen** » Sun Dec 19, 2004 5:12 pm

Yeah... there was one point where the servers were up maybe a total of 12 hours (in like 30-minute intervals) during an entire week... except mobs were bugged, and you couldn't fight anything. ...Then the server would crash again, and lives could resume.

brinstar · Post by **brinstar** » Mon Dec 20, 2004 1:16 am

Lyion wrote:Apparently they aren't testing their code on the test server, or this wouldn't of happened.

wait are you talking about eq1 or eq2 :dunno:

Post by **Lyion** » Mon Dec 20, 2004 7:32 am

I think it equally applies to EQ, SWG, and EQ2.

Hello Mcfly, use an effin test server.

Tacks · Post by **Tacks** » Mon Dec 20, 2004 8:40 am

Hi guys, good luck.

Post by **Tossica** » Wed Dec 22, 2004 5:16 pm

Holy shit. What a bunch of whiny little bitches. It's a fucking game. Get over it.

mofish · Post by **mofish** » Wed Dec 22, 2004 10:07 pm

Tossica wrote:Holy shit. What a bunch of whiny little bitches. It's a fucking game. Get over it.