On today's events

[Balandar]

Well, today has been a rough day. I know I cannot offer any excuses for what's going on, but I can offer some explanations. I posted the formal one up in the Announcements forum already, but I figured you guys might want the chance to yell back at us.

So here's the less dressed-up version of what happened today. Short form, we're jinxed. I say it that was because I am not exactly a hardware or database guru, so I can't give you actual hard details on what's going on.

First there was a hardware failure. We had not only installed more hardware than we had budgeted for any other launch, but we had dark hardware ready to go in the event of overload on the registration servers. As we saw that you were having trouble registering, we lit up the dark hardware. But it turns out that wasn't the problem--the problem was a silent hardware failure in the device that manages the load across the web servers.

That issue should now be resolved, and we apologize that it took so long to identify. If you're still having registration problems, there's some threads on the Tech Support forum for you to post on with reports.

Now, we have a separate problem. We are getting internal Oracle errors of some sort with Oracle 9i. It not only takes servers down, it keeps them from coming back up. This is what you're seeing if you see "Loading..." -- a server failing to come back up. We have Oracle on the phone. We are trying to find out what's up. I don't have more specific details because I do not want to disturb the guys working on this too much while they work on a resolution for you.

I am sure there are a ton of questions, I am sure that there will be many folks who are very upset--justifiably (hey, I'm upset too, this is not exactly how we wanted things to go today. ). Q and I will try to answer what we can and keep you updated as often as we can. I do ask that we try to keep the thread somewhat civil despite the trying circumstances, so that everyone can find the useful info faster and the noise level stays down.

-Raph Koster,