PAFOA Back Online After First Major Downtime

[danp]

As I'm sure everyone is aware, PAFOA suffered from it's first major unplanned downtime in the history of the website Thursday night which unfortunately took us a while to fix.

Basically what happened was a hard drive in our main database server crashed and burned which then caused the database server to shut down requiring physical intervention. Sadly, when PAFOA first set up it's hosting, the only way we could afford the hosting required to handle the amount of traffic we get was to get co-location without on-site support based in Michigan. We manage this normally because we have an old friend and volunteer who lives nearby and always goes out whenever we need physical work done. Unfortunately, in this case he was unable to get out to the data center until today.

Thankfully however once he got there, getting things up and running again was not a big deal. We simply took a hard drive from one of our backup web servers, slapped it into the database and got things up and running again.

Thanks to the excellent server setup done by doug and him in the past, absolutely no data was lost. Every server is set up with two hard-drives in a redundant raid array so even though one of the drives died, the other one still had all the data.

Obviously this incident was a big wake-up call as far as our plans for redundancy go and to make it worse, I was very aware that something like this could happen. My plans were to dedicate time to solving this problem in a more long term way after the 4/27 rally was over, however it's obvious that the universe didn't want to wait very long for me to do that.

While I'm not sure exactly what is going to be done over the next few weeks/months to make sure this doesn't happen again, rest assured we will be exploring our options.

Due to the generosity of our members donating, purchasing subscriptions and buying advertising, PAFOA may in fact be able to upgrade it's hosting setup and purchase any necessary hardware to mitigate similar problems in the future.

One again, we sincerely apologize for the inconvenience this has caused people and thank you for your continued support.

[Pro2A]

Good job getting it back up. I learned I could turn my computer off and go outside. So it's a win win Originally I thought it got Rotz'ed... guess that wasn't the case

[General Geoff]

raid-1 + hot spares are your friends

[jcabin]

back to business.

[danp]

back to business.

So, 9mm vs. .45acp...what's better?

[jcabin]

So, 9mm vs. .45acp...what's better?

Hahaha. I can't even debate that unless the thread started with OC vs CC. Did you know if you open carry the CIA follows you with that big ol' eye in the sky? You will be targeted first. Just conceal, it keeps you off the radar.

[ddefazio]

The scene in Space balls when they switch her from suck to blow and he is yelling air, air, AIR... It was like that when I clicked on my favorites button and was able to get back on PAFOA. Thanks Dan for your hard work.

[RugerNiner]

[Emptymag]

So, 9mm vs. .45acp...what's better?

Without this mind-numbing connection to the interweb, I found myself gravitating to the needs of the real world.
Got some things done that I wouldn't have if I were online and spent most of yesterday not thinking about 9mm vs 45ACP, but rather begonias vs petunias...

we got both - problem solved.




p.s.

Dan, do I have to use vacation time for the days off, or will I need to stay late for the next week or two?

Also, my wife asked if this was all Greg's fault...

[danp]

The scene in Space balls when they switch her from suck to blow and he is yelling air, air, AIR... It was like that when I clicked on my favorites button and was able to get back on PAFOA. Thanks Dan for your hard work.

I know how you feel. As stupid as it sounds, this was quite possibly one of the worst 72 hours of my life.

With what happened there was quite literally nothing that could be done without physical access to the machines. Until this morning I didn't even know if perhaps the entire thing got hit by a meteor or something. I felt completely and utterly helpless.

The feeling when my friend let me know it was just a bad HD and we needed to just reconfigure a few things was overwhelmingly good.

Obviously I have some IT work to do.