PAFOA has switched to new hosting. Report problems here.

[danp]

If you're reading this than it appears the transition from our outdated colocation and into Amazon's Cloud to hang out with all the other cool kids was successful.

In theory this hosting infrastructure should be significantly more stable in the long run since hardware failures are now Amazon's problem, not ours. No more 2 days of downtime while my friend in Michigan gets a chance to pop over to the data center and figure out what went wrong.

Some of the highlights/advantages:

1. Completely automated provisioning of servers: In a catastrophe situation where the entire server infrastructure disappears, I could rebuild it in about 10-20 minutes with one command.
2. Automatic scaling of servers: When load gets high, new servers will automatically be spun up to meet the demand using the previously mentioned provisioning. When traffic dies down, unnecessary servers will be killed off.
3. Self-healing servers: We'll always have a minimum of two servers so that if one craps out, the other one can handle the load until a new one is started based off of that autoscaling.
4. We now have ridiculously redundant backups of 5-minute windows for 30 days on our database. We've never had to recover from a database backup in the past, but if we ever have to in the future we are more than covered.

Some of the disadvantages:

1. Amazon sometimes has to do server maintenance on the cloud database every few months and while we can specify a window in which they do it (around 4am) there are times where the site will be down for 5-10 minutes while they do that.
2. While I won't have finalized numbers for a week or two when I get a better idea of how many of Amazon's servers it takes to keep this thing chugging along, I would assume that we just doubled the cost of our hosting, at least. With that said, we can afford it, but if you've been considering donating or buying a subscription, now couldn't hurt

Amazon is not perfect and has had downtime in the past and likely will in the future, however with this infrastructure recovering from such downtime should be significantly faster and less painful than on our old setup.

Lastly, while CR and I tested stuff pretty thoroughly, there is almost guaranteed to be something I overlooked, if you notice something not right on the forum please report it here and I'll look into it. I'm mostly looking for pages that error, pages that don't load properly, broken images, broken links, etc.

If you find a problem that seems like it should be reported privately (security/privacy issues, etc) PM me directly.

Known Open Issues

• Search rate limiting: I was 99% sure this had been fixed and I can't personally recreate it, but people are still reporting it sporadically. For the time being I have disabled search rate-limiting, we'll see how the servers handle that and if it's not a problem we'll just leave it disabled. Search to your heart's content!
• The Forum RSS feeds are not working: This appears to be sporadic working sometimes but not others.
• Double posting of posts: Not sure exactly why this is happening, one theory is that if one server behind the load balancer is slow to respond it sends the request to another one resulting in two servers handling the same request.
• Sporadic errors when uploading pictures: Working theory is that pictures over a certain size are rejected by our web server. Waiting on file size examples from people reporting the problem.
• Various rate limiting messages: Still not exactly sure about this one.

Please report new issues by replying to this thread.

[Justin///M]

Well done, and thanks!

Justin

[danp]

Well done, and thanks!

Justin

No problem! It's something that needed to be done a while ago, I just couldn't fit it into my schedule with running my business. I finally got a break between clients and jumped on the opportunity. It was quite a bit of work, but I think it will be worth it in the long run.

I just pre-apologize if I broke anything and we just haven't noticed yet

I'm sure there will be some growing pains over the next week or so.

[whoshisface]

Hey, it's back up!

[danp]

Hey, it's back up!

Now to see if it stays up, that's the real metric here!

[danp]

Out of curiosity, does anyone feel that the forum is faster? It feels much faster to me.

[topsykretts]

Yeah bro, feels much faster. Must be them new CIA quantum mirror ports.

[Mr_Gixxer]

Out of curiosity, does anyone feel that the forum is faster? It feels much faster to me.

Still works great on my tablet!

Now, time to subscribe...

[mjfletcher]

So far looks good, but there's almost nobody awake at this hour, let's see what happens tomorrow during legislative disucussions.

[IronSight]

No problems here so far.

Amazon then ... well I won't get into detail about my opinions on that.