PAFOA Forum hosting infrastructure upgraded - Report issues here

[ntouchet]

I was reading a For Sale in Classifieds and linked out to Armslist for pics.

When I came back to the forum most text disapeared. I went to Home page and back to Classifieds with no change.

I closed everything reopened the session and all is well. Will report if this recurs.

[Walleye Hunter]

I'm having issues when posting replies again, I get that leave page message.

[sota]

Not sure if you're still working on it or not, but I just got a 504 Gateway time out error.

ditto. been getting 504 gateway nginx 1.5.7 errors since yesterday.

[danp]

The site appears to have become very unfriendly to Internet Explorer 11. It wouldn't allow me to add reputation (very important) or Reply with Quote. Frankly, I'm surprised the Reply to Thread worked.

I will load up IE and look into this.

I'm having issues when posting replies again, I get that leave page message.

ditto. been getting 504 gateway nginx 1.5.7 errors since yesterday.

My logs show this is still happening a few times an hour, I will have to dial this in over time with the servers and figure out what is causing spikes.

[danp]

I am quite confident this is the same underlying problem as the Human Verification Images. The servers are missing a package that allows them to dynamically generate/manipulate images.

This will be a slightly more complicated fix, and requires a bit more testing, I hope to have this fixed tomorrow, but today may not be likely.

Bob/Streaker: I have just deployed a fix for image generation, can you re-check the images you said were broken yesterday?

I have verified that the Human Verification Images are working again, and I am assuming the same root cause for the broken attachments you were seeing.

[danp]

The site appears to have become very unfriendly to Internet Explorer 11. It wouldn't allow me to add reputation (very important) or Reply with Quote. Frankly, I'm surprised the Reply to Thread worked.

I was just able to give you reputation, and am replying with quote from IE11. Can you provide more information?

[sota]

502 errors here on everything I click through

[sota]

and even more interesting, a 502 error when I submitted my prior post, but shows up if I browse from a different machine.

edit: ok, switching to a browser I haven't used in awhile, and no 502 errors in the same places I've tried so far.
flushed cache data on the "wonky" browser with no affect. going to toss cookies now.
edit 2: flushed everything. testing...
edit 3: that seemed to fix it, more testing to come.

[danp]

502 errors here on everything I click through

The timing here lines up almost exactly when our auto-scaling spun up more instances to handle increasing traffic between 12:09 and 12:12

When we get increased traffic, there is a window of time before the infrastructure launches more servers, but while the existing servers are being stressed, these errors appear to happen during that window.

I'll need to dial in the autoscaling settings so that more servers come online before requests start failing, without being overaggressive and launching too many servers.

[sota]

I won't discount your knowledge of your infrastructure; all I can say is I was getting 502 on mozilla on one machine, no 502 on chrome on another, back to 502 on the mozilla machine, in succession, inside the span of a minute.
I can confirm I appear to be 100% now after tossing everything in the cache on mozilla, including cookies, which would indicate to me some state crumbs possibly.
I offer this a data point.