PAFOA Forum hosting infrastructure upgraded - Report issues here

[danp]

As announced last week, today I upgraded the hosting infrastructure that the forum runs on. This migration has largely gone as planned except for a few minor issues I have found and fixed.

With that said, I fully expect there will be minor issues that we will need to find, debug, and fix. Additionally the forum is now configured to automatically scale servers up and down to match traffic, but I am certain we will need to dial that configuration in over the coming weeks to find the ideal setup that keeps the forum running fast regardless of the traffic levels.

Please use this thread to report bugs, broken functionality, and other weirdness you see.

Please do not use this thread to post feature requests or changes to the forum, only report stuff that is broken.

Server Stability

Problem: We are still seeing occasional 502 Bad Gateway/503 Service Unavailable/504 Gateway Timeout errors when the servers get maxed out until a new one comes online.
Status: Ongoing, but appears to be mostly resolved
Theories:

• Servers are maxing our available memory, may need to adjust php-fpm's process/children settings to leave more headroom and tweak the memory target autoscaling rule
• vBulletin-based searches may be spiking the load and blocking other page requests

Attempted Remediations:

• Increased task memory allocation from 2048 MB to 3072 MB: This appears to have significantly improved reliability and reduced the 50X errors being returned.

Slow/Duplicate posting of threads/replies/quick replies

Problem: There are times when posting is slow and/or times out, and can sometimes result in duplicate posts
Status: Ongoing, but appears to be mostly resolved
Theory: This is likely related to the general server stability


Human Verification Images Not Loading

Problem: Areas where the forum asks you to verify you are a human with an image are loading blank images
Status: Resolved
Theory: Misconfiguration with image manipulation libraries on service
Attempted Remediations:

• Temporarily disabled HVI on registration and password reset forms so they are usable until underlying issue is fixed properly.

Root Cause: libfreetype/libjpeg were not properly configured for the php/gd extension


Attachment Images Not Loading

Problem: Uploaded image attachments are not displaying when posted
Status: Very Likely Resolved
Theory: Misconfiguration with image manipulation libraries on service
Root Cause: libfreetype/libjpeg were not properly configured for the php/gd extension