Update on last week's search issues
A quick update on the search issues last week. Things are back to normal, so I want to apologize for the inconvenience and Expedite the explanation of what was wrong and what was changed.
The main issues were with searching and browsing characters. It resulted in slow and sometimes invalid results, and even completely broken pages. Some of you were even unable to access the site at all.
According to Cloudflare, there was a huge spike in rate-limited traffic for web searches. We assume there was malicious traffic, but there were also issues with how our servers distributed and managed the load. There were also a couple of issues with our infrastructure which extended the time it took for things to fully recover.
We increased the capacity for search, reduced the cost of each query, resolved the issues with the cluster, and separated the processing for search from the other requests, namely the chat requests. We want to ensure that an issue with the search function doesn’t cause an issue with the other functions.
Things are much better, and we really appreciate your patience during the inconvenient “refresh and hope” period. You should be able to find the characters you want and chat with them, rather than being unable to load the page.
For those of you who are interested in the details, I will include the more extensive explanation here.
Technical Details (September 10 - 11)
- Search Overload: Search Servers were continuously running above 90% CPU, with thousands of search tasks queued. Basic listing requests were doing a lot of unnecessary work, including exact result counts and tag aggregations.
- Increased capacity and cheaper queries: We increased our Search Servers from three to ten nodes, added bounded counting for normal listings, and limited deep pagination to 10,000 results. This reduced unnecessary expensive requests and the need for counting to be repeated.
- Recurring cluster-agent restarts: A version and permissions mismatch caused the cluster-agent software on some workers to restart every 15 minutes. During the restarts, the workers were unable to serve traffic. We fixed the permissions issue and performed a series of one-by-one upgrades on the five control-plane servers.
- Traffic-routing bottlenecks: A number of our ingress machines were fully saturated with CPU work. We alleviated this issue by moving the work to a couple of machines in the cluster that had more CPU capacity.
- Backend CPU usage: A tag-validation check was found to be repeatedly rebuilding matching patterns. We prevented the check from performing the work, while keeping the validation rules. This helped, but was not the only cause of the CPU pressure.
- Search isolation: Normal character listings have been isolated to a dedicated API pool, which allows for concurrent requests to be scaled independently.
- Deployment fixes: Some of the recovery changes made during the incident have been replaced by automatic releases that occur one at a time. We added and tested safeguards to run production releases a lot more carefully with tests on ingress nodes before they are deployed to the rest of the cluster.