#AO3's February releases included new admin tools for our Support and Policy & Abuse teams, as well as a bunch of challenge and collection fixes and a host of small updates and improvements. We also upgraded to Rails 8 and Elasticsearch 9! Read more at ao3-news.org/v-0-9-461

AO3 Update

#AO3 is exiting open beta! Learn more about how far we've come since launching open beta, our future plans for improvement, and what you can do to contribute at otw-news.org/4w2dn3tc

Successful login banner on the Change Password page with an error message below it, reading 'Your password has appeared in a data breach from another site and is no longer secure. For your protection, please select a new, unique password.'

To improve account security, we now alert you if your #AO3 password matches a password known to be compromised on other sites. If you see this banner when logging in, please change your password. For more advice on protecting your account: otw-news.org/y8jnay3v

#OTW Systems volunteers have published a postmortem about the issues behind the unplanned #AO3 downtime in early March. Check it out on AO3: ao3-news.org/2v2en3bf

[update] Resolved: #AO3 and #Fanlore are back! Our data center says they have disabled the upstream provider that was causing the outage, but we are continuing to monitor the situation.

[update] Investigating: #AO3 and #Fanlore are currently down. We are investigating.

[update] Resolved: The fix we applied 24 hours ago appears to be holding, so we are resolving this incident. We thank everyone for their patience!

[update] Monitoring: We have made some adjustments based on recommendations from our database vendor and will continue to monitor their impact on system stability.

[update] Investigating: Following guidance from our vendor, we've brought #AO3 back online to help troubleshoot. We expect the site will go back down, although we don't know when that will happen. If you choose to post or update your works while AO3 is up, we strongly recom...

[update] Investigating: AO3 is now fully down. After performing a software upgrade a few days ago, we've been unable to keep our server cluster healthy due to what we believe is a bug or issue with the new release of our database server software.

We're working with our ve...

[update] Investigating: We are investigating reports of errors when trying to load AO3, including errors saying that AO3 is down, which seem to be related to the previous incident. We will update when we have more information!

[update] Resolved: We have resolved the underlying issue, and #AO3 should be behaving as normal for all users again.

[update] Monitoring: We have applied a fix, and #AO3 is accessible again for users. Potential slowness remains and certain VPNs may still be blocked until we verify everything is fully stable again.

[update] Investigating: #AO3 will remain down for at least several hours until we can consult our database vendor's support team and resolve the underlying issue. We're very sorry for the inconvenience.

[update] Investigating: We are investigating reports of errors when trying to load AO3, including errors saying that AO3 is down.

[update] Monitoring: #AO3 is back up! We appreciate your patience. We've got our protections against bot traffic turned up a bit higher than usual, so some users may experience problems if they're using a VPN or similar tools.

[update] Identified: We've identified the problem and are working on a fix. We expect to remain down for several more hours. We apologize for the inconvenience and appreciate your patience!

[update] Investigating: AO3 is currently down. We do not have an estimated return time currently. We're looking into it with one of our third-party support providers and will update when we have more information. Thank you for your patience!

[update] Investigating: We are continuing to investigate reports of slowness and errors when trying to load AO3, including errors saying that AO3 is down.

[update] Investigating: We are currently investigating reports of slowness and errors when trying to load AO3.

We released several batches of bug fixes and code updates in December, focusing on improvements to collections and challenges, error handling, and code optimization. ao3-news.org/v-0-9-452

Resolved: Emails are sending again to Yahoo, AOL, and other related providers. All pending emails have been successfully delivered. Thank you to the Yahoo team for helping us resolve this issue!

We are aware that Yahoo Mail is not currently accepting emails from #AO3, and we are actively working to resolve the issue. We will update you when we know more!

[update] Resolved: AO3 is once again fully functional; all issues should be resolved. If you are still experiencing any issues past this point, please let us know. Thank you for your patience!

[update] Monitoring: We've made some adjustments to reduce the errors and are continuing to monitor the situation.

[update] Investigating: AO3 is currently experiencing slowness and errors. We are aware and are actively investigating.

To improve account security, #AO3 has updated our password change process to prevent users from choosing new passwords that are extremely short. We've also increased the maximum password length. To learn more about how to protect your account, check out: otw-news.org/y8jnay3v

[update] Resolved: This incident has been resolved.

[update] Monitoring: The vendor of our spam checking software has applied some changes, so fewer comments should be incorrectly marked as spam. We will continue to monitor the situation.