mail.python.org recovery

2015-10-29T01:44:25Z

Oct 29, 01:44 UTC
Resolved - The mail.python.org service is now in a stable and reliable state. This incident is resolved.

Oct 27, 16:56 UTC
Update - The new mail.python.org infrastructure is continuing to function as expected. There may be some continued delays for users who's ISP enforces IP based constraints on mail delivery. The postmaster team is working to notify ISPs of our change in IP.

Oct 27, 06:08 UTC
Update - Outbound mail has caught up for the vast majority of recipients. We are monitoring as the backlog of greylisted mail empties. This may take some time. Additionally we'll be keeping track of any functionality which may have been affected by migrating to the new infrastructure.

Oct 27, 03:55 UTC
Update - An issue without outbound mail on the new mail.python.org has been identified and resolved. We are working through the backlog of messages quickly now. We'll continue to monitor all aspects of mail delivery and mailing list activity looking out for any issues. Please don't hesitate to reach out to postmaster@python.org with any unexpected behavior.

Oct 26, 06:15 UTC
Update - The restored mail server continues to operate as expected. We have tuned the Mail Handling Services to more fairly service the backlog incurred during downtime and continue to monitor as it works through the backlog.

Oct 25, 22:34 UTC
Update - Outbound mail from the new mail.python.org server may experience abnormally long delays in delivery due to an IP Address change which was required during restoration. We are continuing to monitor that greylisting retries are occurring, as well as monitoring for whitelisting notifications from other ISPs.

Oct 25, 21:44 UTC
Update - all of mail.python.org's core services are now running and available. We are continuing to monitor for issues. If unexpected behavior is seen, please notify postmaster@python.org

Oct 25, 20:42 UTC
Update - We are continuing to work on bringing Mail Handling and Mailing List processing services fully online.

Oct 25, 19:01 UTC
Monitoring - Required DNS updates have been rolled out to point Mail Handling Services and Mailing List Web traffic to the restored server. We are monitoring and will be working to verify all systems are operational.

Oct 25, 17:14 UTC
Update - Restoration is progressing. The postmaster team is continuing to recover configurations and services to a working state. During this time we are also beginning to validate functionality before updating requisite DNS entries.

Oct 25, 03:22 UTC
Update - The postmaster team is continuing efforts to restore mail.python.org's functionality. We will continue to update with any new information as it is available.

Oct 24, 16:44 UTC
Update - The backup has been transferred to the new instance created for recovery. We are working with the postmaster team to restore services on the new host.

Oct 23, 23:23 UTC
Update - We've been informed that the server hosting mail.python.org has failed. We're brining up a new mail server and restoring from backup.

Oct 23, 21:33 UTC
Identified - The hardware hosting mail.python.org has apparently failed and is unable to boot. We are awaiting more information from the data center facility.

Oct 23, 20:03 UTC
Investigating - Beginning 2015-10-23 10:00, the server hosting mail.python.org became unresponsive and inaccessible. We are currently working to identify the cause of the crash and to get a technician in the data center to ascertain the state of the hardware.

This outage affects all @python.org email accounts, mailing lists, and access to mailing list archives.