Python Infrastructure Status

Python Infrastructure

All Systems Operational

python.org Operational

pypi.python.org Operational

docs.python.org Operational

hg.python.org Operational

bugs.python.org Operational

wiki.python.org Operational

mail.python.org Operational

Mailing Lists and Archives Operational

Message Handling Services Operational

pypy.org Operational

speed.pypy.org Operational

psfmember.org Operational

Content Delivery Network Operational

Fastly US East (JFK) Operational

Fastly Asia/Pacific (HK) Operational

Fastly US East (IAD) Operational

Fastly US East (ATL) Operational

Fastly US East (MIA) Operational

Fastly US Central (DEN) Operational

Fastly US Central (DFW) Operational

Fastly US Central (ORD) Operational

Fastly US West (LAX) Operational

Fastly US West (SEA) Operational

Fastly US West (SJC) Operational

Fastly Europe (FRA) Operational

Fastly Europe (AMS) Operational

Fastly Europe (LCY) Operational

Fastly Europe (LHR) Operational

Fastly Asia/Pacific (SYD) Operational

Fastly Asia/Pacific (SIN) Operational

Fastly Asia/Pacific (TYO) Operational

Fastly Asia/Pacific (NZ) Operational

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

System Metrics Month Week Day

PyPI Errors

Fetching

PyPI CDN Miss Latency

Fetching

Past Incidents

Dec 30, 2017

No incidents reported today.

Dec 29, 2017

No incidents reported.

Dec 28, 2017

No incidents reported.

Dec 27, 2017

OSUOSL Ganeti Cluster Maintenance

Completed - The scheduled maintenance has been completed.
Dec 27, 23:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 27, 17:00 UTC

Scheduled - OSUOSL Staff will be performing maintenance and updates to the Ganeti Cluster. During this time, PSF infrastructure at OSUOSL maybe intermittently inaccessible.
Dec 24, 23:39 UTC

Dec 26, 2017

PyPI: Elevated Error Rates for /packages

Resolved - Errors for /packages urls have subsided. During the course of responding to this incident, we determined that the mirror that is intended to act as a failover in situations like this had run out of disk space.

We have begun creating a new mirror with the necessary storage and verifying that failover can take place.

Additionally, we've added monitoring to alert when mirror space is low which should avoid surprises like this in the future.
Dec 26, 20:30 UTC

Monitoring - Errors have subsided, we're continuing to monitor and take other preventative measures.
Dec 26, 19:56 UTC

Identified - The object store that hosts PyPI's files is experiencing intermittent issues serving packages. The error rate is minimal, but we are continuing to work to resolve the issue.
Dec 26, 19:47 UTC

Investigating - Our automated systems detected an elevated error rate for `/packages` urls. We are investigating the issue and will update shortly.
Dec 26, 19:39 UTC

Dec 25, 2017

No incidents reported.

Dec 24, 2017

No incidents reported.

Dec 23, 2017

PyPI Outage

Resolved - Incident is resolved. During the database failover 37,889 requests received an error at our CDN edge and 3,723 requests to backend servers resulted in an error.

No data loss or ongoing impact to PyPI was detected.
Dec 23, 14:45 UTC

Monitoring - PyPI service is restored and we are actively monitoring and ensuring data consistency. All looks good and this incident will be marked resolved shortly.
Dec 23, 14:42 UTC

Identified - Our high availability database system underwent a failover event causing all backend web servers to stop serving requests.

The database system failover took approximately six minutes but has completed and servers are coming back online.
Dec 23, 14:40 UTC

Investigating - We are investigating hard outage of PyPI's backend servers.

Simple Index and Package installs should continue to be served by the CDN while we investigate.
Dec 23, 14:38 UTC

Dec 22, 2017

No incidents reported.

Dec 21, 2017

OSUOSL Ganeti Cluster Upgrade

Completed - The scheduled maintenance has been completed.
Dec 21, 20:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 21, 17:00 UTC

Scheduled - OSUOSL Staff will be performing a Ganeti Cluster Upgrade from Ganeti 2.6.2 to 2.15.2. During this time VM's at OSUOSL maybe intermittently down as the upgrade is completed.

A list of hosts that maybe affected at OSUOSL is located here: https://github.com/python/psf-chef/blob/master/hostnames.
Dec 17, 23:30 UTC

Dec 20, 2017

No incidents reported.

Dec 19, 2017

No incidents reported.

Dec 18, 2017

mail.python.org Kernel Maintenance

Completed - mail.python.org maintenance is complete!
Dec 18, 03:56 UTC

Update - Backup has completed, and we are completing the maintenance. No further disruptions are expected.
Dec 18, 03:48 UTC

Update - A routine backup which was a planned part of this maintenance window is taking much longer than anticipated and may delay restoration of service for mail.python.org
Dec 18, 03:27 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 18, 03:00 UTC

Scheduled - mail.python.org's Mail Transit and Web access may be unavailable for brief periods of time as we reboot.

Maintenance is expected to take under an hour and we hope for only one reboot leading to a brief window where MTA and HTTP access will be disrupted.
Dec 17, 22:20 UTC

Dec 17, 2017

mail.python.org unavailable

Resolved - This incident has been resolved.
Dec 17, 02:36 UTC

Monitoring - Our hosting provider for mail.python.org is executing a migration to move the instance backing the service to a new host.

The expected maintenance window ends at 2017-12-17 05:00 UTC, we'll update then if the outage extends beyond that time.
Dec 17, 02:29 UTC

Dec 16, 2017

No incidents reported.