GitLab on Tor Project status

[Resolved] Router maintenance at Hetzner

Tue, 03 Dec 2024 03:30:00 +0000

Hetzner has planned an emergency maintenance window on all of their routers, which will cause a network outage on all of our hosts in their datacenters. The maintenance window is planned on December 3rd 2024 from 3:30 UTC to 5:30 UTC, during which the network may experience spurious outages.

Services should come back online automatically as soon as network connectivity is restored by Hetzner.

[Resolved] GitLab migration to another machine cluster

Thu, 28 Nov 2024 18:00:00 +0000

Starting on November 28th at 18:00 UTC, gitlab.torproject.org will be taken offline in order to migrate it to our other machine cluster.

The transfer time is currently estimated at 18h, so gitlab will be coming online on the next day, friday 29th. If we’re lucky the transfer might finish sooner.

If you have any issues during that time, please reach out to us on IRC or via email.

[Resolved] Gitlab and CollecTor outage

Wed, 06 Dec 2023 14:00:00 +0000

We’ve experienced heavy load and unresponsiveness on some of our services (e.g. Gitlab and CollecTor) leading to outages and disruptions.

The issue seems to have resolved itself, investigation seemed to show this was a routing issue upstream.

Update: issue have crept up again, root cause was elevated temperature with the hard drives on the affected server. Upstream has replaced fans in the server and situation has returned to normal.

See issue tpo/tpa/team#41429 for detailed analysis and updates.

[Resolved] outage at main provider

Sat, 04 Feb 2023 01:57:31 +0000

We are witnessing a large outage at our main service provider, Hetzner. According to the information we have gathered so far, four switches (4!) have failed and that affects four (yes, again, 4!) of the servers in the 8-node Ganeti cluster.

Affected servers:

check-01
chives
colchicifolium
cupani
fsn-node-01
fsn-node-02
fsn-node-04
fsn-node-07
gitlab-02
gnt-fsn
henryi
loghost01
majus
materculae
media-01
onionoo-backend-01
onionoo-backend-02
onionoo-frontend-02
perdulce
polyanthum
relay-01
static-master-fsn
staticiforme
submit-01
tbb-nightlies-master
weather-01

Upstream resolved the situation after a few hours of downtime. According to Hetzner it was “a short disruption of a line card in one of our access routers”. Details of the incident are available in tpo/tpa/team#41057.

[Resolved] routing issues at main provider

Tue, 25 Jan 2022 06:00:00 +0000

We are experiencing intermittent network outages that typically resolve themselves within a few hours. Preliminary investigations seem to point at routing issues at Hetzner, but we have yet to get a solid diagnostic. We’re following that issue in issue 40601.