ThousandEyes Status

Internet Insights Outage detection missing data

Monitoring - Operation teams has identified a service degradation for Internet Insight affecting the Outage Detection and Outage Map services. Users may experience missing alerts, clearing alerts, and reports for outages between 14:36 to 15:26 UTC.
Feb 29, 2024 - 18:53 UTC

Uptime over the past 90 days. View historical uptime.

Agent Configuration and Data Collection Operational

90 days ago

99.92 % uptime

Today

Cloud and Enterprise Agents: Registration controller Operational

90 days ago

100.0 % uptime

Today

Cloud and Enterprise Agents: Test assignment and configuration controller Operational

90 days ago

100.0 % uptime

Today

Cloud and Enterprise Agents: Data ingress Operational

90 days ago

99.64 % uptime

Today

Endpoint Agents: Test assignment and configuration controller Operational

90 days ago

100.0 % uptime

Today

Endpoint Agents: Data submission collector Operational

90 days ago

100.0 % uptime

Today

ThousandEyes Platform and API Operational

90 days ago

99.97 % uptime

Today

Platform Availability Operational

90 days ago

99.98 % uptime

Today

API Availability Operational

90 days ago

100.0 % uptime

Today

Test data availability Operational

90 days ago

99.79 % uptime

Today

Reports and Dashboards Operational

90 days ago

99.98 % uptime

Today

Snapshots and Share Links Operational

90 days ago

100.0 % uptime

Today

SAML/SSO Operational

90 days ago

100.0 % uptime

Today

Configuration and Management Operational

90 days ago

100.0 % uptime

Today

Embedded Widgets Operational

90 days ago

100.0 % uptime

Today

Customer Success Chat Operational

90 days ago

100.0 % uptime

Today

Alerts and Notifications Operational

90 days ago

100.0 % uptime

Today

Alert processing Operational

90 days ago

100.0 % uptime

Today

Notification dispatching Operational

90 days ago

100.0 % uptime

Today

Webhook and Integrations Operational

90 days ago

100.0 % uptime

Today

Email notifications Operational

90 days ago

100.0 % uptime

Today

Internet Insights collection and processing Operational

90 days ago

100.0 % uptime

Today

BGP Operational

90 days ago

100.0 % uptime

Today

Public BGP collection and processing Operational

90 days ago

100.0 % uptime

Today

Private BGP Collection & Processing Operational

90 days ago

100.0 % uptime

Today

Agent Repositories and Downloads Operational

90 days ago

100.0 % uptime

Today

Ubuntu repository Operational

90 days ago

100.0 % uptime

Today

RHEL / CentOS repository Operational

90 days ago

100.0 % uptime

Today

Appliance downloads Operational

90 days ago

100.0 % uptime

Today

Custom Appliance Downloads Operational

90 days ago

100.0 % uptime

Today

Endpoint Agent downloads Operational

90 days ago

100.0 % uptime

Today

Cloud Agents: Americas Operational

Data Centers: AMER Operational

Broadband Providers: AMER Operational

Mobile Providers: AMER Operational

Azure: AMER Operational

AWS: AMER Operational

Google Cloud: AMER Operational

Alibaba Cloud: AMER Operational

Cloud Agents: Europe, Middle East and Africa Operational

Data Centers: EMEA Operational

Azure: EMEA Operational

AWS: EMEA Operational

Google Cloud: EMEA Operational

Alibaba Cloud: EMEA Operational

Cloud Agents: Asia Pacific Operational

Data Centers: APAC Operational

Azure: APAC Operational

AWS: APAC Operational

Google Cloud: APAC Operational

Alibaba Cloud: APAC Operational

ThousandEyes' corporate website Operational

90 days ago

100.0 % uptime

Today

ThousandEyes corporate website Operational

90 days ago

100.0 % uptime

Today

Internet Outages Map Operational

90 days ago

100.0 % uptime

Today

ThousandEyes for OpenTelemetry Operational

90 days ago

100.0 % uptime

Today

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

Scheduled Maintenance

Planned maintenance for aptproxy.thousandeyes.com and yumproxy.thousandeyes.com Mar 4, 2024 04:30-06:00 UTC

Maintenance for aptproxy.thousandeyes.com and yumproxy.thousandeyes.com is planned on Monday, March 4th 2024, between 04h30 and 06h00 UTC.

During this change, the ThousandEyes Engineering Operations team will perform planned maintenance activity on aptproxy.thousandeyes.com and yumproxy.thousandeyes.com domains. Customers might face intermittent issues while connecting to these aptproxy.thousandeyes.com and yumproxy.thousandeyes.com domains.

Workaround: none
Posted on Feb 28, 2024 - 14:30 UTC

Maintenance in multiple Cloud locations Mar 4, 2024 13:00-21:00 UTC

Update - We will be undergoing scheduled maintenance during this time.
Feb 26, 2024 - 17:19 UTC

Scheduled - Description: The Cloud Agents Operations Team will perform a capacity increase on multiple Cloud Agents. This also includes adding IP addresses to the Cloud Agents.

When: 2024-03-04 from 13:00 to 21:00 UTC

Impact: No impact is expected

The IP addresses added on Bangkok, Thailand (Alibaba ap-southeast-7):

- 8.213.215.26

The IP addresses added on Fuzhou, China (Alibaba cn-fuzhou):

- 47.122.38.229

The IP addresses added on Guangzhou, China (Alibaba cn-guangzhou):

- 8.134.153.132
- 8.138.119.50
- 8.134.85.16

The IP addresses added on Heyuan, China (Alibaba cn-heyuan):

- 47.113.224.143

The IP addresses added on Nanjing, China (Alibaba cn-nanjing):

- 47.122.28.44

The IP addresses added on Qingdao, China (Alibaba cn-qingdao):

- 47.104.73.0

The IP addresses added on Riyadh, Saudi Arabia (Alibaba me-central-1):

- 8.213.36.1

The IP addresses added on Seoul, South Korea (Alibaba ap-northeast-2):

- 8.220.201.136
- 8.213.159.185
- 8.213.148.216

The IP addresses added on Ulanqab, China (Alibaba cn-wulanchabu):

- 8.130.82.243

The IP addresses added on Zhangjiakou, China (Alibaba cn-zhangjiakou):

- 47.92.206.53

The IP addresses added on Atlanta, GA (AWS Local Zone):

- 15.181.88.93

The IP addresses added on Boston, MA (AWS Local Zone):

- 68.66.113.243
- 68.66.120.108

The IP addresses added on Chicago, IL (AWS Local Zone):

- 15.181.178.97
- 15.181.179.151

The IP addresses added on Copenhagen, Denmark (AWS Local Zone):

- 96.0.25.0
- 96.0.25.17

The IP addresses added on Lagos, Nigeria (AWS Local Zone):

- 96.0.46.66
- 96.0.45.136

The IP addresses added on Los Angeles, CA (AWS Local Zone):

- 15.254.38.106
- 70.224.235.118

The IP addresses added on Santiago, Chile (AWS Local Zone):

- 96.0.49.175
- 96.0.49.69

The IP addresses added on Milan, Italy (AWS eu-south-1):

- 18.102.77.115
- 18.102.56.251
- 18.102.220.244
- 35.152.93.212
- 18.102.217.154
- 18.102.185.73
- 15.161.191.111
- 18.102.198.171
- 18.102.97.159
- 15.161.81.158

The IP addresses added on Zurich, Switzerland (AWS eu-central-2):

- 16.63.209.254
- 16.63.151.100

The IP addresses added on Doha, Qatar (GCP me-central1):

- 34.18.17.249

The IP addresses added on Montreal, Canada (GCP northamerica-northeast1):

- 34.118.144.31
- 34.118.191.107

The IP addresses added on Milan, Italy (GCP europe-west8):

- 34.154.134.143

The IP addresses added on Paris, France (GCP europe-west9):

- 34.155.102.172

The IP addresses added on Tel Aviv, Israel (GCP me-west1):

- 34.165.5.57

The IP addresses added on Turin, Italy (GCP europe-west12):

- 34.17.27.153

Workaround:None
Mar 04, 2024 - 13:00 UTC

IP replacement in Spokane, WA - IPv6 Mar 4, 2024 14:00-21:00 UTC

The Cloud Agents Operations Team will change the addresses of the Cloud Agent in the continued effort to improve quality of service. The current IP address space won't be used after the maintenance.

When: 2024-03-04 from 14:00 to 21:00 UTC

Impact: No impact is expected

The new IP address space is:

2606:a8c0:0:3::3-22/64
Posted on Feb 26, 2024 - 13:07 UTC

Past Incidents

Feb 29, 2024

Maintenance in Chicago, IL

Completed - The scheduled maintenance has been completed.
Feb 29, 15:31 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 29, 14:00 UTC

Scheduled - Description

Affected Cloud Agents:

Chicago, IL

Chicago, IL (AT&T)

Chicago, IL - IPv6

Chicago, IL - IPv6 (AT&T)

Start time: 2024/02/29 14:00 UTC

End time: 2024/02/29 15:30 UTC

Impact: Possible missing test rounds, destination unreachable, high loss/latency/jitter

Description: Network device maintenance
Workaround: none
Feb 23, 09:30 UTC

Feb 28, 2024

Maintenance in New York, NY

Completed - The scheduled maintenance has been completed.
Feb 28, 02:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 27, 14:00 UTC

Scheduled - Description: Server relocation

Affected Cloud Agents:
New York, NY
New York, NY - IPv6
New York, NY (Cogent)
New York, NY - IPv6 (Cogent)

Start time: 2024/02/27 14:00 UTC
End time: 2024/02/28 02:00 UTC

Impact: Possible missing test rounds, destination unreachable, high loss/latency/jitter

Workaround: none

ThousandEyes support is available to be of assistance 24 hours a day. Please open a chat from within app.thousandeyes.com or from status.thousandeyes.com to speak with a support engineer if you have any questions.
Feb 20, 16:40 UTC

Feb 27, 2024

Planned Maintenance - Seattle, WA (Charter) + Seattle, WA - IPv6 (Charter)

Completed - The scheduled maintenance has been completed.
Feb 27, 14:01 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 27, 08:00 UTC

Scheduled - Description: The Telco is performing planned maintenance in our circuit

Affected Cloud Agents:

Seattle, WA (Charter)
Seattle, WA - IPv6 (Charter)

Start time: 2024/02/27 08:00 UTC
End time: 2024/02/27 14:00 UTC

Impact:

The service will be impacted up to 5 Minutes during the 6 Hours maintenance window.
Possible missing test rounds, destination unreachable, high loss/latency/jitter

ThousandEyes support is available to be of assistance 24 hours a day. Please open a chat from within app.thousandeyes.com or from status.thousandeyes.com to speak with a support engineer if you have any questions.
Feb 15, 16:06 UTC

Planned maintenance for aptproxy.thousandeyes.com and yumproxy.thousandeyes.com

Completed - The scheduled maintenance has been completed.
Feb 27, 05:30 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 27, 04:30 UTC

Scheduled - Maintenance for aptproxy.thousandeyes.com and yumproxy.thousandeyes.com is planned on Tuesday, February 27th 2024, between 04h30 and 05h30 UTC.

During this change, the ThousandEyes Engineering Operations team will perform planned maintenance activity on aptproxy.thousandeyes.com and yumproxy.thousandeyes.com domains. Customers might face intermittent issues while connecting to these aptproxy.thousandeyes.com and yumproxy.thousandeyes.com domains.

Workaround: none
Feb 22, 12:24 UTC

Degraded performance from Endpoint Agents

Resolved - This incident has been resolved.
Feb 27, 00:23 UTC

Monitoring - All Endpoint Agents services have come back to their expected response times.
Feb 27, 00:08 UTC

Update - Dashboard data for Endpoint Agents on the Local Network Sources is still being processed. Data for the all other services is already processed.
Feb 26, 18:43 UTC

Update - A fix has been applied. Services response time gradually going back to their expected response times.
Feb 26, 17:12 UTC

Identified - A release made at 3:00 PM UTC today caused a significant amount of data lag in our processing pipeline. We rolled back the release at 3:43 PM as well as increased our processing capacity temporarily to make up for the accumulated data. We are seeing lag decrease. We're expecting data to catch up in the following minutes.
Feb 26, 16:51 UTC

Investigating - Endpoint Agents are experiencing issues that affect data for data collection. Is expected delay on data posted to the backend. Issue is currently under investigation.
Feb 26, 16:39 UTC

Feb 26, 2024

Name replacement in Dublin, Ireland

Completed - The scheduled maintenance has been completed.
Feb 26, 21:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 26, 13:00 UTC

Scheduled - The Cloud Agents Operations Team will perform agent name changing to clarify the network provider.

When: 2024-02-26 from 13:00 to 21:00 UTC

Impact: No impact is expected

New agent naming:

Dublin, Ireland → Dublin, Ireland (euNetworks)

Dublin, Ireland - IPv6 → Dublin, Ireland - IPv6 (euNetworks)

Workaround: none.
Feb 16, 10:49 UTC

Degraded performance from Cloud agent in Singapore with Singtel service provider

Resolved - The issue has been resolved and after monitoring no further service degradation was detected.
Feb 26, 16:48 UTC

Investigating - The cloud agent in Singapore with Singtel is having reduced performance on DNS service traffic. We are working with the provider to remediate this situation ASAP.
Jan 11, 22:05 UTC

Feb 25, 2024

No incidents reported.

Feb 24, 2024

Maintenance in Ahmedabad, India (Reliance)

Completed - The scheduled maintenance has been completed.
Feb 24, 00:30 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 23, 18:30 UTC

Scheduled - Affected Cloud Agents:
Ahmadabad, India (Reliance)
Ahmadabad, India - IPv6 (Reliance)

Start time: 2024/02/23 18:30 UTC
End time: 2024/02/24 00:30 UTC

Impact: Possible missing test rounds, destination unreachable, high loss/latency/jitter

Workaround: None

ThousandEyes support is available to be of assistance 24 hours a day. Please open a chat from within app.thousandeyes.com or from status.thousandeyes.com to speak with a support engineer if you have any questions.
Feb 23, 16:22 UTC

Maintenance in Mumbai, India (Reliance)

Completed - The scheduled maintenance has been completed.
Feb 24, 00:30 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 23, 18:30 UTC

Scheduled - Affected Cloud Agents:
Mumbai, India (Reliance)
Mumbai, India - IPv6 (Reliance)

Start time: 2024/02/23 18:30 UTC
End time: 2024/02/24 00:30 UTC

Impact: Possible missing test rounds, destination unreachable, high loss/latency/jitter

Workaround: None

ThousandEyes support is available to be of assistance 24 hours a day. Please open a chat from within app.thousandeyes.com or from status.thousandeyes.com to speak with a support engineer if you have any questions.
Feb 23, 16:20 UTC

Feb 23, 2024

Emergency maintenance on Private BGP Monitors

Completed - The scheduled maintenance has been completed.
Feb 23, 02:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 23, 00:00 UTC

Scheduled - Summary
ThousandEyes Engineering Operations team will perform maintenance on all Private BGP Collectors on 2024-02-23 between 0:00 and 02:00 UTC to fix an issue with the BGP Monitor status in the dashboard.

Affected BGP Collectors
All Private BGP Collectors will be restarted on February 23, 2024, between 0:00 and 2:00 UTC.

Anticipated Impact
During the maintenance window, our Engineering Operations team will restart the BGP routing daemon on all Private BGP Collectors. We anticipate up to 5 minutes of downtime during the restart.

This will lead to the unavailability of any existing BGP sessions with these collectors, thereby preventing the capture of Private BGP events (via BGP Monitors). Customers won’t be required to take any actions, as BGP sessions should be restored promptly after the restart.
Feb 22, 19:11 UTC

Feb 22, 2024

Maintenance in Chicago, IL (CenturyLink)

Completed - The scheduled maintenance has been completed.
Feb 22, 12:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 22, 06:00 UTC

Scheduled - Description: Network device maintenance

Affected Cloud Agents:
Chicago, IL (CenturyLink)
Chicago, IL - IPv6 (CenturyLink)

Start time: 2024/02/22 06:00 UTC
End time: 2024/02/22 12:00 UTC

Impact: Possible missing test rounds, destination unreachable, high loss/latency/jitter

Workaround: None

ThousandEyes support is available to be of assistance 24 hours a day. Please open a chat from within app.thousandeyes.com or from status.thousandeyes.com to speak with a support engineer if you have any questions.
Feb 8, 17:30 UTC

Planned Maintenance - Ashburn, VA (Charter) + Ashburn, VA - IPv6 (Charter)

Completed - The scheduled maintenance has been completed.
Feb 22, 11:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 22, 06:01 UTC

Scheduled - Description: The Telco is performing planned maintenance in our circuit

Affected Cloud Agents:

Ashburn, VA (Charter)
Ashburn, VA - IPv6 (Charter)

Start time: 2024/02/22 06:00 UTC
End time: 2024/02/22 11:00 UTC

Impact:
The service will be impacted up to 60 Minutes during the 5 Hours maintenance window
Possible missing test rounds, destination unreachable, high loss/latency/jitter

ThousandEyes support is available to be of assistance 24 hours a day. Please open a chat from within app.thousandeyes.com or from status.thousandeyes.com to speak with a support engineer if you have any questions.
Feb 15, 16:12 UTC

Service Degradation affecting Internet Insight for Network Outages

Resolved - This incident has been resolved.
Feb 22, 02:19 UTC

Monitoring - A fix has been implemented and we are monitoring the results.
Feb 22, 02:07 UTC

Investigating - We are investigating a possible service degradation for Internet Insight affecting Network Outages alerts. Users may experience delays in changes made to Test/Alert Rules configuration taking effect for Network Outages Alerting within the last 10 hours.
This does not affect Alerts for test and rules that are pre-configured Network Outages rules.
Feb 21, 22:37 UTC

Feb 21, 2024

Capacity addition in Minneapolis, MN

Completed - The scheduled maintenance has been completed.
Feb 21, 21:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 21, 13:00 UTC

Scheduled - The Cloud Agents Operations Team will perform a capacity increase on the Cloud Agent. This also includes adding IP addresses to the Cloud Agent.

When: 2024-02-21 from 13:00 to 21:00 UTC

Impact: No impact is expected

The IP address spaces are:

74.113.248.96/27

74.113.250.0/27 (new)

2607:ffd0:702::/48

Workaround: none
Feb 14, 10:04 UTC

Feb 20, 2024

Maintenance in Dublin, Ireland

Completed - The scheduled maintenance has been completed.
Feb 20, 18:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 20, 17:00 UTC

Scheduled - Description: Network device maintenance

Affected Cloud Agents:

- Dublin, Ireland
- Dublin, Ireland - IPv6
- Dublin, Ireland (Eircom)
- Dublin, Ireland - IPv6 (Eircom)

Start time: 2024/02/20 17:00 UTC
End time: 2024/02/20 18:00 UTC

Impact: Possible missing test rounds, destination unreachable, high loss/latency/jitter

Workaround: None
Feb 15, 17:04 UTC

Feb 19, 2024

Delayed planned maintenance on several BGP Collectors

Completed - The scheduled maintenance has been completed.
Feb 19, 20:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 19, 18:00 UTC

Scheduled - ThousandEyes Engineering Operations team will perform maintenance on the following BGP Collectors between 18:00 AM and 20:00 UTC on February 19th, 2024.:

bgpc-11.prd.sfo2.thousandeyes.com (13.56.218.135)

Status Page
Listed BGP Collectors will be restarted between 18:00 AM and 20:00 UTC on February 19th, 2024.

Anticipated Impact
During the maintenance window, our Engineering Operations team will reboot the listed BGP Collector instances to fix a degraded hardware issue. We anticipate a few seconds (and no more than 5 minutes) of disruption while instances shut down and then complete the bootup process.

This will cause any established BGP session with these collectors to become unavailable and we will be unable to capture BGP events while the BGP sessions are unavailable. Customers won’t be required to take any actions as BGP sessions should be restored promptly when the BGP collectors complete the bootup process.

Workaround: none.
Feb 19, 14:30 UTC

Service degradation affecting NAT Traversal services in the US region

Resolved - The issue is resolved after applying the mitigation.
Feb 19, 15:00 UTC

Monitoring - We have implemented mitigating steps, now monitoring the status.
Feb 19, 13:15 UTC

Identified - We are now verifying the cause of the loss of service.
Feb 19, 13:11 UTC

Investigating - We are currently investigating a possible service degradation to Enterprise Agent NAT Traversal services in the US region.
Feb 19, 11:22 UTC

Planned maintenance on several BGP Collectors

Completed - The scheduled maintenance has been completed.
Feb 19, 13:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 19, 11:00 UTC

Scheduled - Summary
ThousandEyes Engineering Operations team will perform maintenance on the following BGP Collectors between 11:00 AM and 13:00 UTC on February 19th, 2024.:

bgpc-11.prd.sfo2.thousandeyes.com (13.56.218.135)

Status Page
Listed BGP Collectors will be restarted between 11:00 AM and 13:00 UTC on February 19th, 2024.

Anticipated Impact
During the maintenance window, our Engineering Operations team will reboot the listed BGP Collector instances to fix a degraded hardware issue. We anticipate a few seconds (and no more than 5 minutes) of disruption while instances shut down and then complete the bootup process.

This will cause any established BGP session with these collectors to become unavailable and we will be unable to capture BGP events while the BGP sessions are unavailable. Customers won’t be required to take any actions as BGP sessions should be restored promptly when the BGP collectors complete the bootup process.

Workaround: none.
Feb 16, 10:47 UTC

Feb 18, 2024

No incidents reported.

Feb 17, 2024

No incidents reported.

Feb 16, 2024

No incidents reported.

Feb 15, 2024

Service degradation affecting ThousandEyes Web Application

Resolved - This incident has been resolved.
Feb 15, 16:59 UTC

Monitoring - We have mitigated the issue and are monitoring.
Feb 15, 14:49 UTC

Identified - The root cause has been identified, we keep working on it.
Feb 15, 14:26 UTC

Investigating - We are currently investigating a possible degradation to the ThousandEyes Web Application in the US1 region. Some of the requests may be failing.

This notification will be updated in the event that platform health has been determined to be degraded.
Feb 15, 13:36 UTC

ThousandEyes Engineering Operations team will perform maintenance on BGP Collectors

Completed - The scheduled maintenance has been completed.
Feb 15, 13:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 15, 11:01 UTC

Scheduled - Summary
ThousandEyes Engineering Operations team will perform maintenance on the following BGP Collectors between 11:00 AM and 13:00 UTC on February 15th, 2024.:

bgpc-7.prd.sfo2.thousandeyes.com (52.9.247.80)
bgpc-1.prd.sfo2.thousandeyes.com (13.56.170.74)
bgpc-23.prd.sfo2.thousandeyes.com (52.9.44.74)
bgpc-11.prd.sfo2.thousandeyes.com (13.56.218.135)

Status Page
Listed BGP Collectors will be restarted between 11:00 AM and 13:00 UTC on February 15th, 2024.

Anticipated Impact
During the maintenance window, our Engineering Operations team will reboot the listed BGP Collector instances to fix a degraded hardware issue. We anticipate a few seconds (and no more than 5 minutes) of disruption while instances shut down and then complete the bootup process.

This will cause any established BGP session with these collectors to become unavailable and we will be unable to capture BGP events while the BGP sessions are unavailable. Customers won’t be required to take any actions as BGP sessions should be restored promptly when the BGP collectors complete the bootup process.
Feb 9, 10:02 UTC

Related