Final Update: Friday, 14 October 2016 20:06 UTC We’ve confirmed that all systems are back to normal as of 14 October 2016 ~14:50 UTC. Our logs show the incident started on 14 October 2016 ~08:30 UTC and that during the 6 hours that it took to resolve the issue 40% of build request in North central US…
Performance issues with Hosted Build in Visual Studio Team Services – 10/13 – Resolved
Final Update: Thursday, 13 October 2016 23:21 UTC We’ve confirmed that all systems are back to normal as of 13 October 2016 ~21:40 UTC. Our logs show the incident started on 13 October 2016 15:25 UTC and that during the 6 hours 15 minutes that it took to resolve the issue, 29% of build requests experienced longer than usual…
Issues with Release Management under Visual Studio Team Services – 10/11 – Resolved
Final Update: Wednesday, 12 October 2016 13:04 UTC We’ve confirmed that all systems are back to normal as of 12 October 2016 11:00 UTC. Our logs show the incident started on 11 October 2016 15:15 UTC. Customers will no longer experience any issues while working with Releases. We understand that customers rely on VS Team Services as a critical service…
Performance issues with Visual Studio Team Services – 10/12 – Resolved
Final Update: Thursday, 13 October 2016 03:56 UTC We’ve confirmed that all systems are back to normal as of 13 October 2016 02:30 UTC. Our logs show the incident started on 12 October 2016 22:30 UTC and that during the 4 hours that it took to resolve the issue, customers might have faced long queue…
Issues with Hosted Build Services in Visual Studio Team Services – 10/5 – Resolved
Final Update: Tuesday 11th October 2016 5:15PM UTC Root Cause: As part of our overall security regimen in Visual Studio Team Services, we perform a rotation of all of our secrets on a regular basis. During the most recent security exercise, some of our older certificates that were no longer being used was deleted. During…
Reports of multiple email notifications being sent for expired extension trial – 10/8 – Resolved
Final Update: Saturday, 8 October 2016 03:49 UTC We rollout a hotfix and the issue has been mitigated as of 08 Oct 2016 ~03:06 UT. We understand that customers rely on VS Team Services as a critical service and apologize for any impact this incident caused. Sincerely, Arvin Initial Update: Saturday, 8 October 2016 00:18 UTC We are…
Issues with my.visualstudio.com new assignments – 10/7 – Resolved
Final Update: Friday, 7 October 2016 22:05 UTC We’ve confirmed that Subscription ingestion is back to normal 07 October 2016 ~21:45 UTC. This is a re-occurrence of yesterday’s issue. We have applied temporary mitigation and are in the process of testing a broader solution for bulk uploads that we plan to release next week. We understand that customers rely on VS…
Issues with my.visualstudio.com new assignments – 10/6 – Resolved
Final Update: Friday, 7 October 2016 02:32 UTC We’ve confirmed that Subscription ingestion is back to normal as of 06 October 2016 02:31 UTC. Root Cause: The failure was due to upstream service bulk upload to single account. Chance of Reoccurrence: Medium Lessons Learned: We have deployed new code to the subscription Ingestion to handle bulk uploads. Incident Timeline: …
Processing issues with Visual Studio Team Services in South Central US- 10/5 – Mitigated
Final Update: Thursday, 6 October 2016 00:00 UTC We’ve confirmed that all systems are back to normal as of 05 October 2016 23:45 UTC. Our logs show the incident started on 05 October 2016 21:48 UTC and that during the 2 hours that it took to resolve the issue, customers may have noticed issues with Hosted Builds, Queuing…
Performance issues with Visual Studio Team Services – 9/30 – Resolved
Final Update: Friday, 30 September 2016 16:06 UTC We’ve confirmed that all systems are back to normal as of 30 September 2016 15:39 UTC. Our logs show the incident started on 30 September 2016 13:38 UTC. Customers should now be able to use Search functionality and TFS Timetracker feature. We understand that customers rely on…