11/8/2023 0 Comments Duo mfa outage![]() This caused the MFA service to slow down processing of requests, initially impacting the West EU DCs (which services APAC and EMEA traffic). “Unfortunately, this change introduced more latency and a race-condition in the new connection management code, under heavy load. The outage began in the European data centers but then began to spread as customers in other regions came online and Microsoft’s engineers scrambled to reroute traffic to mitigate the problem, without success. The software update that Microsoft rolled out that led to the outage was designed to improve the connections of Microsoft Entra ID servers to caching services to speed up service. “Unfortunately, this change introduced more latency and a race-condition in the new connection management code, under heavy load." The third factor in the outage was related to the MFA server running out of resources to process any more authentication requests, even though the servers showed up as healthy in Microsoft’s monitoring systems. “The second root cause is a race condition in processing responses from the MFA backend server that led to recycles of the MFA frontend server processes which can trigger additional latency and the third root cause on the MFA backend,” Microsoft said. When the traffic volume kicked off the latency problem, that led to the second issue, which involved the way that the Microsoft Entra ID frontend interface was processing responses from the MFA server. This issue began under high load once a certain traffic threshold was reached,” Microsoft said in an explanation of the event. “The first root cause manifested as latency issue in the MFA frontend’s communication to its cache services. 19, they hit a traffic threshold in the Microsoft data centers that triggered the first problem introduced in the software update, which then led to the second issue, and eventually to the third. It turns out that the problem began when customers in Europe began coming online on Mov. ![]() ![]() Microsoft began investigating the issue and eventually was able to mitigate it by the end of the day. On the following Monday morning, Microsoft Entra ID customers who had MFA enabled began noticing timeouts when they tried to authenticate. 13, Microsoft started a code update at some of its Azure data centers, and finished the rollout by the end of the week. Microsoft officials said that there were three root causes for the problem, two of which were part of a software update at some of the company’s data centers, while the third came into play once the other two issues were triggered. The service outage lasted for 16 hours and affected customers of Microsoft Entra ID who were trying to authenticate to Office 365, Dynamics, and other cloud services using MFA. 19 outage on Microsoft’s Azure cloud platform for customers who had multi-factor authentication set up as a requirement. A pair of issues that were introduced as part of a code update in mid-November helped lead to the Nov.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |