Executive Summary

Summary

On August 7th, 2023, at 4:06 PM CDT, Beyond Identity released a hotfix to address the Windows Desktop Login issue. The build pipeline encountered an issue, and we released a broken build. Once upgraded to the broken build 2.84.1, the Windows Platform Authenticator didn’t launch due to a missing reference. This prevented users from using the product entirely. The initial after-release to production testing detected the issue, and we reverted the release at 5:14 PM CDT. A service bulletin was released on August 8th at 11:44 AM to address the issue.

The impact continued to the users who had received the upgrade dialog but had not reacted to it. If they chose to upgrade even after we had reverted the release, it could still install the broken version. If the user chose to dismiss the dialog, they were not impacted. The broken binaries were removed from storage services on August 8th at 1:06 PM CDT preventing the installation of the broken version if the dialog was still open.

Once the hotfix 2.84.2 was released on August 8th at 10:02 AM CDT, the fix was to upgrade to that version. Before the fixed version was released, the fix was to uninstall and re-install the previous version. The service bulletin document was updated accordingly.

Impacted Customers

This issue impacted all customers using Windows Platform Authenticator (not Windows Desktop Login version) without using version control.

External Links

• Service Bulletin #230 - Windows Release 2.84.1 Issue – Beyond Identity

Incident Details

Leadup

Beyond Identity released a hotfix 2.84.1 to address the Windows Desktop Login enrollment issue, which also required a change in the Windows Platform Authenticator — both WDL and Windows PA releases were broken.

Fault

Windows Platform Authenticator did not launch at all. The users were unable to use the product at all.

Detection

The support team validated the WDL fix as a post-release check and detected that the Platform Authenticator didn’t launch after upgrading.

Root causes

• Build pipeline errored with dependencies.
• Rust dependency generation error caused the build pipeline to pull the wrong library version.
• The manifest and downloads page was updated automatically after the binaries were released as a part of the hotfix release flow.
• When we identified the issue, reverting the release did not include removing the broken binaries.
• A broken build got released.

Mitigation and resolution

• The release was reverted from the release manifest.
• The download page was reverted.
• Hotfix 2.84.2 was released.
• Broken binaries were removed from production.
• A service bulletin was released.

Lessons learned

• The hotfix release process needs to have a stop-gap between publishing binaries and releasing the manifest and download page update. (SRE-1721)
• Reverting the release needs to prevent access to the reverted release-related binaries. (SRE-1720)
• Identified a new test case for the QA team to perform in the pre-release environment. (BIT-1566)
• The statuspage needs to be updated to have more detailed services listed. (SUPO-820)

Timeline

2023-08-07 4:06 PM CDT - Build pipeline completed the release and updated manifest and download page
2023-08-07 4:16 PM CDT - An issue with the build was identified
2023-08-07 4:20 PM CDT - The engineering team started investigating the issue
2023-08-07 4:58 PM CDT - The issue was reproduced, and the build was identified as defective
2023-08-07 5:05 PM CDT - The site reliability engineering team dispatched to revert the release
2023-08-07 5:14 PM CDT - Release manifest reverted
2023-08-07 5:32 PM CDT - Downloads page reverted
2023-08-07 7:27 PM CDT - Root cause identified and fix implemented
2023-08-07 9:47 PM CDT - The QA team started the release testing process
2023-08-08 10:02 AM CDT - Hotfix 2.84.2 was released
2023-08-08 11:44 AM CDT - Service bulletin released

Metrics

Time to Detection - 10 minutes
Time to SME investigation - 4 minutes
Time to Repair - 54 minutes
Time to Recovery - 17h 56 minutes