The CrowdStrike outage on Friday 16th July 2024 is widely accepted as the biggest IT failure in history, making global headlines, impacting millions around the world and wiping more than $20bn off CrowdStrike’s own valuation.
Thinking back to my time at Procter & Gamble (P&G), it’s not hard to imagine what went through the minds of internal teams when they saw this happen. Though the fault did not occur in SAP systems, the SAP community understands all too well what can happen when even the smallest changes are incorrectly deployed. This outage is a stark reminder of the need for a comprehensive change management process.
What happened?
A flaw residing within the sensor configuration update 7.11 in the CrowdStrike Falcon platform had a direct channel through to the Microsoft Windows OS thanks to the Windows kernel process. As a result, the update crashed, and users experienced BSOD (blue screen of death).
It is unusual that companies are given kernel access, however Microsoft was forced to provide this level of open access due to EU regulation – which ultimately rendered Microsoft Defender anti-competitive.
The incident occurred at the OS layer. While rules within the system may have been working as expected, the change had unforeseen impacts. Yes, change and subsequent outages can have consequences, but it’s rare to see a complete breakdown across all operational touchpoints, geographies, customers, etc.
It begs the question: how sure are you about SAP changes and their downstream effect?
3 lessons learned
While this specific incident is not SAP related, the industry would do well to pay close attention and learn from the mistakes made.
Here are three examples that I picked out.
- Don’t be complacent – The impact of a correct workflow and SAP functioning properly can still cause unintended operational consequences.
- Not every change is alike – consider what the appropriate level of testing, approval and validation is dependent upon the risk level and impact of the change.
- More testing is not the answer – blindly executing more testing is not the answer, certainly it may reduce risk, but even with 100% test coverage, you could have risks with performance, security, deployment, etc.
Steps to ensure secure and effective change
With any change, having as much visibility as possible from the outset minimizes the risk of fault and ensures business and operational continuity. Basis Technologies’ two solutions, ActiveDiscover and ActiveControl, support SAP teams at each and every stage.
Plan & Develop
- Ensure approval of changes with full understanding – Change planning, impact, scoping, design
- Work only on approved changes – Integration with Jira/ServiceNow to ensure that change work is triggered only for approved changes
- Ensure quality of change – Automate and enforce code quality analysis and block release of changes which fall short
- Automate change analysis – Understand content of change and highlight critical objects, table indeces, missing dependent objects, …
- Enforce peer review and separation of duties – Ensure more than one person is involved in the approval / deployment of changes
Test & Deploy
- Determine the right level of testing – Understand what to test, who needs to be involved, security, interfaces, predicted differences
- Conduct regression test automation – Trigger tests in external automation tools
- Assess regression environment fit for purpose – System comparison to understand how different Test system is from Production – is regression test representative?
- Leverage an adaptive workflow – Handle changes according to their risk level to provide additional approvals, extra test deployments,…
- Understand and manage relationships – Identify missing dependent objects, overtake, downgrade, orchestrate changes across landscapes, transport sequences, integrations with other pipelines e.g. ADO
- Automate deployment – Avoid human error, ensure changes deployed in right sequence, identify best deployment window
- Highlight outliers – Identify dormant changes, missing deployments (e.g. in global template)
- Enable rapid rollback – Enable automatic backout as an insurance policy
Are you looking to take control of change within your organization to minimize risk of major disruption? Get in touch with our team of SAP experts, or book a demo of ActiveDiscover or ActiveControl today.
