When a worldwide outage halted flights, interrupted healthcare services, and caused chaos in the financial sector on Friday, IT technicians were on the front lines of handling the outpouring of concern, alarm, and even fear from business leaders and the general public alike.
Managed services providers (MSPs) who handle some or all of the technology needs of their clients are no strangers to handling tough situations. So when CrowdStrike pushed an update that ultimately resulted in the blue screen of death for millions of Microsoft devices, these providers were ready to jump into action.
Providers work quickly to assist businesses, bring systems back online
“We had several customers impacted, and basically their entire environments were down due to the outage,” said Carlos Marques, Director of Technology at Rochester, N.Y.-based Mainline Information Systems. “Of course, your first thought when you’re pulled out of bed at three in the morning is to assess what seems to be wrong, and as CrowdStrike issued the notice, we jumped into triage and restored their systems as soon as we could.”
In fact, the prevailing view throughout the channel on Friday and into the weekend seemed to be one of wanting to help those impacted while also looking forward to the next outage incident. Yes, there will be a next one.
“Of course, in the moment the outage becomes clear, the goal is to be reactive to that and work to get everyone back online,” said Terry Richardson, Chief Revenue Officer at Blue Mantis in New Hampshire. “But now, as systems come back online and we mitigate the damage, we need to now proactively reach out our customers and talk about their resiliency to potential future problems.”
Difficulties with vendor partners appear once more
Frustrations between MSPs and the vendors they provision are nothing new in the channel, but major events can exacerbate tensions at a time when urgency and concern are at their highest levels.
When providers have their customers on one end stressed and perhaps even angry about their inability to work due to an incident, they often have vendor partners on the other end unable to provide updates or help as quickly and efficiently as they need.
“The dependence on vendors to fix issues without having the staff or team to support is a challenge for MSPs,” said Phillip Walker, CEO at Network Solutions Provider in California.
“When things like this happen, you kind of are at the mercy of the vendor in terms of your ability to deploy a fix,” said Richardson. “That said, though, I do think some of us as MSPs can be a voice on behalf of all of our clients to escalate concerns and needs up to these large companies.”
Additionally, the CrowdStrike and Microsoft outage has illuminated just how interconnected many solutions are, and how reliant on key vendor companies many organizations are today.
“This will be hard to solve for, but it will question the role of agent-based software like CrowdStrike, and also might encourage some to look at more cloud native solutions,” said Marques. “Ultimately, this is all a question of assessing risk against reward, and each partner and business will have to do that for their own needs.”
“Every business needs to look at their whole environment, including SaaS applications that might run on larger systems,” said Richardson. “You have to think about downstream implications when choosing solutions, and we always say ‘if you can’t see it or you don’t know about it, how can you protect it?’”
Assessing, planning, and updating for the future comes next
While many will claim to be, no product, solution, or company is truly foolproof. And while non-breach related incidents like Friday’s outage are infrequent, they are far from impossible. So, MSPs need to prepare themselves and their clients for a world of possibility. To do that, both parties, the provider and the customer, must come together to address gaps and set priorities.
“This incident underscores the critical need for a multi-layered backup system and failover protocols,” said Walker. “MSPs must be prepared to quickly isolate affected components and seamlessly transition to alternative solutions to minimize client downtime.”
“Business resiliency, including the ability to recover backups, kind of goes in waves in terms of its importance to businesses,” said Richardson. “I think this incident will bring it back to the forefront. Businesses can never afford everything they want, but resiliency needs to be weighed as a risk and reward to the organization.”
“The phrase ‘business continuity’ can no longer be an idea on paper but an executed plan for all businesses in our new threat landscape,” added Walker.
The tools and plans MSPs should focus on
Every partner is unique, and every end user’s needs are different. Still, there are a few tools and processes that all MSPs should reconsider in the wake of events like Friday’s outage, including:
- Continuity systems and backup technology
- Recovery tools and standardized processes to activate those tools
- Open lines of communication between vendor partners, MSP technicians, and business leaders
- Well-documented and tested processes that address all elements of business resiliency, including preparing for cybersecurity incidents
- A plan to assess and improve, if necessary, all of the above once an incident has been resolved
“There’s an understanding that using events like these, or scare tactics generally, aren’t helpful ways to sell to customers or gain their trust,” said Marques. “That said, we are absolutely, once we all calm down a bit from the frustration, going to meet with all of our customers individually to assess whether our processes and continuity plans worked, and how we and they can improve moving forward.”
“We want all of our customers to make informed decisions on their needs, and we have an opportunity to leverage our advisory services to help them make those decisions,” said Richardson. “This is probably not the last time something like this will happen, so businesses need to ask how they can prepare to mitigate future problems before they happen.”
CrowdStrike has said in blog posts as recently as July 24 that they have deployed a fix to the problem and are working with customers to return every endpoint to full functionality. Microsoft has repeatedly said they are working with CrowdStrike to resolve issues stemming from Friday’s outages.
The managed security services market continues to grow and mature in 2024. Learn more about trends within the space and some of its major players.