Yesterday was not a good day for users of the second biggest mobile network operator in the United Kingdom, Telefonica UK which is better known as O2, as its customers lost access to the 4G network. Nor was it a good day for Telefonica itself which also owns GiffGaff and provides virtual network access to users of Lyca Mobile, Sky Mobile and Tesco Mobile who also found themselves without 4G data. In total, 32 million users found themselves without data or SMS message services from breakfast to bedtime. Users of SoftBank mobile services in Japan also found themselves caught up in the communications crisis. Now Ericsson, the Swedish cellular network infrastructure giant which provides the common link in all of this, has revealed the reason for the data blackout. It was, frankly speaking, a rather ridiculous one.
Usually, network problems that cause access to data or SMS are solved pretty darn quickly. Sure, customers are inconvenienced for a short period but things are sorted within the hour. Not yesterday. The 4G network effectively went into meltdown all day, with customers still being inconvenienced this morning as I write this. So what, exactly, was the ridiculous reason?
While it was apparent pretty much from the get-go that the problem was not directly an O2 one. The company issued a service status update that said, “one of our third party suppliers has identified a global software issue in their system.” That supplier turned out to be Ericsson which issued a statement from Marielle Lindgren, CEO for Ericsson UK & Ireland, confirming as much: “The cause of today’s network issue is in certain nodes in the core network resulting in network disturbances for a limited number of customers across the world, including in the UK. We have been working hard on resolving the UK data issue since early this morning. The faulty software that has caused these issues is being decommissioned.”
Now it has been revealed precisely what that ‘software issue’ was. Ericsson has confirmed that “an initial root cause analysis indicates that the main issue was an expired certificate in the software versions installed with these customers.” Yep, you read that right, an expired certificate. Börje Ekholm, global president and CEO at Ericsson, says that “the faulty software that has caused these issues is being decommissioned” and issued an apology to “not only to our customers but also to their customers.”
I am both shocked and at the same time not at all surprised that certificate expiration was behind all of this service disruption. Shocked as I would have expected a company as large as Ericsson to know better and have the relevant failsafe processes in place to prevent such an event. It does, after all, describes itself as “one of the leading providers of Information and Communication Technology to service providers” and approximately 40% of the world’s mobile traffic is carried through its networks. I’m not surprised though, because disruptive certificate expiry is something that those of us who inhabit the cybersecurity world are all too familiar with. You probably are as well; how many times has your web browser warned you that a site or service you were about was insecure and so blocked your connection? That will almost always be courtesy of an expired secure sockets layer (SSL) certificate.
Why I say that it was a ridiculous reason for such a widespread and damaging mobile network outage is that, regardless of whether you are talking about certificates of the SSL or code signing variety, certificate management is hardly rocket science. What happened to enterprise certificate discovery? What happened to certificate management workflows? These are the questions that I’m looking forward to finding out the answers to as the Ericsson internal investigation progresses. They will be of little comfort to the 32 million users, and the businesses that became collateral damage (Uber, Deliveroo and Transport for London to name but three), but they will maybe serve as both a warning and a solutions template to other large enterprises for the future.
Share this post if you enjoyed! 🙂