Why millions of Brits' mobile phones were knackered on Thursday: An expired Ericsson software certificate

Ericsson says an expired software certificate caused the outage that left tens of millions in the UK unable to call or text from their mobile phones, nor use 4G connections, on Thursday.

The Swedish equipment maker, which manufactures much of the backend gear in the world's cellular networks, said today the downtime was due to an expired certificate in a version of its management software used by European telcos to provide services to subscribers.

Expired certificate equals non-working software equals non-working mobile service for folks. The solution presumably involves reissuing certs to all affected core network nodes, and given how long the downtime has lasted, this may require manually updating the software node by node.

"During December 6, 2018, Ericsson has identified an issue in certain nodes in the core network resulting in network disturbances for a limited number of customers in multiple countries using two specific software versions of the SGSN–MME (Serving GPRS Support Node – Mobility Management Entity)," the supplier said, rather downplaying tens of millions of screwed-over punters as "a limited number of customers."

Crucially, it added:

While the expired certificate was present in software used by O2 and its parent Telefonica, the outage also rolled downstream to the likes of GiffGaff, Sky Mobile, Lyca, and Tesco Mobile, who rely on O2's network for their services. As a result, some 32 million people were without cellular service, and had to resort to Wi-Fi where possible.

Ericsson said that, with the cause of the issue now isolated, it was working to get all of its customers – the mobile telcos – back online.

"During the course of December 6, most of the affected customers’ network services have been successfully restored," Ericsson explained. "We are working closely with the remaining customers that are still experiencing issues."

While a full report on what exactly went wrong is still being put together, Ericsson CEO Börje Ekholm has already vowed to take the offending software round back behind the barn.

"The faulty software that has caused these issues is being decommissioned and we apologize not only to our customers but also to their customers," Ekholm said. "We work hard to ensure that our customers can limit the impact and restore their services as soon as possible."

The network outage isn't just forcing British mobile users to put down their phones and talk to one another. Transport for London's timetable service was knocked out in the blackout, as were some healthcare providers, for example.

"The [National Health Service] trust I work for has lost connection of all the Apple iPads that are used for patient report forms," a Reg reader told us. "This is extremely worrying seeing that every emergency service will be using a 4G network for the entirety of their critical communications. This outage would've put lives at risk if ESN was live!"

Meanwhile, O2, at least, is still partially knackered in the UK. Having forced everyone to 3G to restore some connectivity, the operator hopes to get its 4G services running again by 3am Friday.

"I want to let our customers know how sorry I am for the impact our network data issue has had on them, and reassure them that our teams, together with Ericsson, are doing everything we can," said Mark Evans, CEO of Telefonica UK.

"We will continue to work with Ericsson, through the night, who have assured us that a full service will be restored for customers by the morning. We fully appreciate it’s been a poor experience and we are really sorry." ®