The Oldsmar, Florida, water breach is two months behind us, but the lessons learned will continue to reverberate for thousands of budget-constrained municipal utilities in North America, as well as other regions across the world.
Lesson #1: Technology Budget Constraints
Oldsmar, like many other municipal utilities, occasionally needed remote access to their site, so they chose TeamViewer because it “didn’t cost anything extra.” Reading between the lines, the key point here is that the IT department had already purchased TeamViewer for their needs and had extra licenses that OT could use. The IT department probably had secure infrastructure around TeamViewer, but they could not forklift this infrastructure over to the water treatment plant because it would be too expensive to replicate for a few “critical” HMIs and other systems. TeamViewer in itself is not the issue – the problem is with the complex and expensive proposition of scaling IT cybersecurity architecture to OT.
Lesson #2: Cybersecurity Resource Constraints
Senior plant managers have mechanical and/or electrical engineering backgrounds and are not well versed in IT protocols, 2FA, firewalls, VPNs and Jump Servers, etc. They don’t have time or the expertise to manage IT cybersecurity stacks. If they have to remote into a plant at 2am and check systems, they want something that just works. Some utilities may invest in integrating a cybersecurity tool, but plant managers will not know if everything is properly configured and just want it to work. The need for easy access to the plant could drive behavior away from complex secure remote access through IT infrastructure and over to “give me the free ‘easy’ button.”
Lesson #3: IT and OT Cultural Differences – Confidentiality vs. Availability
A utility’s IT network of consists of billing, accounting and HR systems, which contain PCI and PII data that must be kept confidential. IT operations and cybersecurity personnel need to make sure that access to these systems is limited and controlled through several integrated secure authentication and authorization mechanisms. IT operations is hyper-focused on providing secure access to sensitive and confidential data for its users.
The OT network consists of process and automation controls and distributed control systems for valves, pumps, meters, etc., as well as human machine interface (HMI) computing systems and SCADA applications that interact with these real-time systems. The safety and availability of these real-time systems is paramount.
The very culture of OT operations is keeping systems running. IT is focused on protecting confidential data. These differing priorities mean that cybersecurity in the OT context needs to be built-in with unique features for both senior managers and technicians.
The Final Lesson: IT Remote Access Solutions Can Increase Risks to Public Safety in OT Environments
The nature of OT requires a very secure and simple remote operations platform that doesn’t break the bank. IT/OT converged networks can create complexity where insecure protocols such as RDP can be exposed into the IT network and out to the internet. Critical OT systems that have exposed protocols can be found with tools such as Shodan. Complex IT cybersecurity infrastructure and Security Operations Centers are focused on IT networks and not built to look for issues within OT networks. While larger utilities do implement OT-specific cybersecurity stacks, smaller municipalities cannot usually afford these, as was the case with the breach in Oldsmar.
In addition, there are specific operational needs that require OT-specific secure remote operations platforms. OT-specific user access and operations can reduce risks to public safety by including unique features such as:
- User access screen recording on HMIs and other OT systems – this can help diagnose user errors and help with training junior technicians to mitigate automation and control issues that could lead to disastrous consequences
- Granular role-based access controls such as a Remote Access Manager and File Transfer Manager – these roles can be given to specific individuals for specific tasks, thus limiting access privileges and mitigating risks associated with oversubscribed access to non-IT OT managers
- Live user connection monitoring – which provides senior managers visibility to technician input to walk through processes and provide real-world training
Enterprise IT remote access technologies such as VPNs and Jump Servers, when used with multi-factor authentication, intrusion detection systems and firewalled network segmentation can reduce risks associated with confidential data compromise; however, these integrated enterprise technologies cannot be forklifted and replicated for OT. Often, an OT staff will deploy a subset of these technologies to enable remote access, which then opens up the OT network to compromise. OT has very specific needs to ensure operational availability and public safety. They cannot afford the vulnerabilities associated with incomplete enterprise remote access tools or complex full stacks, which are too expensive to acquire and maintain in resource-limited OT environments.
To learn about XONA’s user access solution built for OT that puts all of these lessons into action, schedule a demo now.