Accessing Production Data vs Cybersecurity? Why not both?
Many say that you can’t have it both ways. If you want live process data from your plant, you need to compromise on cybersecurity. Or, if you want to keep your production systems as secure as possible, you need to completely lock them down―air-gap them if possible.
But today’s requirements are forcing companies to look deeper. Industrie 4.0 and competitive pressures demand that companies pull real-time or historical data from production and feed it to IT departments at head office or in the cloud. The question has become: How to do that securely?
Weighing the options
There are several options available, each with its own advantages and drawbacks. The first thought that may come to mind is to use a VPN. But, as we shall see, VPNs are not as safe a bet as you might expect. Another option is to combine OPC and MQTT for in-plant and plant-to-cloud security. This approach draws on the strengths of both protocols but falls short when it comes to connecting isolated networks through DMZs.
There is a third option: secure tunnelling. Implemented well, tunnelling real-time and historical production data over TCP ensures minimal impact on existing systems, and facilitates the implementation of security, through SSL encryption and certificates. OPC servers and other data sources on the OT (Operations Technology) side make local connections to a middleware component, while receiving clients on the IT side do the same. The right kind of middleware allows data to flow bi-directionally, in real time, through DMZs and closed firewall ports, supporting network proxies as needed. In many use cases this provides the best of both worlds―remote access to production data, as well as security.
DMZ – essential for OT/IT cybersecurity
Recommended by industry and governmental agencies alike, the most secure way to connect OT and IT, is to segregate networks by using a DMZ (“demilitarized zone”). Last year the NIS 2 Directive from the European Commission mandated higher security for networking data between the production and corporate levels of a company, encouraging the use of DMZs. A memo released from the White House about the same time stated: “It’s critically important that your corporate business functions and manufacturing/production operations are separated, and that you carefully filter and limit Internet access to operational networks, identify links between these networks and develop workarounds or manual controls to ensure ICS networks can be isolated and continue operating if your corporate network is compromised.”
A reference document for both directives, NIST SP-800-82, sums up zero-trust OT network segmentation like this: “The most secure, manageable, and scalable control network and corporate network segregation architectures are typically based on a system with at least three zones, incorporating one or more DMZs.”
These three zones are the control zone (OT), the corporate zone (IT), and the DMZ itself. Using a DMZ ensures that there is no direct link between corporate networks and control networks, and that only known and authenticated actors can enter the system at all. The SP-800-82 document describes the value and use of firewalls to separate these zones, and to ensure that only the correct data passes from one to the other.
Connecting through a DMZ
Implementing a DMZ in an Industrial IoT environment is problematic for both OPC UA and MQTT. Getting data out of a plant through a DMZ typically requires two or more servers, chained together one after the other. The OPC UA protocol is simply too complex to reproduce well in a daisy chain like this. Information will be lost in the first hop. The synchronous multi-hop interactions needed to pass data across a DMZ would be fragile on all but the most reliable networks, and would result in high latencies. And there would be no access to the data at each node in the chain. MQTT can be chained, but it requires each node in the chain to be aware that it is part of the chain, and to be individually configured. The QoS guarantees in MQTT cannot propagate through the chain, making data at the ends of the chain unreliable.
A secure tunnelling implementation, on the other hand, can support daisy-chained servers across a DMZ because it can mirror the full data set at each node. Its middleware components provide access to that data, both for qualified clients as well as the next node in the chain. The tunnelling middleware used should be able to guarantee consistency, so that any client or intermediate point in the chain will be consistent with the original source.
Securing open firewall ports
To provide the highest level of cybersecurity, secure tunnelling keeps all inbound firewall ports closed. This is something that most industrial protocols were not designed to do. For example, OPC DA and OPC UA both use a client/server architecture, in which the client initiates a connection and the server accepts it. The server must listen for the connection on a TCP port, and that port must be open for incoming connections on the server’s firewall, and on any other upstream firewalls between the client and the server. To provide access to the data via OPC means opening at least one port on each of those firewalls, a significant risk.
Any open incoming firewall port constitutes a security exposure. Network attacks are not made on a port, nor are they made on a protocol – they are made on an application. The risk is that the listening application has an exploitable flaw that may or may not be due to the protocol implementation.
For example, an application may perfectly implement the OPC UA protocol, yet be vulnerable to flaws in OpenSSL. The application may have no exploitable flaws in OPC UA or SSL, but still have a buffer overrun in a string processing function executed on incoming data. No application is free of bugs. Every open incoming port is an opportunity for an attacker to probe an application for exploitable flaws and can give instant access to a network if one is discovered.
The best security practice is to not open any incoming ports at all in the secure network’s firewall. With the right design, a tunnelling approach can meet this requirement. If the tunnelling middleware makes an outbound TCP connection from the server side to the client side, then there is no need to open any inbound firewall ports. This eliminates the attack surface altogether.
The MQTT option
Probably the most common approach to making outbound connections while keeping plant firewall ports closed is by using MQTT. Devices and programs connect to a server called a “broker”, and publish their data to it and/or receive data from it. The MQTT broker does not examine the data payload itself, but simply passes it as a message from each publisher to all subscribers.
This “push” architecture avoids the client-server architecture problem of OPC, allowing devices to make outbound connections without opening any firewall ports. And, by using a central broker, it is possible to establish many-to-many connections, allowing multiple publishers to connect to multiple subscribers. MQTT thus solves some IoT communication and security problems.
Despite these architectural advantages, MQTT has drawbacks that must be addressed if it is to be truly useful for OT/IT and Industrial IoT communication scenarios.
- MQTT is a transport protocol. It does not specify a payload format, which poses interoperability problems among applications from different vendors. Sparkplug B has been created to address this issue, and is starting to gain traction among vendors and users.
- An MQTT broker does not preserve time order among data values on different topics. This means that events in the physical system that occur in the order A then B then C could be delivered to an application as C then B then A, or any other ordering, which is an error in many industrial-control use cases.
- MQTT brokers do not have a way to indicate that a data source is disconnected. The consuming application cannot tell the difference between an old value from a sensor that has failed, or a current value that has simply not changed recently. The “last will” mechanism in MQTT designed to deal with this requires unreasonable levels of coupling between the producers and consumers of data, resulting in duplicate configuration and increased integration and maintenance costs.
- MQTT quality of service (QoS) levels are not adequate for use across a DMZ as they only apply to single connections. Since connecting through a DMZ requires multiple hops, the data producer will never know whether an MQTT message has reached a user, regardless of the QoS it selects. What is needed is guaranteed eventual consistency, so that each value transmitted by a producer will either reach the user, or be superseded by a more recent value.
- MQTT brokers do not handle overload situations well. If data is arriving faster than a consumer can process it, it is likely that data consistency and time order between data producer and consumer will be lost.
Summing up, MQTT does keep inbound firewall ports closed, making it useful for some Industrial IoT applications, particularly “first mile” data collection from embedded devices. But its inability to guarantee consistency of data between producer and consumer, particularly through a DMZ, does not make it an ideal candidate for OT/IT connections.
Avoiding VPNs
Using a DMZ and closing all inbound firewall ports is far more secure than using a VPN. In fact a VPN doesn’t actually provide the level of security needed by an industrial system. Instead, VPNs effectively extend the security perimeter beyond the plant network to include the IT network. A security breach on the VPN exposes all systems on both networks to attack.
The drawbacks of using a VPN for the IoT are examined in detail by Clemens Vasters, a Microsoft Developer. In a paper titled Internet of Things: Is VPN a False Friend? Vasters said, “VPN provides a virtualized and private (isolated) network space. The secure tunnels are a mechanism to achieve an appropriately protected path into that space, but the space per-se is not secured, at all.”
There is no valid reason to provide external access to the whole network. By establishing a one-way tunnel connection from specified data sources to a DMZ you can protect those sources, as well as the rest of your system. You get remote access to your data without exposing your internal network.
Securing the tunnel
In addition to using a DMZ and keeping all inbound firewall ports closed, security best practices should be implemented in the tunnel connection itself. For example:
- SSL encryption is essential, preferably with support for the most recent versions, such as TLS 1.2 and TLS 1.3. The system should use and enforce server certificates.
- User authentication is best applied on a per-connection basis. Each client program should be required to transmit a username and password, an access token, or a client certificate to authenticate.
- Any potential connection over plain-text transports should be avoided because unencrypted data, possibly including passwords, could be retrieved with a network packet capture program.
Data access AND cybersecurity
To stay competitive, companies of all sizes are demanding remote access to industrial process data, whether it be real-time data, alarms, or historical logs. But to meet this demand should not force them to compromise on cybersecurity. There’s no need to make that choice. Tunnelling technologies offer a way to connect OT and IT systems securely for one-way or bidirectional data flow in real time. It is possible to achieve a level of security comparable to an in-plant network, and still access the production data you need.
