Top Performance for Industrial IoT

T he Industrial IoT is different from the regular IoT. Mission-critical industrial systems are not like consumer or business IT applications. Performance is crucial. Most IT systems are built around a relational database, a repository of data that clients can add to or access, where a response time of a second or two is acceptable. IT data is typically sent across a network via HTML or XML, which adds complexity to the raw data, and consumes bandwidth. Although fine for office or home use, these technologies are not sufficient for the Industrial IoT.

In a typical industrial system, the data flows in real time. It moves from a sensor, device, or process through the system, often combining with other data along the way, and may end up in an operator’s control panel, another machine or device, or special-purpose data historian. As plant or field conditions change, the data arrives in real time, and the system or operator must react. A robotic arm or other device can send hundreds of data changes per second. Tiny, millisecond fluctuations in the data set can have significant effects or trigger alarms, and often each minute detail needs to be accessed in a trend chart or historical database.

Achieving this kind of performance on the Industrial IoT demands an exceptional approach to data communication.

  • A real-time, in-memory database keeps the data moving. The data needs to flow quickly and effortlessly through the system, and an in-memory database is needed to support these rapid value changes. A relational database, the familiar workhorse of the IT world, is not built for this specialized task. It takes too long to write records, process queries, and retrieve information. Thus, an in-memory, flat-file database, is a good choice, allowing for higher data throughput.
  • High-speed data integration connects any data source with any user. A key task of the in-memory database is to integrate all sources of incoming data. If all communication is data-centric (see below), then every data source can be pooled together into a single, universal data set. This design keeps the data handling as simple as possible, allowing any authorized user to connect to any specified combination of data inputs in real time.
  • Publish/subscribe beats polling. In a publish/subscribe, event-driven model, a user makes a one-time request to connect to a data source, then gets updates whenever they occur. By contrast, polling sends regular, timed requests for data. This wastes resources when data changes are infrequent, because multiple requests might return with the same value. At the same time, polling is also inaccurate during rapid change, because a burst of several value changes may occur between polling cycles, and will be completely lost.
  • High-speed “push” data sources are most effective. The data should be pushed out to the system, and then pushed to the user. In addition to being a better security model, this approach is also more efficient. To “pull” data from a source requires polling, which takes longer and uses too much bandwidth, because each data update requires two messages: a request and a reply. Push technology only requires one message, which is more efficient, consumes less bandwidth, and also enables machine-to-machine communication.
  • Data-centric, not web-centric, design gives the best performance on the cloud. Transcoding data at the source takes time, and requires resources on the device which many smaller sensors may not have. By keeping the data in its simplest format, with no HTML or XML code, the lowest possible latency can be achieved. The raw data flows from the source, through the cloud, to the user as quickly as possible. When it arrives it can be converted to other formats, such as HTML, XML, SQL, etc. Different users, such as web browsers, databases, spreadsheets, and machine-to-machine systems can access a single data source at the point of its arrival, reducing the volume of data flow in the system.

Skkynet’s implementation

Following these principles, Skkynet’s SkkyHub™ and DataHub® provide in-plant or IoT networking speeds of just a few milliseconds over network latency, with a throughput of up to 50,000+ data changes per second. Their high level of performance is achieved by combining real-time, in-memory database technology with publish/subscribe, pushed data collection and a data-centric approach to communication.

The “Hub” technology in DataHub and SkkyHub is a real-time, in-memory, flat-file database, used in hundreds of mission-critical systems worldwide for over 15 years. Designed from the ground up for industrial data communications, the DataHub and ETK work by converting all incoming data into a simple, internal, raw-data format. This raw data can be integrated and transmitted at very high speeds.

At the plant level, the DataHub collects, integrates and redistributes process data in real time. Selected sets of data can be passed seamlessly to the IoT simply by connecting the DataHub or ETK to SkkyHub. At the cloud level, SkkyHub provides the same real-time data collection, integration, and distribution. IoT performance now approaches the actual network propagation speeds of the Internet, with virtually no added latency.

Quite honestly, we shouldn’t expect the typical IoT platform to provide this level of performance. Few, if any, were designed for the Industrial IoT. It should come as no surprise that a concept as disruptive as “Industrial Internet of Things” may require new approaches for proper implementation. And in addition to performance, industrial applications have unique security and compatibility requirements. When choosing a solid, robust platform for Industrial IoT, these are all critical factors to consider.

Tunnelling OPC DA – Know Your Options

Since OPC was introduced over fifteen years ago, it has seen a steady rise in popularity within the process control industry. Using OPC, automation professionals can now select from a wide range of client applications to connect to their PLCs and hardware devices. The freedom to choose the most suitable OPC client application for the job has created an interest in drawing data from more places in the plant. Industry-wide, we are seeing a growing need to connect OPC clients on one computer to OPC servers on other, networked computers. As OPC has grown, so has the need to network it.

The most widely-used OPC protocol for real-time data access is OPC DA.  However, anyone who has attempted to network OPC DA knows that it is challenging, at best. The networking protocol for OPC DA is DCOM, which was not designed for real-time data transfer. DCOM is difficult to configure, responds poorly to network breaks, and has serious security flaws. Using DCOM between different LANs, such as connecting between manufacturing and corporate LANs, is sometimes impossible to configure. Using OPC DA over DCOM also requires more network traffic than some networks can handle because of bandwidth limitations, or due to the high traffic already on the system. To overcome these limitations, there are various tunnelling solutions on the market. This article will look at how tunneling OPC DA solves the issues associated with DCOM, and show you what to look for in a tunnelling product.

Eliminating DCOM

The goal of tunnelling OPC DA is to eliminate DCOM, which is commonly done by replacing the DCOM networking protocol with TCP. Instead of connecting the OPC client to a networked OPC server, the client program connects to a local tunnelling application, which acts as a local OPC server. The tunnelling application accepts requests from the OPC client and converts them to TCP messages, which are then sent across the network to a companion tunnelling application on the OPC server computer. There the request is converted back to OPC DA and is sent to the OPC server application for processing. Any response from the server is sent back across the tunnel to the OPC client application in the same manner.

OPC Tunnelling

This is how most tunnellers for OPC DA work, in principle. A closer look will show us that although all of them eliminate DCOM, there are some fundamentally different approaches to tunnelling architecture that lead to distinctly different results in practice. As you review tunnelling solutions, here are four things to look out for:

  1. Does the tunnelling product extend OPC transactions across the network, or does it keep all OPC transactions local?
  2. What happens to the OPC client and server during a network break?
  3. How does the tunnel support multiple client-server connections?
  4. Does the tunnelling product provide security, including data encryption, user authentication, and authorization?

1. Extended or Local OPC Transactions?

There are two basic types of tunnelling products on the market today, each with a different approach to the problem. The first approach extends the OPC transaction across the network link, while the second approach keeps all OPC transactions local to the sending or receiving computer.

OPC Tunnelling Comparison

Extending the OPC transaction across the network means that a typical OPC client request is passed across the network to the OPC server, and the server’s response is then passed all the way back to the client. Unfortunately, this approach preserves the synchronous nature of DCOM over the link, with all of its negative effects. It exposes every OPC client-server transaction to network issues like timeouts, delays, and blocking behavior. Link monitoring can reduce these effects, but it doesn’t eliminate them, as we shall see below.

On the other hand, the local OPC transaction approach limits the client and server OPC transactions to their respective local machines. For example, when the tunnelling program receives an OPC client request, it responds immediately to the OPC client with data from a locally cached copy. At the other end, the same thing happens. The tunnelling program’s job is then to maintain the two copies of the data (client side and server side) in constant synchronization. This can be done very efficiently without interfering with the function of the client and server. The result is that the data crosses the network as little as possible, and both OPC server and OPC client are protected from all network irregularities.

2. Handling Network Issues

There is a huge variety of network speeds and capabilities, ranging from robust LANs, to WANs running over T1 lines on multi-node internets, and on down to low-throughput satellite connections. The best tunnelling products give the best possible performance over any given kind of network.

To protect against network irregularities and breaks, any good tunnelling application will offer some kind of link monitoring. Typically this done with a “heartbeat” message, where the two tunnel programs send messages to one another on a timed interval, for example every few seconds. If a reply isn’t received back within a user-specified time, the tunnelling application assumes that the network is down. The OPC client and server may then be informed that the network is broken.

In practice this sounds simple. The problem arises when you have to specify the timeout used to identify a network disconnection. If you set the timeout too long, the client may block for a long time waiting for a reply, only to discover that the network is down. On the other hand, setting the timeout too short will give you false indications of a network failure if for some reason the connection latency exceeds your expectations. The slower the network, the greater the timeout must be.

However, this balancing act is only necessary if the tunnelling product uses the extended OPC approach. A product that offers local OPC transactions still provides link monitoring, but the OPC client and server are decoupled from the network failure detection. Consequently, the timeout can be set appropriately for the network characteristics—from a few hundred milliseconds for highly robust networks to many seconds, even minutes for extremely slow networks—without the risk of blocking the OPC client or server.

How the tunnelling product informs your OPC client of the network break also varies according to the tunnel product design. Products that extend the OPC transactions generally do one of two things:

  1. Synthesize an OPC server shutdown. The OPC client receives a shutdown message that appears to be coming from the server. Unaware of the network failure, the client instead operates under the assumption that the OPC server itself has stopped functioning.
  2. Tell the client nothing, and generate a COM failure the next time the client initiates a transaction. This has two drawbacks. First the client must be able to deal with COM failures, the most likely event to crash a client. Worse yet, since OPC clients often operate in a “wait” state without initiating transactions, the client may think the last data values are valid and up-to-date, never realizing that there is any problem.

Products that provide local OPC transactions offer a third option:

  1. Maintain the COM connection throughout the network failure, and alter the quality of the data items to “Not Connected” or something similar. This approach keeps the OPC connection open in a simple and robust way, and the client doesn’t have to handle COM disconnects.

3. Support for Multiple Connections

Every tunnelling connection has an associated cost in network load. Tunnelling products that extend OPC transactions across the network may allow many clients to connect through the same tunnel, but each client sends and receives data independently. For each connected client the network bandwidth usage increases. Tunnelling products that satisfy OPC transactions locally can handle any number of clients and servers on either end of the tunnel, and the data flows across the network only once. Consequently, adding clients to the system will not add load to the network. In a resource-constrained system, this can be a crucial factor in the success of the control application.

If you are considering multiple tunnelling connections, be sure to test for cross-coupling between clients. Does a time-intensive request from a slow client block other requests from being handled? Some tunnelling applications serialize access to the OPC server when multiple clients are connected, handling the requests one by one. This may simplify the tunnel vendor’s code, but it can produce unacceptable application behavior. If one client makes a time-consuming request via the tunnel, then other clients must line up and wait until that request completes before their own requests will be serviced. All clients block for the duration of the longest request by any client, reducing system performance and increasing latency dramatically.

On the other hand, if the tunnel satisfies OPC requests locally, this situation simply does not happen. The OPC transactions do not cross the network, so they are not subject to network effects nor to serialization across the tunnel.

4. What About Security?

Whenever you get involved in networking plant data, security is a key concern. In fact, security is a primary reason for choosing tunnelling over DCOM. DCOM was never intended for use over a wide area network, so its security model is primarily designed to be easily configured only on a centrally administered LAN. Even making DCOM security work between two different segments of the same LAN can be extremely difficult. One approach to DCOM security is to firewall the whole system, so that nothing gets in or out, then relax the security settings on the computers inside the firewall. This is perhaps the best solution on a trusted network, but it is not always an option. Sometimes you have to transmit data out through the firewall to send your data across a WAN or even the Internet. In those cases, you are going to want a secure connection. Relaxed DCOM settings are simply not acceptable.

Most experts agree that there are three aspects to network security:

  • Data encryption is necessary to prevent anyone who is sniffing around on the network from reading your raw data.
  • User authentication validates each connecting user, based on their user name and password, or some other shared secret such as a private/public key pair.
  • Authorization establishes permissions for each of those authenticated users, and gives access to the appropriate functionality.

There are several options open to tunneling vendors to provide these three types of security. Some choose to develop their own security solution from the ground up. Others use standard products or protocols that many users are familiar with. These include:

SSL (Secure Socket Layer) – Provides data encryption only, but is very convenient for the user. Typically, you just check a box in the product to activate SSL data encryption. The tunneling product must provide user authentication and authorization separately.

VPN (Virtual Private Network) – Provides both encryption and authentication. VPN does not come as part of the product, per se, but instead is implemented by the operating system. The tunneling product then runs over the VPN, but still needs to handle authorization itself.

SSH (Secure Shell) Tunneling – Provides encryption and authentication to a TCP connection. This protocol is more widely used in UNIX and Linux applications, but can be effective in MS-Windows. SSH Tunnelling can be thought of as a kind of point-to-point VPN.

As none of these standard protocols covers all the three areas, you should ensure that the tunnelling product you chose fills in the missing pieces. For example, don’t overlook authorization. The last thing you need is for some enterprising young apprentice or intern to inadvertently link in to your live, production system and start tweaking data items.

How Can You Know? Test!

The concept of tunnelling OPC DA is still new to many of us. Vendors of tunnelling products for OPC DA spend a good deal of time and energy just getting the basic point across: eliminate the hassles of DCOM by using TCP across the network. Less attention has been put on the products themselves, and their design. As we have seen, though, these details can mean all the difference between a robust, secure connection, or something significantly less.

How can you know what you are getting? Gather as much information as you can from the vendor, and then test the system. Download and install a few likely products. (Most offer a time-limited demo.) As much as possible, replicate your intended production system. Put a heavy load on it. Pull out a network cable and see what happens. Connect multiple clients, if that’s what you plan to do. Configure the security. Also consider other factors such as ease of use, OPC compliance, and how the software works with other OPC-related tasks you need to do.

If you are fed up with DCOM, tunnelling OPC DA provides a very good alternative. It is a handy option that any engineer or system integrator should be aware of. At the very least, you should certainly find it an improvement over configuring DCOM. And with the proper tools and approach, you can also make it as robust and secure as your network will possibly allow.

Download White Paper (PDF)

Security for Industrial IoT

T he issue of remote data access to data from an industrial system is not new.  For years plant owners have been creating ways for their managers, operators, maintenance technicians and partners to gain access to the valuable real-time information generated by their plants.  Innovative business practices, such as globalization and just-in-time manufacturing, have driven a need to have low-latency remote access to this data, often through untrusted networks to semi-trusted end users.  For example, a manufacturer may want to share production information with a remote supplier, but not provide login access to the manufacturing system or database.

Several fundamental security problems have arisen from this need for remote real-time access:

Exposure to attack from the Internet.  When a plant allows a user to access the system remotely, it naturally creates an attack surface for malicious actors to attempt to also gain access to the system.

Exposure to attack from the IT network.  If a plant allows a user to access the system remotely, it also needs to expose itself to the network infrastructure of the company’s IT system.  Generally the plant network is a subnet within the larger company network.  Entry into the plant will be via the IT infrastructure.  Attacks from the IT network are less likely, but some kinds of problems in the IT network could disrupt normal network data flow on the plant network.  It is wise to separate the IT and plant networks as much as possible.

Remote access beyond the required data.  Giving a remote user access to a desktop, such as Microsoft RDP, means that a curious or malicious user can try to gain access to programs and data beyond what was intended.  Even if the user is trustworthy, but the user’s system is compromised, a remote access program becomes a point of attack into the plant system.

Exposure of a portion of the plant network.  Some plants have chosen to use VPN connections to limit Internet attacks.  However a VPN effectively puts all participants onto a local sub-network, which gives the participating machines effectively local network access to one another.  Compromising any machine on the network (such as a remote supplier) produces an opportunity for an attacker to hack into the plant network via the VPN.

High cost.  VPNs, RDP entry points, firewalls and routers require ongoing attention and effort from IT personnel.  This cost increases as the number of participants in the system increases.  Plants that do not devote the resources to IT are more likely to implement their remote data access less securely.

How can Skkynet Help?

Skkynet’s unique solution, SkkyHub™, provides a mechanism for dealing with all of the traditional security problems in remote plant data access.

NO Exposure to attack from the Internet.  Users of Skkynet’s SkkyHub install an agent within the plant that collects plant information and pushes it out to Skkynet’s real-time data servers.  Since this connection is outbound, from the plant to the Internet, there is no requirement for the plant to open any inbound TCP ports, and thus the plant never exposes itself to attack from the Internet.

NO Exposure to attack from the IT network.  It is good practice to isolate the plant from the IT network using a router that allows only out-bound connections from the plant into the IT network.  Using the SkkyHub service, the IT network can be treated as untrusted by the plant, and a firewall placed between the two that allows no inbound connections into the plant.  Disruptions on the IT network will not affect data flow within the plant network, though they could affect data flow from the plant to the Skkynet service.  The plant remains secure and functional, even if remote data access is degraded.

We designed a solution to address all traditional security problems in remote plant data access.

NO Remote access beyond the required data.  Using SkkyHub, the plant decides which data to make available remotely.  The plant engineer can choose any subset of the data produced by his plant, and make it available to remote users in data groups.  Each group has its own read/write permissions as well as limits based on the remote user name and the IP address from which the remote user is connecting.  The remote user has no mechanism to extend his access to data beyond what the plant engineer allows him.

NO Exposure of a portion of the plant network.  The SkkyHub service does not create a VPN, or any kind of general-purpose network construct.  It only makes a TCP connection for the transmission of data.  Consequently, no participating machine is ever exposed to any other via a local network or VPN. The data can be routed through network proxies, data proxies and DMZ servers to ensure that the plant network never has a direct connection to the Internet, even for outbound connections.  Participating systems in the Skkynet service never need to share a network.

NO High cost.  SkkyHub eliminates many security hurdles, thereby substantially reducing the IT cost of implementation.  Often, a plant can participate in the Skkynet service without any change to existing IT infrastructure.  The plant has no need to hire extra IT expertise or to install extra networking equipment.  Often the only cost is for configuration of the Skkynet agent at the plant and the Skkynet service itself.

Skkynet’s technology follows good industry practice by using SSL connections for all Internet traffic, and by validating the trust chains of certificates.  This enhances your security for Industrial IoT, and protects you against network snooping and against man-in-the-middle attacks.

What is a Good Approach to Industrial SaaS?

“A chain is only as strong as its weakest link,” goes the old saying. How true that is in industrial control systems. One small glitch on an assembly line can force a shutdown. Lack of a single ingredient in a chemical plant or food processing system might wreak havoc. Factory automation, power generation, resource extraction and processing, transportation and logistics are all supported by chains of mechanisms, hardware, and software, as well as operators and engineers that must each carry out their mission to produce the expected output, product, or service.

From time to time, new technologies come along that provide cost savings, better performance, and ease of use—new links for the chain. Electrical relays, pneumatic controls, DCSs, PLCs, RTUs, SCADA, plant networks, and fieldbus were all at one time proposals on the drawing board. Each had its evangelists and skeptics, and each was thoroughly tested. Once its value was proven, each one has become a strong link in the automation chain.

One of the latest technologies to be proposed is software as a service (SaaS). SaaS provides access to hosted software over a network, typically the Internet, and is closely related to the concepts of smart factories, cloud computing, industrial Internet, machine-to-machine (M2M), and the Internet of Things (IoT). Adding SaaS to an industrial process control system means adding data collection, integration, and/or distribution capabilities beyond the limits of most existing in-house systems.

SaaS can open wider access to plant data in real time, which can support real-time monitoring of processes, supply the big data needed to drive predictive maintenance programs, provide the ability to outsource customer care facilities, deliver real-time KPIs, and otherwise leverage the value of new or existing SCADA investments. Implemented well, software as a service should also provide a significant cost savings over the traditional avenue of software ownership.

To be truly useful, though, industrial software as a service should be secure, quick, and robust, as well as adaptable and convenient to use.

Secure

Industrial systems require the highest possible level of security. “IT security was the most oft-cited obstacle to setting up smart factories,” according to Lopez Research in their January 2014 article Building Smarter Manufacturing With The Internet of Things (IoT). A comprehensive report from GE titled Industrial Internet: Pushing the Boundaries of Minds and Machines states, “The Industrial Internet will require putting in place a set of key enablers and catalysts,” including, “a robust cyber security system and approaches to manage vulnerabilities and protect sensitive information and intellectual property.” Achieving this level of security requires a comprehensive approach, including secure servers, authorization and authentication of users, encrypted data transport mechanisms, and keeping all firewall ports closed at the plant level.

Quick and Robust

Industrial software as a service should provide as close to real-time performance as the network or Internet infrastructure will support. This means that data updates should be in milliseconds, not seconds or minutes. It should be able to handle thousands of data changes per second, and support redundant connections with hot swap over capability.

Adaptable

Industrial systems are diverse, built from a wide range of equipment and controls, using various data protocols, and come in all sizes. Industrial SaaS should be able to connect seamlessly to any new or installed system at any number of locations with no changes to hardware or software. It should use open data protocols and APIs. Ideally it should work with any size of system, from complete DCS and SCADA systems down to a single machine or embedded device. Running as a cloud-based service, it should also readily scale up or down depending on user needs.

Convenient

To gain acceptance in the market, industrial SaaS should be convenient to use. It should be easy to demo, sign up for a service plan, configure connections, and monitor usage and costs. It should offer off-the-shelf tools to get your data to and from the cloud with no programming, provide the ability to easily integrate data from multiple sources, and include options like data storage and HMI displays–all without disrupting the industrial process in any way.

Redesigning for Security

Among these requirements, the most challenging is security. Without the ability to fully protect mission-critical processes and their data, industrial SaaS is simply a non-starter. And yet, a fundamental characteristic of virtually all industrial systems presents a significant security risk for any cloud-based system: a firewall port must be kept open.

The current approach to this problem is to implement VPN (virtual private network) technology. A VPN creates a secure space on a network that is isolated from all other traffic. However, this is not an ideal solution because a VPN allows every connected device and user full access to every other device and user. For a single control room, this may not seem to be too much of an issue. But in the world of cloud computing, operators and field engineers will expect to have access to data on tablets and cell phones, which can easily fall into the wrong hands.

Ironically, using a VPN might even turn a plus into a minus. A strong selling point of SaaS is its potential to act as a platform for sharing limited data sets with authorized suppliers, customers, and other parties. Few IT departments would be willing to hand over the keys to the store by providing these players with access to the corporate VPN.

A better approach is needed. Although VPN might be useful under certain circumstances, it doesn’t address the fundamental design issue of virtually all industrial data communication, which is the client-to-server architecture. To get data out of an industrial system, a client needs to request it from a server. So, if any kind of cloud service needs access to process data from a server located behind a plant firewall, a port on that firewall must be kept open at all times. And open firewall ports are inherent security risks.

What is required is a new design. SaaS transmits data over the Internet, and there is a TCP protocol that supports a different connection model: WebSocket. With the right kind of engineering this protocol can be applied to industrial data communications in a way that allows only out-bound connections from the plant to the cloud. No in-bound connections are necessary; no plant firewall ports need to be left open. Once the connection is established, the data can flow in both directions. Or you can choose to make all or some of your data read-only, preventing any write back from the cloud. Whichever approach you take, the data flows through a closed firewall, making your plant effectively invisible to Internet attacks.

In addition to protecting the plant, with this design no primary or mission-critical control need be performed by the service. All local control can remain untouched. The system manager has complete flexibility over what data gets passed to the service, and the connection can be configured as read-only if desired.

Real-Time Data

Next to security, a good industrial SaaS solution should perform well. When you mention anything related to cloud computing, most people conjure up an image of a giant database sitting up in the air somewhere in which you store data, and pull it out when you need it, like Gmail or Dropbox. Although that model works fine for storing static data, industrial systems function in real time. The true value of industrial SaaS should be realized through real-time performance, which requires a fundamentally different architecture on the cloud.

One good approach is for the service provider to host a real-time, memory-resident database that can receive and retransmit data at speeds of tens of thousands of data changes per second. Achieving these speeds is possible through a publish/subscribe method of data delivery, an event-driven model in which a client registers for data changes one time and then receives subsequent updates immediately after they occur. This kind of low-latency system adds almost nothing to the overall data transmission time, effectively keeping throughput speeds to just a few milliseconds more than network propagation time.

To further speed up throughput, all data should be handled in the simplest possible format, by taking a data-centric approach. This kind of system is able to work with all kinds of data sources and users, such as control systems, OPC servers, databases, spreadsheets, web pages, and embedded devices. When a data source connects, its data gets stripped of all unnecessary formatting (XML, HTML, OPC, SQL, etc.) and is added to a universal data set comprising data from all connected sources. Updates to any point in this data set are immediately passed to any client registered for them. At the receiving end the data can be transformed back into its original format, or into whatever other format the client might need.

Making it Work

Anyone who has spent any time in industrial automation soon discovers that every system is unique. Different industries, plants, and project requirements demand a wide range of tools, machines, and other devices which are provided by hundreds of independent suppliers, and installed by a multitude of diverse system integrators and plant engineers worldwide.

Good industrial SaaS should fit as many of these different types of systems, protocols, and brands of equipment as possible. It should use open, standard protocols like OPC, TCP, and ODBC. If it is completely vendor agnostic, it is in the best position to leverage investments in existing equipment or enhance new installations with real-time data connectivity. Ideally it should be capable of being added to a SCADA system, function as an HMI for an individual machine, or access RTUs and even individual embedded devices.

As a cloud-based system, we would expect the service to be able to scale up or down to meet the needs of its users. This means the ability to handle bursts of high-speed activity in the data flow at any given moment, as well as the capacity for quick reconfiguration to support expansion requirements of a growing system. Users should be able to add data points to a particular device, or bring on new devices, new SCADA systems, even new locations and installations through an easy-to-use interface.

Finally, data from the service should be readily available to key decision-makers in the way they can best use it, be it an operator using an HMI for monitoring and supervisory control, a field engineer picking up the latest figures from the home plant, an analyst running real-time scenarios from facilities spread across three continents, a just-in-time supplier whose system is keyed to current production levels, or a plant manager responsible for production at a group of isolated facilities. Good industrial software as a service should be a solid link in the chain, reducing costs, meeting the needs of all players, and doing it securely, quickly, and conveniently.

Can I Store and Forward OPC Data?

Every once in a while we get asked: “Can I store and forward my OPC data?” It seems like a reasonable question. I’m connecting an OPC server and client across a network, possibly with OPC UA, or more likely OPC DA—using DCOM or an OPC tunnelling product. In any case, should there be a network problem and the connection gets broken, I don’t want to lose any of my data. Why not store it, and forward it later when the connection is back up? A little understanding of the purpose of OPC, and how it works, though, should make it clear that this impossible.  The answer to this question is effectively “No–not real-time data, anyway.”

Let’s look at the OPC DA scenario first, since this is where the question comes up most often. Put simply, you cannot store and forward OPC DA data. The OPC DA protocol is for live, real-time data. OPC DA clients need to know what is happening right now in the system. An OPC DA server does not maintain a history of past values, and an OPC DA client only works with the most recent value of an OPC tag.

Despite this reality, suppose some bright spark decided he wanted to go ahead and add store-and-forward capability to OPC DA. When the connection to the client breaks, somehow the OPC server would need to start storing values―and this raises questions. How would the client identify which items in the server need to replay past values? How would the server know how long to store the data? What should the server do if this limit is exceeded? There is no facility in OPC for the client to specify any of this information, and it could not be robustly specified at the server. Not all clients would have the same requirements.

Then, when the connection gets re-established, the OPC server would start sending those values to the OPC client. Here it runs into more trouble. How would the server know that this was the same client reconnecting? How long has the client been disconnected? Has the client configuration changed since the last connection and therefore the previous store-and-forward configuration is out of date? How quickly should the values get sent? Should the OPC server replicate the time gaps between value changes? Then it would never catch up to the present. Or should it send all the stored values in a sudden burst of changes?

But the biggest problem is that old values are, by definition, wrong – only the most recent value is correct. In a store-and-forward scenario the client would experience a series of changes that could manifest as wrong indications in an HMI, or worse, wrong control signals to a physical system. Would you really want a piece of equipment turning on and off repeatedly as the data history catches up on the client? That could be inefficient, damaging or dangerous. In practice, most OPC clients can only hold one value for any given tag, and only now matters. The client doesn’t need and can’t use a replay of historical data, it needs to know what’s happening this instant.

It is a fact of life that network connections drop occasionally. When this happens, the client should present the data with a Bad or Not Connected quality, or show a Last Known Value warning until the connection is restored. Once the connection is back the client should receive the current data values. Whatever happened in the meantime is unimportant or downright misleading.

The idea of store-and-forward makes perfect sense, though, for data logging, archiving, and historical data. This is where it is possible to maintain a complete record of everything that has taken place on the OPC server. Although OPC DA itself does not offer this capability, a user can add an OPC HDA server, or connect a database to the OPC DA server. The responsibility for store-and-forward is now in the hands of the historian, where it belongs.

Turning now to OPC UA, from a fundamental design perspective the store–and-forward story is actually pretty much the same as it was for OPC DA. Nothing has really changed for an OPC UA client – it still expects to receive the latest data possible. The only slight difference is that the OPC UA architecture integrates historical data access better than OPC Classic does. This makes it easier for OPC UA servers to support historical data, and for OPC UA clients to request historical information. But whether data storage or archiving abilities are built into the server or provided separately, the reality of real-time data communications is that a real-time client cannot handle data that has been stored and forwarded. Nor is that its purpose. There are other tools designed for that.

So, the answer to the question, “Can I store and forward my OPC data?” is two-fold. On the one hand, you can record your OPC data, and then store and forward it as archived data. But you cannot store and forward data in real time.

 

Relational Database or Real-Time Historian for Logging Process Data?

Quite a few years ago, while living in a part of the world where personal computers were a relatively new phenomenon, I happened to be in an office watching a secretary busily typing away at her new PC. She was thrilled to have such powerful tool to use. “Look!” she said excitedly. “Now I can write and correct my work so easily!” I looked at the screen, and had to smile. She was composing an entire business letter within a single cell of an Excel spreadsheet.

What determines the right tool for the job? For that secretary, the right tool was the one that was available and that she knew how to use. What’s the right tool for logging data from a process control application? Sometimes a CSV file is all that is needed. Sometimes Excel will do. More often, though, engineers and system integrators will use either a relational database or a real-time historian to store permanent records of process data.

Relational databases have the advantage of availability and familiarity. SQL Server, MySQL, Oracle, or any other relational database, including Microsoft Access, are usually already installed at the company. They offer a common interface, ODBC, and the IT department is familiar with them. It’s no surprise that relational databases are being used for logging process data, particularly when the requests for data come from management personnel familiar with this kind of database.

But a relational database may not be the ideal choice for all process control applications. Designed for the needs of corporations, businesses, and banks to store transactional data, relational databases are optimized for analyzing complex relationships between data. These databases can afford to focus processing power on these relationships because the data itself gets updated relatively infrequently, usually in terms of hours, days, or months. While analytical power is good for business applications, process control applications typically don’t need it. What they do need is speed.

Blog-HistorianorODBCA real-time historian, on the other hand, is like a flight-recorder for process data. Rather than relational, it is a temporal database, storing its records in a flat file, consisting of simply the name, value, quality, and timestamp for a data point. The historian is designed for speed of storage and retrieval of data, and can typically process millions of transactions per second. This kind of performance is valuable for processes with variables that may change many times per second, and where capturing every event over the course of each eight-hour shift is vital.

Despite the performance advantages of a real-time historian, some companies opt for using relational databases for logging process data. This is completely understandable, since those are the tools that company IT staff and upper management are most familiar with. But there are three important reasons why this approach may not be sufficient:

  1. A real-time historian logs every data change for a point, even when the values change rapidly. Using highly efficient storage algorithms, the complete data set can be stored for long time periods. A relational database, in contrast, typically needs to drop some or most of the data as it is being logged, because it is not optimized for storing high volumes of data. Unfortunately, the data is dropped regardless of its importance. So you might end up logging routine changes and throwing away the unusual events that could lead to alarm conditions. In addition to detecting every change, big or small, the high volume capabilities of a real-time historian are useful for detecting subtle trends that may only appear over months or years.
  2. A strong advantage of a real-time historian is its native ability to process time-based queries. For example, you might need the standard deviation of a point that changes on average 25 times per second, in 10-second windows for the last two minutes. A good historian will provide an easy way to submit such a query, and will return the results quickly, with a minimum drain on system resources. Built-in query functions typically allow you to select any time period, from a few seconds to weeks or more, and retrieve averages, percentages of good and bad quality, time correlations, regressions, standard deviations, and so on. All of this may be possible through SQL queries on a relational database, but through much more programming effort and greater system resource use.
  3. The two above advantages of a real-time historian may perhaps best be appreciated when working with live trend displays. Calculating and displaying a moving line that updates several times per second requires not just the ability to log all the data points in real time, but also to re-emit them quickly and repeatedly for the moving display. And if a user wants to scroll backwards and forwards through the data set as it is being logged, the historian has to be able to manage rapid, continuous queries to the data set. This kind of task is nearly impossible for an off-the-shelf relational database, unless the screen refresh rate is annoyingly slow.

Even with these points in mind, there are many applications for which logging process data in a relational database works just fine. In fact, sometimes logging to a CSV file is sufficient. To be fair, these are really not the same level of technology mis-match as writing a complete business letter in one cell of a spreadsheet. The well-informed system integrator or engineer will understand the values of each approach, look at the needs of the project and resources available, and employ the right tool for the job.