Industrial IoT that Works

/by
Data Communication for Industrial IoT – Conclusion

The Industrial IoT holds a lot of promise for improving productivity and cutting costs for industrial systems.  Yet the IIoT is different from both the consumer IoT and traditional SCADA systems.  In particular, data communications have unique requirements that you need to keep in mind if you are planning to implement an IIoT project that works well.

Re-Thinking Assumptions

Because industrial data communication was developed independently of the Internet, the merging of these two technologies requires a re-thinking of some basic assumptions.  The most secure and reliable approach is to focus on the data, and to allow access to the data only.  Accessing a device means that the device is open to an attack.  Accessing the network leaves the network exposed, even if you use a VPN.  A more secure-by-design approach allows the data source and the data user to make outbound connections to a public or private cloud service that holds only the data.  This keeps all plant firewall ports closed, and provides a secure spot independent of the plant where the data can be accessed by authorized users.

Data Protocol Problems

A number of data communications protocols have been proposed for the IIoT, each with its advantages and drawbacks.  UDP works for VOIP and streaming media, but it lacks the accuracy and completeness so necessary for good industrial communication.  MQTT offers a publish/subscribe mechanism and many-to-many connectivity, but lacks a standard data protocol and the ability to handle messages intelligently.  OPC UA is a good choice for in-plant connectivity, but suffers from the traditional server/client design that requires an open firewall port to connect from the Internet.  REST over HTTP is popular for general IoT applications, but has issues with bandwidth, latency, scalability, symmetry, and robustness when faced with the high speed and large number of connection requirements of the IIoT.  To implement security Blockchain may sound good in theory, but a closer look shows why it will fail in practice.

A New Approach

Clearly, a new approach specifically designed for IIoT is needed.  This approach should use the robust foundation of TCP, the security of a publish/subscribe model like MQTT, and the in-plant connectivity of OPC UA.  Its bandwidth use, latency, and scalability should far exceed RESTful HTML.  This new approach should support edge processing, and in fact, provide the means for edge processing, cloud processing, and fog processing, as dictated by the circumstances on the ground or in the field.  It should be available as a public cloud, a private cloud, or a hybrid combination of public and private clouds.

Something That Actually Works

Most important, this approach should actually work.  You should be able to install it in greenfield or brownfield projects.  It should connect to existing systems, use open protocols, and provide secure, robust, real-time performance at speeds not much slower than Internet propagation speeds.  And it should be available today, right now.  If you’re interested, give us a call.

Go back to Table of Contents

Hybrid Cloud

/by
Part 15 of Data Communication for Industrial IoT

A hybrid cloud is a combination of in-house and cloud functionality in the same system.  For industrial applications, this usually means the use of a public cloud for remote access, and a locally managed, private “cloud” for interaction among application within the plant.

A public cloud is a service like SkkyHub, Azure or GE Predix that is hosted by a separate company, whose services are available to the general public.  It provides a point of contact for a remote user to access the data collected from the industrial system.  Depending on the cloud service it might also provide redistribution, storage, analysis, alerts and other processing on the data.

I put private “cloud” in quotes because the local server does not need to be a cloud service in the sense of on-demand resource balancing on a distributed collection of virtual machines.  Usually it is an aggregation and distribution point for data from the local network that allows various plant subsystems to communicate with one another.  It may fulfill the same communication role as a cloud server without the implied implementation of a cloud service.

A private cloud is run by the company that uses it, often on company hardware at a company location.  It might be, for example, be run inside the control system network, in a DMZ between different plant locations or between the management offices and the shop floor.  In terms of security and reliability, one key difference is that data communications for the private cloud typically run across the company network, while for the public cloud communications go across the Internet.

A Very Good Approach

There are several reasons why a hybrid cloud might be the best approach to data communications for the Industrial IoT.  First, a good hybrid cloud system can provide separate data paths for in-house applications and publicly available services.  Implemented correctly, this can mean exposing lower-level data to plant engineers and managers, while providing access to higher-level data like accumulated statistics for the IT department, or aggregated supply levels to raw materials vendors.  This kind of data separability, where everyone gets just the data they need, is a hallmark of a secure system.

In a similar way, running separate servers for in-plant and public cloud keeps the critical data paths on the LAN where higher data rates and more bandwidth are typically possible.  It also means that if the Internet connection goes down, the plant can continue to function unimpeded.  To achieve this, the design would need to ensure that none of the primary control necessary to run the plant is in the public cloud.  And if the system is designed correctly, when the network comes back the data should continue to flow to remote users via the public cloud, using the latest values.

Despite these advantages, there are a few practical considerations.  Costs can be somewhat higher, as you are effectively running two systems.  That may not be a huge consideration since you probably need the communication infrastructure within your plant in any case.  The requirement then becomes ensuring that your in-plant communication software can interoperate with your choice of public cloud provider.  A software or hardware protocol gateway may be all you need to act as a private “cloud” server.

Avoid Cloud Dependencies

Some cloud service companies may not encourage a hybrid cloud solution just due to their business model.  For example, consider a typical REST-based cloud service.  These services offer the ability to transmit data via HTTP directly from a sensor or embedded device to the cloud server.  However, if that sensor needs to be incorporated into a larger control system, sending the data to a cloud service and then back to the control system is both impractical and fragile.  Yet the cloud provider will not let you run a copy of their server software in your own network, as that would eliminate their service revenue.

This brings up an important requirement when selecting an IIoT cloud provider – a good IIoT device should be equally comfortable connecting to a local system or a cloud system.  The cloud service provider should formally support local receipt of data using the same protocol they use in the public cloud, through cloud-compatible software or hardware, so that the device can transmit either to the local system or the public cloud with just a configuration change.  Local cloud-compatible servers should support the protocols that the plant uses, like OPC, Modbus, etc.

When a hybrid cloud system is implemented correctly it can eliminate dependencies on the Internet connection, yet still provide all of the remote accessibility benefits of the cloud.  It can make it possible to re-purpose devices between the cloud and the local system trivially, and provide for graceful degradation in the event of a network failure by maintaining local communication and control even though remote visibility has been lost.  When the purpose of the system is process control, you don’t want to lose control of your process.

Continue reading, or go back to Table of Contents

The cloud is good, except when it’s not.

/by
Part 14 of Data Communication for Industrial IoT

Cloud computing can be quite useful in industrial systems to gather data and do supervisory control in some application spaces.  “Big data” services help managers and engineers to locate inefficiencies, coordinate predictive maintenance, and boost productivity.  The cloud model of software as a service (SaaS) offers a convenient way to add new functionality to existing systems, and it shifts costs from capital to operating expenses.

Despite the advantages of cloud systems, system integrators and key decision-makers in industrial facilities are reluctant to try it.   Some of the reasons for this might include:

  • License enforcement — “Will this cloud-based system be used to ensure software license compliance, in the same way my kids need an internet connection to play a single-player computer game?”
  • Vendor lock-in — “If all the processing power of the system is in the cloud service, how can I switch services?”
  • No edge processing — “There are too many cloud services that are basically just Internet-accessible databases. That’s not flexible enough for me.”
  • Security — “Once my data leaves my plant, is it safe from prying eyes?  And if I connect my plant to the cloud, will my plant be open to attack?”
  • Loss of connectivity — “If my Internet connection goes down, will I lose my ability to control my plant?”

So should we avoid cloud services altogether?  No.  They provide capability and efficiency you can’t get any other way.  In addition to data-gathering, cloud services can be used to support remote connectivity over the Internet.

Cloud as Intermediary

If we link an operation center in one city to a production system in another, there must be a network.  If we make a direct connection, then one or the other must accept an inbound connection from the Internet.  Using a cloud system as an intermediary means that neither the operation center nor the production system needs to open its firewall, thereby improving security by moving the point of attack outside either system.

Limited Data Sets

Should IIoT devices send all of their data to the cloud?  No, it’s usually not necessary.  Only the data necessary for remote monitoring and control needs to be accessed.  Device information is not monolithic – you should be able to pick and choose what the cloud has access to.

Support for Local Capability

But what happens when cloud is not available?  What happens if a cloud provider goes out of business (think Google/NEST Revolv)?  The system should degrade in a way that essential functions still remain available. The goal should be to support fundamental local capability, enhanced with cloud services.  We should still be able to use our devices when the Internet is not available.

Like most things in life, the cloud has its strong points and its weaknesses.  The most successful implementations will take full advantage of the strong points, and design around the weaknesses.  For industrial applications, that means keeping remote devices and in-plant systems behind closed firewalls and protecting them from any network slowdowns or outages.  This can be accomplished through edge and fog processing mentioned previously, and/or by implementing a hybrid cloud, which we will discuss next.

Continue reading, or go back to Table of Contents

Cloud, Edge, and Fog Processing

/by
Part 13 of Data Communication for Industrial IoT

With the huge number of devices comprising the IoT, we can imagine a looming problem with processing power.  If a user wants to perform some logic specific to a device, he usually has two choices: perform that logic in a cloud application, or perform that processing directly on the device (edge processing).  There are clearly advantages to both strategies.

Cloud

Moving complex logic to the cloud means that the device can be made with lower power requirements and less expensive parts.  The device just needs to be able to read raw values from transducers or A/D converters and transmit them using a low-level protocol like Modbus.  The cloud application can decode the raw data, apply scaling, deadbands, alerts and closed-loop control.  Since you should have no direct access to the device (see Access the Data, not the Device), you may not be able to change the logic at the device.  Placing processing in the cloud lets you modify your system logic without endangering device security.  In addition, cloud applications let you write logic that uses information from multiple devices, since all of the device information is available at the cloud server.

On the other hand, placing logic in the cloud means that the device is dumb.  If the network connection is lost, the device can do nothing but collect data.  If the logic in the cloud is important to the operation of the device or the system in which it is running then the system is effectively crippled until the connection is re-established (see The Cloud is Good, Except when it Isn’t—coming next).  In many instances, particularly in industrial systems, this is simply not acceptable.

In addition, placing logic in the cloud presents a scaling issue.  If the cloud server must handle a few data change events for a handful of data items, no problem.  If it must handle millions of data changes for hundreds of thousands of data items, which is more in line with industrial processes, then the simple availability of CPU, memory and network bandwidth become an issue.  The devices already have a certain amount of processing capability on them, so why not use it reduce the load at the server?

Edge

It is easy to see how the device can perform computations like engineering unit conversion, simple alarm generation, and even some closed-loop control before ever transmitting its information to the cloud.  With a little more power on the device it could, for example, read an image from a camera, analyze it for a production error and then send only an indication of whether the image contains an error to the cloud.  Instead of transmitting 100 KB of image, it can send a few bytes of result.

In addition to reducing network bandwidth, this approach also improves response time for closed-loop control.  The latency of data arriving at the device from the physical system can be a few milliseconds.  The device can process the data and determine a response within microseconds, resulting in an event-to-response time of just a few milliseconds.  Compare this to the latencies typical of Internet round-trip times, and it’s clear that edge processing for device control can save two orders of magnitude in response time.  The difference between 2 milliseconds and 200 may not always matter, but in industrial applications it is commonly a consideration.

Fog

There is, commonly, a third option between the edge and cloud—fog.  In industrial applications it is rare, and usually undesirable, to have plant-floor equipment connect directly to the cloud.  It is far more likely that devices and equipment connect first to a local aggregator or gateway.  Even in home automation applications the devices usually talk to a gateway rather than directly to a cloud server.  Processing at the gateway or concentrator is called fog processing.

The purpose of the gateway is to collect information from multiple devices using their native protocols and to convert that information to a form suitable for transmission to a cloud service.  In industrial applications these protocols are typically standards like Modbus, Profibus, Ethernet/IP, or OPC.  In home automation you might see something like ECHONET Lite.  The communication from the gateway to the cloud is determined by the cloud service, and may bear no resemblance to the device protocol.  In essence, the gateway acts as a collection point between the devices and the cloud service.

There is no reason why the gateway cannot perform fog processing on the device data before sending it to the cloud.  In fact, since the gateway is likely to have far more computing power than the edge devices, it often makes sense to perform complex tasks there instead of at the device itself.  Latencies are low, since the device and gateway share a local connection, whether it is Ethernet, WiFi, Zigbee or some other short-range network.  Response times for fog processing are higher than for edge processing, but the difference is now a few milliseconds, not tens or hundreds of milliseconds as it would be with cloud processing.

For larger systems, there is even a fourth option—a hybrid cloud—which I will touch on later.

Continue reading, or go back to Table of Contents

What is Edge Processing anyway?

/by
Part 12 of Data Communication for Industrial IoT

Edge processing refers to the execution of aggregation, data manipulation, bandwidth reduction and other logic directly on an IoT sensor or device.  The idea is to put basic computation as close as possible to the physical system, making the IoT device as “smart” as possible.

Is this a way to take advantage of all of the spare computing power in the IoT device?  Partially.  The more work the device can do to prepare the data for the cloud, the less work the cloud needs to do.  The device can convert its information into the natural format for the cloud server, and can implement the proper communication protocols.  There is more, though.

Data Filter

Edge processing means not having to send everything to the cloud.  An IoT device can deal with some activities itself.  It can’t rely on a cloud server to implement a control algorithm that would need to survive an Internet connection failure.  Consequently, it should not need to send to the cloud all of the raw data feeding that algorithm.

Let’s take a slightly contrived example.  Do you need to be able to see the current draw of the compressor in your smart refrigerator on your cell phone?  Probably not.  You might want to know whether the compressor is running constantly – that would likely indicate that you left the door ajar.  But really, you don’t even need to know that.  Your refrigerator should recognize that the compressor is running constantly, and it should decide on its own that the door is ajar.  You only need to know that final piece of information, the door is ajar, which is two steps removed from the raw input that produces it.

Privacy

This has privacy and information security implications.  If you don’t send the information to the Internet, you don’t expose it.  The more processing you can do on the device, the less you need to transmit on the Internet.  That may not be a big distinction for a refrigerator, but it matters a lot when the device is a cell tower, a municipal water pumping station or an industrial process.

Bandwidth

Edge processing also has network bandwidth implications.  If the device can perform some of the heavy lifting before it transmits its information it has the opportunity to reduce the amount of data it produces.  That may be something simple, like applying a deadband to a value coming from an A/D converter, or something complex like performing motion detection on an image.  In the case of the deadband, the device reduces bandwidth simply by not transmitting every little jitter from the A/D converter.  In the case of the motion detection, the device can avoid sending the raw images to the cloud and instead just send an indication of whether motion was detected.  Instead of requiring a broadband connection the device could use a cellular connection and never get close to its monthly data quota.

Data Protocol

There is just one thing to watch for.  In our example of the motion detection, the device probably wants to send one image frame to the cloud when it detects motion.  That cannot be represented as a simple number.  Generally, the protocol being used to talk to the cloud server needs to be rich enough to accept the processed data the device wants to produce.  That counts out most industrial protocols like Modbus, but fits most REST-based protocols as well as the higher level protocols like OPC UA and MQTT.

Continue reading, or go back to Table of Contents

Where does Blockchain fit into the IIoT?

/by
Part 11 of Data Communication for Industrial IoT

Nothing I’ve read suggests that blockchain will replace SSL for IoT security.  Blockchains are “distributed ledgers” that are known to be tamper-proof (though there are ways to tamper with them in actuality if you own enough of the computing power validating the transactions). This design works fine for certain Internet applications like bitcoin, but I don’t see the blockchain fitting well into the IIoT.

Size matters

First of all, since there is no central ledger, all participating devices must contain, or have access to, the entire ledger.  No entry can ever be removed from the ledger.  As the number of devices grows, and the number of transactions it contains grows, the size of the ledger grows geometrically.  The size of the bitcoin blockchain is roughly doubling every year and currently is over 60GB.  For an IoT node to fully trust the blockchain it would need a geometrically growing amount of storage.  That’s obviously not possible.

So, individual devices can prune the block chain and store only the last few minutes or seconds of it, hoping that nearby peer devices will provide independent confirmation that their little piece of the blockchain is cryptographically secure.  That produces a possible line of attack on the device, where nearby devices could lie, and produce a satisfactory probability of truth in the “mind” of the target device.

Thus security is based on the availability of massive storage, and attempts to reduce that storage requirement diminish security.  As far as I can tell this is an unsolved problem right now.

Too much connectivity?

The second problem with blockchains is that they assume that every transaction in the system must be transmitted to every participant in the blockchain.  Yes, when somebody’s fridge turns on in Paris, every one of the billions of devices participating in the blockchain must be told.  If they are not, then their local copy of the blockchain is inconsistent and they cannot trust the next transaction, which they might actually be interested in.  As the number of devices and transactions rises, the amount of worldwide network bandwidth required to maintain the integrity of the blockchain grows geometrically.  One article I read says that on a 10Mbit Internet connection the theoretical maximum number of transactions in the entire bitcoin universe that connection could sustain would be 7 transactions per second.  Seven.

The result of these two limitations is that a blockchain probably cannot be used to carry the actual data that the devices produce.  Instead it is more likely to be used as an authentication mechanism.  That is, a device that is legitimately on the blockchain can be verified as being itself based on something that the blockchain knows.  My personal opinion is that it sounds very much like the blockchain would become a distributed certificate authority.  Instead of having the current SSL “chain of trust” of certificates, you would have a “blockchain of trust”.  But since an individual device could not contain the entire blockchain you would still need a server to provide the equivalent of certificate validation, so there’s your point of attack.

There are some examples of IoT devices using blockchains, like a washing machine that buys detergent using bitcoins, that are using misdirection to claim the use of blockchains.  Yes, they are using blockchains in their bitcoin transactions because that’s how bitcoin works, but the maintenance data they produce (the real point of the blockchains-for-IoT conversation) are not being transmitted via blockchain at all.

I have yet to see a practical application of blockchains to IoT data or even to IoT authentication.  The conversation at the moment is in the realm of “it would be nice” but the solutions to the implementation problems are not clear.  Incidentally the same problems exist for bitcoin and there are no clear solutions in that space either.

Continue reading, or go back to Table of Contents