Forging a Path Between the OT/IT Divide

Jason Shepherd (IoT CTO, Dell Technologies): We want to address the OT/IT divide, especially in case any of you in these respective organizations made a resolution this year to work together more in 2019. Along the way, you’ll find some personal anecdotes from each of us. I come from the IT (Information Technology) world but have learned a lot about OT (Operations Technology) over the past few years, meanwhile, Todd started his career in the industrial controls space and picked up a bunch of IT tricks along the way.

Let’s face it, within a manufacturing plant, the OT people who run production typically don’t interact so well with the IT team. The OT group is often turned off by what they see as IT’s rigorous policies, and the IT professionals in charge of the plant infrastructure would rather not have to deal with the OT side of the house and their “Wild West” mentality.

Each organization is motivated by different things, and, even within these key stakeholder groups, there are many sub-groups that have conflicting goals based on how they are measured. For example, in OT, the production supervisor’s goals of uptime and throughput can often be at odds with the goals of those measured on quality and safety. Meanwhile, in the IT world, the data center people care about different things than those that manage networks and end-user devices like PCs and phones.

But when it comes to digitally transforming your business and implementing Industrial IoT (IIoT) projects that promise to unlock enormous efficiency gains and even new business models, all these stakeholders need to work together for optimal success. OT knows their process, and IT knows about security and management at scale, hence why we like to say that IoT starts with OT and scales with IT.

The key to a better collaborative relationship is for each group to understand respective needs and develop an IIoT implementation methodology that takes each side’s needs into account. This can typically boil down to three foundational considerations:

1. Priorities
2. Connectivity
3. Security

1. Priorities

Uptime, keeping the production line moving, is the number one, two, and three priority for the manufacturing professional. If the line stops, it can mean tens of thousands of dollars of lost production per minute, causing people to lose their jobs. OT people tend to have an “If it ain’t broke, don’t fix it” mindset, where downtime is avoided at all costs and having the latest features and patches takes a backseat to having reliable operations.

To that end, manufacturing organizations have turned to specialized, very rugged, extremely reliable computer control systems to run their plants. These systems are called PLCs (Programmable Logic Controllers), and they are used everywhere to orchestrate the operation of the sensors, motors, valves, and actuators within machines and across the factory floor.

Beyond understanding OT’s foundational need for uptime, it’s also important to align on terminology around “real time,” which is one of the more generalized phrases used in IoT conversations. Rapid data processing is important in the IT world, for example, with credit card authorizations, but these transactions don’t typically have major implications if there’s a slight delay or even all-out failure in execution. Meanwhile, communication between PLCs and the devices they control needs to be both extremely fast and deterministic, meaning highly predictable.

Then again, even OT organizations have different time scale needs when it comes to running their operations depending on what they’re doing.

[Jason]: I was once talking with two OT experts – one focused on energy monitoring in buildings and another in the manufacturing space.  After a few minutes of talking past each other about the importance of “real time” communication for IoT solutions, it became apparent to me that we weren’t on the same page. I asked the buildings person what he considered to be “real time”, the answer – 15 minutes. The manufacturing expert’s response – milliseconds.

For this reason, it’s helpful to think in “relevant time”, with a balance made between leveraging embedded, fixed-function PLCs for deterministic process control paired with more IT-centric data acquisition and analytics solutions that are software-defined for flexibility while still being appropriately responsive.

The key to a successful IIoT implementation is for IT to understand OT’s need for reliable, uninterrupted operation and to maintain separation of concerns between the essential control functions served by PLCs and solutions above for data extraction, normalization, and analytics. Think of this as essentially building a virtual data layer above the control network to extract valuable operational intelligence without interrupting the process.

2. Connectivity

Since industrial controls and PLCs are so reliable, in any typical manufacturing plant you will likely find an assortment of controls ranging from state-of-the-art systems to 40-year-old PLCs and legacy equipment running Windows 3.1 or even DOS operating systems.

Todd Edmunds (Director of Industrial IoT Strategy, Dell Technologies): I personally know of customers who have spent $35,000 to purchase an obsolete 30-year-old PLC processor to run a critical piece of equipment that was down.

One of the big challenges in the industrial world is how to get all this legacy equipment to connect to a modern-day information network and get it to give up its valuable operational secrets.  Given that PLCs are built for the sole purpose of reliable control, they aren’t typically set up to share the information they possess about the process with the outside world.

[Todd]: I was once told by a very bright programmer that he was writing a “killer” predictive maintenance application for a packaging machine and wanted me to get him a list of the APIs for the PLC. I had to explain to him that PLCs do not have APIs and most don’t even have ethernet ports!

The challenge is not just the age of the equipment, but the communication methods that are needed to interface with these dinosaurs.

IT vs. OT Network Requirements:

• While most IT systems are designed for the open flow and sharing of information with other devices, control systems in the OT realm are designed to do one thing extremely well and very reliably: control a local system. Sharing information and providing communications is largely an afterthought, where data must be extracted rather than simply published.
• Where IT professionals discuss connection speeds in the range of 100 Megabits, 1 Gigabit, 10 Gigabits, and even 40 Gigabits per second, many plant devices lumber along communicating at 19.2 thousand bits per second, if they even communicate at all!
• IT systems have the luxury of almost everything speaking the same standardized IP-based protocols. While modern PLCs are increasingly leveraging IP-native protocols like OPC-UA, most plant floor communication is still a random mash-up of legacy serial and proprietary Fieldbus protocols including RS232, RS485, Modbus, Profibus, EtherCAT, HART, and many more.
• IP Addresses in a process control network are typically static – manually assigned. Most Ethernet-based control systems have no way of identifying devices by their domain names, they instead rely on hard-coded IP addresses to connect to sensors, actuators, and other control devices.

This is why it’s essential to have a well thought out IIoT infrastructure with scalable edge compute devices used to interface to legacy systems and translate the myriad OT protocols into common data sharing formats and provide valuable context. IT must work with OT to understand the connectivity and latency needs and build a reference architecture on top of open standards that can scale. With these open frameworks combined with device and application management and built-in security capabilities, the job of bringing these valuable legacy assets online is exponentially less challenging.

3. Security

Security is an especially important consideration in the OT world because a successful hack often immediately impacts production, or worse, life and limb. Hence why security by obscurity (i.e. isolating operations from broader networks, including the internet) has historically been the norm for OT. The degree of this risk depends on the nature of the Applications. For example, connecting a control system in a nuclear plant to the cloud is quite different than a system that simply monitors energy use within a building.

Enter a foundational catch-22 for Industrial IoT: to drive new outcomes, OT systems need to be connected, and to keep connected devices secure, they need to be updated, and updates that aren’t properly scheduled can cause unplanned downtime, which is a big no-no to the OT person. As such, a key challenge is to balance IT's need for enabling security and manageability at scale without impacting production uptime.

Since security can be a real challenge to integrate, implementing security is usually left as an afterthought. This means that secure operations must be built into everything at the beginning, as it will be assumed to be available.

Most OT professionals are extremely wary of IT implementing their typical security policies in their plant floor production kingdom, as they have experienced typical IT procedures, like patching and upgrades, to cause systems to quit functioning.

[Todd]: As a production manager at a snack manufacturer once said to me, “Our IT department has caused more downtime in our plant than all malware combined.” If an email gets delayed for a few minutes due to a virus scan or security concern, usually nobody notices. But if the signal to stop the flow of creme filling is delayed by even one second, then Twinkiemageddon is the result!

It's critical that security procedures in manufacturing take a “keep operating with a detected threat and manage” point of view rather than the normal “shut down access to detected threat” response from the IT world.  Beyond mitigating specific threats, updates need to be able to be scheduled during downtime versus being pushed. This is where robust, context-aware manageability tools are critical.

In addition, the tools used to implement, manage, and report on secure operations must be as user-friendly as possible. Graphical user interfaces are mandatory as are plain-language instructions. IT Security professionals like to use SSH and command line tools and enjoy the power that commands typed into Bash shell gives them.

[Todd]: To an OT person, ‘Bash” is something you do with a hammer to a malfunctioning security appliance and then tell the witnesses to “SSHhhh”.

Getting together

We often come across OT organizations doing shadow IT for IoT projects, avoiding working with IT by bypassing their networks altogether.

[Jason]: We were working with one large company whose OT department decided to go around their IT organization on a bid for a new Industrial IoT solution. Meanwhile, at the same organization, the IT resources were telling me that they wanted to figure out how to do more with OT.  My suggestion: “Talk to one another!” It was like a bad episode of the TV show Three’s Company in which it’s always a big misunderstanding.

That said, we’re seeing more and more collaboration between OT and IT organizations, including companies adding executive roles to drive programmatic collaboration across these groups and incentivizing both sides to work together for the benefit of the overall business.

Conclusion

Forging the path through the IT/OT divide can seem challenging, but it can be successfully navigated with the right amount of collaboration, communication, caution and commitment.

1. Set up regular collaborative sessions and really understand respective priorities.
2. Communicate abundantly and regularly.
3. Be careful to protect the production process at all costs, while realizing that security is an absolute necessity.
4. Get commitment from all stakeholders early on – from OT, IT, and especially executives.

Follow these recommendations, and you will soon be on the road to real, meaningful digital transformation and Industrial IoT efficiency with the riches it promises!

Co-authored by Jason Shepherd, IoT CTO, Dell Technologies and Todd Edmunds, Director of Industrial IoT Strategy, Dell Technologies