A Five-Phase Approach to Retroactively Harden Your Existing OT PCS

According to the IBM X-Force Threat Intelligence Index, digital attacks targeting industrial control systems (ICSs) and operational technology (OT) increased by more than 2,000 percent from 2018 to 2020. This huge increase in attacks made manufacturing the second most targeted industry for cyberattacks in 2020 – up from eighth in 2019 and ranking behind only the insurance and finance industry. This same report also found that 35 percent of cyberattacks are leveraging existing vulnerabilities in systems, making scan-and-exploit the number one source of attacks over phishing for the first time.   

As a manufacturer, this report may seem both shocking and frightening, which means it is probably time to ask yourself a few questions. First, how long have your OT PCs been in use on your plant floor? When those PCs were deployed, were these six strategies implemented to protect your assets? And when is the last time your organization evaluated your OT PCs for potential vulnerabilities?  

If the answer to any of these questions is, “I have no idea” or is measured in years, you are not alone. At Applied Control Engineering (ACE), we see that OT PCs are rarely hardened at installation, which means over time initial vulnerabilities will only grow. We also see that many organizations put a lot of effort and money into establishing a strategy for network cybersecurity, especially for the enterprise, yet OT PCs are frequently not part of this plan because they are usually thought of as being air gapped or removed from the network in some way. But this is often not the case.   

Just like even the most well-built castle will start to crack and crumble if not maintained, OT PCs that are not monitored for weaknesses and tested against emerging threats will have vulnerabilities that could be exploited by attackers. To avoid this, as part of your defense-in-depth cybersecurity strategy, consider implementing the following five-phase approach to retroactively harden your OT PCs.   

Phase 1: Aligning Key Stakeholders and Identifying Requirements  

Like many improvement-focused projects, success starts by holding productive conversations with key stakeholders. When it comes to OT assets, these stakeholders should include members of plant operations, key OT engineering staff, and potentially a representative from IT depending on how your organization’s cybersecurity is maintained. The two main goals of these initial discussions should be to assess your organization’s cyber posture in relation to your OT PCs and to understand the software and services operators require on each PC for successful operations.  

From these discussions, you should be able to remove assumptions about the way your OT PCs operate and eliminate the need for implementing vague or overly restrictive security measures. For example, locking down all network ports on a PC adds a big layer of security, but this approach may impact an operator’s ability to do their job if they can no longer connect to other systems that are crucial to operations.    

Additionally, this group also needs to identify any regulatory requirements or security standards your industry or application should adhere to. This could include industry-wide requirements such as Chemical Facility Anti-Terrorism Standards (CFATS) for the chemical industry or the requirements set forth in the North American Electric Reliability Corporation Critical Infrastructure Protection (NERC CIP) plan. Or this may involve determining standards your organization needs to, or really should consider, complying with such as those set by the National Institute of Standards and Technology 800 series (specifically the 800-82 overlay, which is a guide to ICS security) or the US Department of Defense Security Technical Implementation Guidelines (STIGs). A skilled systems integrator such as ACE can help you review best practices for meeting these types of requirements and standards and create the most appropriate configuration to use on your PCs.   

Once these discussions are complete, this is a good time to perform a risk analysis on your OT PCs to ensure the most important gaps are identified. This can be done internally or by partnering with a third-party expert such as ACE and can use an internal standardized cybersecurity program or an external framework such as the NIST Cybersecurity Framework (CSF).  

Phase 2: Performing a Thorough Investigation 

The goal of the investigation phase is to identify everything that is running on or has access to your OT PCs and to then determine the operational purpose of that software or access. Just like the first phase, this approach requires a lot of discussion and open communication with operators to be successful. This is because it is not always clear what is running on every OT PC and what the function of each service or piece of software truly is. And sometimes, even if the purpose is understood, the software or service may be functioning in a manner that is different than what was previously assumed.   

Because we frequently see a level of ambiguity when it comes to what is running on an organization’s OT PCs, when we start an investigation, we typically evaluate the entire configuration. A well-documented holistic evaluation also helps later phases run more smoothly since we are trying to convert as many “unknown, unknowns” into “knowns” or “known, unknowns.” It probably goes without saying, but there is not a one-size-fits-all approach to this evaluation process as each plant is unique and what is acceptable for one plant is not always acceptable to another. 

During the investigation phase we often reveal hidden dependencies and vulnerabilities due to unnecessary program operations that no one knew about. For example, if you are using PanelView, it may be running a service you are not really using, which means in the next phase we would likely recommend you disable that function since it increases the attack surface of your system.  

Phase 3: Planning, Executing, and Documenting Your Procedures 

After the investigation phase, the next step is to combine what is now known about existing operations with your identified requirements from phase 1 to determine where changes are needed. From here, a prioritized remediation plan that closes the identified discrepancies and gaps needs to be generated and agreed upon by all stakeholders. To assemble this plan, it is best to supplement the cybersecurity knowledge of your IT department by bringing in an organization such as ACE that has expert knowledge of OT components.  

We draw on our many years’ experience working with manufacturers to harden new and existing OT assets to develop, plan, and coordinate the best option for your organization that also minimizes or eliminates downtime when implementing the necessary changes. We also are familiar with industry and vendor best practices to ensure the identified requirements and standards decided on in phase 1 are followed. We help simplify the execution of any required changes by standardizing our recommended upgrades and changes for as much as the site will allow across all assets. These recommendations can be used to develop hardening guidelines to use with future system implementations and test procedures. 

We know the logistics of implementing changes, especially when something appears to be functioning correctly on the surface, is frequently a sticking point for manufacturers since it is unlikely downtime can be scheduled just for OT PC upgrades. We try to design our hardening project plans so that a full outage is not needed for most of the required changes. We do this by using the upgrade procedures outlined by many software vendors or by implementing and testing required changes on a testbed first if one is in place. If an outage is necessary, we recommend coordinating outage times with an existing shutdown for a process such as routine maintenance.  

Phase 4: Test – Performing Post-Implementation Testing 

While easy to overlook, post-implementation testing is a critical phase of this approach that helps ensure changes made to existing OT PCs do not have unknown consequences. While every effort is made in the first three phases to turn all unknowns into knowns or “known unknowns,” there still can be unintended consequences when changes are made in the field. This is because OT PCs in many plants have morphed over time as one-off changes and upgrades were made as necessary. A well-defined testing and acceptance document helps identify any complications caused by field conditions so they can be corrected during implementation.   

It is also important to note that as you perform your testing, it is crucial to have the key group of stakeholders identified in phase 1 involved in this phase as well. These stakeholders can help quickly identify unintended operational impacts since they are most familiar with the functionality of the systems. Then, once one successful round of changes and testing is executed, the execution plan can be adjusted to account for new information exposed during the initial implementation. A project plan that alternates between execution and testing phases limits operational risk and improves the final configuration. 

Phase 5: Continuous Improvement – Plan to Reevaluate Your OT PCs Regularly  

It is not ideal to only think about vulnerabilities in your OT PCs every five to ten years. That is why the documentation that comes out of this approach to retroactively harden your OT PCs should also include a plan for ongoing maintenance. Examples of information that should be documented in your reevaluation plan include software and hardware lifecycle management, procedures for updating and patching software, and processes for evaluating the addition of any new software or services to an OT PC. Having these procedures in place will help eliminate possible vulnerabilities cyber attackers can exploit.   

Remember, even the best built castle needs routine maintenance and upgrades to patch any weaknesses and ensure ultimate protection. Learn more about the ACE approach for bringing the latest cybersecurity practices to your plant floor.