The Challenge

Upgrading the aging distributed control systems (DCSs) at a chemical facility that control critical plant services including the hydrogen recovery system, ammonia stripper, plant tank farm, and process wastewater treatment. 

 

The Existing System

The existing plant services control system included four Moore Products APACS controllers with five ProcessSuite (now Siemens) operator stations all connected over a redundant ModulNET backbone (Figure 1). The ModulNET communications were a token-based communication protocol over a coaxial trunk that provided for communications between the controllers and data acquisition, control, and configuration from workstations equipped with a communication card. 

Figure 1. A diagram of the existing system’s control hardware and communication structure.

To start, these plant services were spread out over a large distance in the facility, and none of the services were located near the group leader workstation where the maintenance group is located. While they had a coax network (ModulNET or MNET) running along telephone poles and the systems were connected using a proprietary communication protocol, the HMIs did not talk to each other. Each DCS was an island, which meant that even tasks such as updating the HMIs required employees to walk around with a floppy disk to load the changes. 

Additionally, each DCS was developed under different iterations of ProcessSuite, and conventions and blocks were changed over time. While the interface was blended, the system lacked a single convention for the implementation of everything from graphics and symbols down to the device level control. Plus, once we got into the design for these upgrades, we realized drawings for the I/O did not exist, which meant we had to devise a cabling assembly from scratch.

Project Goals 

1. Preserve resiliency in the network architecture: The existing DCSs had independent operator workstations that each had access to all the controllers on a redundant coax-based network trunk. The customer wanted to preserve this system resiliency with the upgrade.

2. Provide a common look and feel: The existing graphics for the four areas were developed by different integrators over time with different graphics conventions, which presented a disjointed experience when switching between screens.  

3. Create and implement a standard code base for the controller logic: While the controller and I/O hardware were consistent between the various areas, the configurations were not. This increased the learning curve for new engineers and technicians supporting the systems.

4. Improve the connection between areas of the plant and implement a method for remote support: The coaxial backbone allowed for proprietary communications between the controllers involved in plant services and from the controllers to the operator workstations, but there was not a network in place that allowed for updates to the individual operator clients. This often led to some clients having updates that the others did not.

5. Keep downtime to a minimum: The plant services systems are critical to the operation of the overall plant, so we were asked to find ways to limit downtime when replacing the controls in each of the areas.

The Solution

Since Applied Control Engineering (ACE) had supported these control systems for more than a decade, when the client decided it was time to reduce the growing operational risks of running these obsolete systems, they engaged our team in conversations around modernizing the obsolete DCSs for each plant service. As we explored the current system described above and shown in Figure 1, we recommended that the client use this modernization as an opportunity to make several additional changes to their network and control systems to eliminate other issues and inefficiencies.

As we planned for the modernization, we had to consider that these were critical systems for this facility and that each system had a limited window for downtime – a week for the ammonia stripper, a few days for the hydrogen recovery system and tank farm, and less than a day for wastewater treatment. This meant we needed to take a multi-phase approach to the implementation.  

 

Phases 1 and 2:

New Plant Network and Wastewater Treatment and Tank Farm DCSs

In the first phase, we implemented the first part of a new OT network. It was set up with managed switches, based on Moxa networking hardware, arranged in a plant loop with one chain (another loop that connects and overlaps the initial loop), and covered the areas where new SCADA and PLC assets would be placed in subsequent phases. This formed the basis of the plant’s new fault-tolerant network. In this phase, we also assisted the plant’s IT department in the development of a Cisco VPN device that would allow the plant to more easily receive remote support when problems arise.

In the second phase, we replaced the existing InTouch HMI stations as well as the APACS control systems for the wastewater treatment facility and plant tank farm with Rockwell Automation Allen-Bradley ControlLogix PLCs with the Wonderware System Platform. To do this within the allotted time frames, we put the new systems in place next to the legacy systems and began cutting those over primarily through the use of cabling. For the wastewater treatment system especially, this method was crucial because we only had eight hours to cutover the control system. However, due to unprecedented rainfall during this time, the wastewater was at an all-time high and we had to complete the cutover in just three hours.

We converted the graphics to have an updated look and feel using ArchestrA graphics. We also included a Wonderware Historian for data collection, client-side trend displays, and reporting as well as centralized security that incorporated a local OT domain with two domain controllers to manage users and thin client sessions. During this upgrade we left two PCs in place to act as legacy tag servers for the remaining APACS systems. The new architecture was based on virtualized system servers, with two virtual terminal servers serving client sessions out to the various areas of the plant utilizing thin client technology and ACP’s ThinManager software.

One of the big challenges we faced during this phase was the need to design cabling for the ControlLogix PLCs that would work with the existing legacy I/O terminations. We addressed this by wiring directly into the existing systems using a cable we developed that connects directly into the Marshall termination panel of the APACS system. Another issue that came up during this phase that we had to address was in the tank farm where we saw scan time issues with daisy chained HART devices. We had to work with Prosoft and bring in some of our specialists to reliably reduce the scan times of all channels with multiple drops to under 10 seconds.

 

Phase 3:

Upgrading DCSs for the Hydrogen Recovery and Ammonia Stripper Systems

In phase three, we migrated the remaining APACS control systems to the ControlLogix platform and reworked the affected ArchestrA graphics to include the new objects. One of the biggest challenges for this cutover was that the hydrogen recovery system required a custom connector that we had to design so that it could plug directly into the termination cable. Upgrading the ammonia stripper control system was also difficult as we had to remove the whole backplane, put a new one in, and rewire the I/O. Figure 2 shows the new hardware and network configuration.

Figure 2. A diagram of the upgraded control hardware and communication structure.

As we got into the upgrade, we realized we also had to develop a method for interfacing to the facility’s custom radio system. They were using a callout system based on WonderWare’s alarm printer installed on a local HMI client in the Hydrogen area to pass serial messages to the Motorola call-out system, which would not work with the new server-based architecture and the use of thin clients and virtualization. 

To make this legacy alarm system work with the new control system, we had to build an interface that used a Moxa NPort device as a remote connection to our new virtualized HMI servers. Those servers used a custom service developed by ACE that determined which server would provide the alarms to the legacy alarm system. In the event of a failure, the local server would notice the change in state and designate the surviving server as the alarm provider.

We later had the opportunity to further upgrade the alarm system by removing the 1990s computer that was in place and replacing the custom system with SeQent industrial alarm notification software. The software module we used with SeQent was designed to plug into System Platform, eliminating the need for the custom services we developed and the legacy message manager they were using. SeQent could now communicate directly with the Motorola radio system to pass on the messages the client needed to see. In addition to eliminating all the legacy bits of the interface, this new system was now easier to maintain and administrate.

 

Benefits of a Modern DCS and Connected Plant Services Network

With this new network architecture and more modern control hardware and software in place for these plant services, the facility experienced multiple benefits. First, their control systems were no longer using obsolete hardware and the risk of a prolonged outage due to the inability to quickly get replacement parts was mitigated. Second, their islands of automation were eliminated and all the information for these disbursed systems is now on the same network and can be viewed from a centralized group leader workstation.

Third, since we began implementing a site-wide network with this upgrade that would eventually connect their various buildings in the OT space, personnel, especially the maintenance team, now has more visibility across the whole plant and the client can get more usage out of the new network. Fourth, since we installed one of the virtualized servers at the group leader workstation and the other in the tank farm, systems can continue to operate independently if there is a network outage somewhere.

Learn more about our extensive control system modernization experience and how we can work with you to evaluate your aging system to ensure you get the most out of your upgrade.

Learn more about our control system modernization services