|Brought to you by ControlGlobal.com and Putman Media
||June 14, 2006|
Headlines from Today's Activities
Securing open control systems the ExxonMobil way
The increasing use of "open" systems—which he further classified as general-purpose operating systems (especially Windows), interconnected TCP/IP networks, and standard data exchange mechanisms—within the process control environment offers many benefits, he explained. But at the same time, these open systems introduce security risks that must be carefully managed, he said.
Nye went on to explain how ExxonMobil manages security risks as an integral part of its process control practices. In addition to the technical factors that are most often discussed, there are policy aspects and people aspects that need to be addressed, he said. "We integrate security into our safety and reliability practices."
One aspect of security risk mitigation at ExxonMobil has to do with the separation of the process control environment from both safety and business systems. Firewalls separate basic controls from supervisory control systems, which are further firewalled from MES-level systems. Safety and control system integration is limited to read-only access of the safety data by the control system. "The separation of control and safety may well prevent a security event from becoming a safety event as well."
Nye further described ExxonMobil's policy of "no Windows" at the basic process control layer, as well as rigorous procedures for managing and controlling anti-virus protection, home-grown node health monitoring agents, and patch management practices for all Windows-based nodes. External network interfaces, including any wireless access points, are centrally managed across the corporation, each having undergone a formal risk assessment process. Authentication and encryption are standard features of any external link, he said.
Using all this information technology for process control tasks also has impacted the company's training and hiring processes. Process control professionals are trained in IT topics and IT professionals are indoctrinated in the priorities of the process control environment. "As far as new hires go, we prefer a dual degree in chemical engineering and computer science," Nye said.
"It's fundamental to note that we're talking about a process control system that uses IT, not an IT system that does process control," Nye explained, adding that an IT system's first security priority is often the data, which often can best be protected by shutting down access. "But with a process control system, it's not about protecting the data--it's all about protecting the process."
"Security is a journey not a destination—new threats are emerging all the time." ExxonMobil’s Johan Nye.
ExxonMobil's Top 10 Windows Security Practices:
1. Use Windows Integrated Security for authentication and authorization
2. Give each role the minimum permissions needed
3. Disable anonymous access
4. Turn off unused Windows services
5. Have end-user change "built-in" passwords and account names
6. Store and transmit passwords in cyphertext
7. Run all client applications under logged in user's credentials
8. Applications should write only to data directories—not bin or system32
9. Log security events to Windows Security Event Log
10. Products should meet "Certified for Microsoft Windows" requirements
DuPont DeLisle rebounds from Katrina
DuPont quickly got Honeywell involved in the rebuild effort. The project involved 85 HPM/C200 cabinets, 34 HPM/C200 redundant processors, 20,000 hard I/O points, 100,000 wiring terminations, and 42 custom marshalling cabinets. If the size of the project wasn't daunting enough, the DuPont work posed special demands because it required special components instead of standard off-the-shelf units, noted program manager Bob Eubank, who headed Honeywell efforts onsite.
From September 28 to October 4, engineering and construction resources were mobilized, including those of Triad, which Honeywell brought on as its construction subcontractor. DuPont identified the priorities for rebuilding the electrical control rooms. Meanwhile, the Honeywell factory set up a staging area specifically for the DuPont project.
Delivery schedules were shaved from the usual 12 to only 6 weeks. Dedicated truck shipments from Phoenix cut shipping time to 36 hours. Getting the powerhouse up and running was DuPont's first target, and that was achieved on November 16, with the environmental systems online a week later.
All the control rooms were ready ahead of schedule, and the entire project was completed 11 days early, in 11 weeks, notes Eubank. "Factory quality was excellent on cabinets - no rework had to be done onsite," he adds with pride. In addition, there were few wiring errors among the 100,000 re-terminations.
While DuPont had adopted "replacement in kind" as its basic strategy, "DCS recovery was an opportunity to replace outdated equipment," noted Wiles. DeLisle had 23 Logic Managers and already was planning to upgrade to Experion C200 controllers. "While four Logic Managers are still running in other parts of the plant, the rest have been replaced with 5 C200s."
Despite the push to get the plant back in operation, safety procedures were stringently enforced, noted Eubank. "Throughout the five month period, no recordables or first-aid injuries were reported."
"We faced a 28-ft. storm surge with a 20-ft. levee." Electrical engineer Guy Wiles related tales of extraordinary effort and teamwork to rebuild after Katrina.
Alarm management gone wrong, wrong, wrong...
They had persistent alarm problems after the cutover, including large numbers of active and cycling alarms, over 30 system alarms, numerous Bad PV alarms, and major problems with serial annunciator communications. Alarms were being ignored, and this contributed to a Level 2 environmental event, and a significant union grievance.
"Sometimes it is fun to be a controls engineer," Swindler noted ruefully, "but this project made me want to hide in a corner."
The site's operator training and culture was based on the TDC2000, with a hardwired annunciator for alarms. The operators were accustomed to "operating to alarm."
"The operators believed that the only real alarms came from the annunciator," Swindler said. "We didn't change the site culture," Swindler went on, "and the engineering contractor was very weak on alarm management and rationalization skills. We didn't hire an alarm management professional to lead the rationalization process."
Another real problem was the Triconex-to-Experion link. Because there is no native way for Triconex to talk directly to Experion, the decision was made to use a Matrikon OPC server in a redundant (1oo2) configuration. "If we had to do it over again," Swindler said, "we'd have gone with Modbus. In my experience OPC is a complex and, I think, fragile technology."
The operators were overdependent on the annunciator, and were upset when communications loss caused the Ronan annunciator to freeze, and then, when it came back up, there would be sporadic re-flash of existing alarms. A firmware upgrade in the annunciator ultimately fixed the problem.
"Working with Experion requires a different skill set than working with older systems," Swindler added. "Our operators didn't have the IT background to get the best out of the training they received. They were uncomfortable with working on the system, and they still are—but we're working on getting them the training they need."
Swindler also described a problem with Honeywell's treatment of HART data in R210. The outputs can't be switched from HART to regular without deleting the tag and rewriting one, he complained. "Needless to say, the operators don't think too highly of doing that while they are trying to operate the plant."
Then there were the project management issues, which applied to both the DCS and SIS. The project was behind schedule, and the contractor performed poorly. Lyondell decided not to do a FAT (factory acceptance test), didn't do as thorough a SAT (site acceptance test) as they should have, and therefore didn't see many of the problems until they started to cutover. They also didn't do a field survey, and they didn't have a functioning MOC system to deal with changes.
"We only did about 50% of our loops as hot cutover," Swindler said. "If we had done them all hot, we would have discovered the problems loop by loop, instead of all at once."
So what lessons can be learned from this project?
"First, change the culture and provide effective training," Swindler said. "Second, do your alarm rationalization properly. Third, exercise caution with new technology, like OPC. Fourth, an SAT is critical, and shouldn't be shortened or done without, regardless of the impact on the schedule," he said. "Fifth, properly engineered PV settings and alarm bandwidth are important and should not be minimized," he continued, "and sixth, I recommend you maximize hot cutover regardless of what the 'experts' say."
"Operator input should be balanced by process and controls engineering perspective for an effective alarm rationalization effort." Lee Swindler on a recent alarm "mismanagement" project at Lyondell.
|Sign up to receive information on Honeywell Users' Group and next year's event as it becomes available. Send an e-mail to firstname.lastname@example.org.|
© 2006 Putman Media Inc.