Modernizing a Nuclear Reactor Control System
""Challenge
The Transient Reactor Test Facility (TREAT) at Idaho National Laboratory was built in the 1960s and ran until the early 1990s. Since the 90s it has been dormant. In 2017 the reactor was brought back online with its legacy systems to resume testing. Updating 30-year-old electronics and software would allow for better control and easier system maintenance.
Solution
We reverse engineered the existing control system from design documents and built a baseline replacement with off the shelf system components offering the modern reliability of the PXIe platform. We also built a test system that could emulate the reactor plant that was used as a test bed for verifying the replacement control system.
Introduction
TREAT is a transient test reactor whose main purpose, unlike typical commercial power reactors, is to create extreme environments which are used to test and develop new fuels and materials that are designed to live within a reactor. The reactor itself was built in the 1960s with transients’ through the early 90s. In the 90s the reactor was placed in hibernation as national nuclear research interest shifted priorities. Following the Fukushima nuclear disaster in 2011 the plant was scheduled to be brought back online to resume nuclear research. The plant became operational again in 2017 with the hardware and control systems of the late 80s. In the last 30 years technology has advanced a lot and the reactor control system was due for an upgrade.
The TREAT reactor is designed to create power excursions of up to 20 gigawatts in less than one second. To achieve this sort of control the current revision of the Automated Reactor Control System (ARCS) was designed in the late 80s to allow engineers to control the reactor at a 2ms resolution. This system was built on the intel 8086 platform and was written largely in assembly code. Maintaining computers whose components were last produced more than 3 decades ago is a real challenge.
The TREAT reactor is designed to create power excursions of up to 20 gigawatts in less than one second. To achieve this sort of control the current revision of the Automated Reactor Control System (ARCS) was designed in the late 80s to allow engineers to control the reactor at a 2ms resolution. This system was built on the intel 8086 platform and was written largely in assembly code. Maintaining computers whose components were last produced more than 3 decades ago is a real challenge.
Process
Establishing a Baseline
In 2017, we started to reconstruct the ARCS software in LabVIEW. This consisted of many long hours of reverse engineering assembly code and studying documentation to come as close to a replica system as possible. The goal was to reproduce exactly how the original system controlled the reactor to ensure that the new system would interface properly with the existing reactor hardware. This type of reconstruction added the additional challenge of keeping many of the constraints of the 8086 platform. At the same time the baseline allowed us to demonstrate that the new ARCS systems would meet requirements of the old system. This was accomplished by running a series of tests originally designed to test the functionality of the old system against the new system and showing that the resulting output data was the same.
In 2017, we started to reconstruct the ARCS software in LabVIEW. This consisted of many long hours of reverse engineering assembly code and studying documentation to come as close to a replica system as possible. The goal was to reproduce exactly how the original system controlled the reactor to ensure that the new system would interface properly with the existing reactor hardware. This type of reconstruction added the additional challenge of keeping many of the constraints of the 8086 platform. At the same time the baseline allowed us to demonstrate that the new ARCS systems would meet requirements of the old system. This was accomplished by running a series of tests originally designed to test the functionality of the old system against the new system and showing that the resulting output data was the same.
Enhancements
After building out the new software to match the old software as closely as possible we set out to achieve some enhancement goals as well as provide some quality of life improvements to the people who use the software on a daily basis. Some of those enhancements included a graphical interface to control and interact with the ARCS. Previously, it had been controlled by handing the ARCS its input parameters and telling it to run. The reactor operator had little feedback from the system itself and monitored its progress from the plant indicators. In addition to those still existing plant indicators, the graphical interface now allows the operator insight into what the software is doing, how close to the prescribed limits the automatic control software is getting and what type of control the software is currently exercising. It also provides useful debugging information including live feeds of all available system IO. Other enhancements included modernizing the data input and output so that operators and engineers could more accurately design test campaigns as well as get data feedback instantaneously from the system rather than having to download and then convert the data from binary to human readable data.
After building out the new software to match the old software as closely as possible we set out to achieve some enhancement goals as well as provide some quality of life improvements to the people who use the software on a daily basis. Some of those enhancements included a graphical interface to control and interact with the ARCS. Previously, it had been controlled by handing the ARCS its input parameters and telling it to run. The reactor operator had little feedback from the system itself and monitored its progress from the plant indicators. In addition to those still existing plant indicators, the graphical interface now allows the operator insight into what the software is doing, how close to the prescribed limits the automatic control software is getting and what type of control the software is currently exercising. It also provides useful debugging information including live feeds of all available system IO. Other enhancements included modernizing the data input and output so that operators and engineers could more accurately design test campaigns as well as get data feedback instantaneously from the system rather than having to download and then convert the data from binary to human readable data.
Simulation to expedite development and testing
One of the challenges that we faced along the way was the inability to test what we were developing on the reactor plant. To solve this problem, we built out an entire replica set of the hardware that we would later use in the plant. This set of hardware was configured to be identical to what was going to run in the plant. In addition to this set of replica test hardware, we developed a new system called the Reactor Plant Emulator (RPE). This system was designed to be a stand-in for the plant so that we could test the new software against the plant months before the deployment and acceptance testing. This system was built to have the same IO connections as the actual plant as well as simulate the reactor behavior, the behavior of the nuclear instruments and the behavior of the surrounding safety systems and auxiliary utilities. Additionally, we provided an interface to the RPE that allowed for control of the simulated system in many of the same ways that the TREAT reactor is controlled. This included a virtual control room, virtual access to the various modes of plant operation as well as the ability to set the virtual safety hardware to match physical test boundaries to simulate edge case scenarios of testing in the plant. Because of the RPE, we were able to not only test that the software would behave as expected, but that the procedures being written to use the software were correct. The greatest benefit of building a simulation system came when it was time to install the new system in the plant. Because of all of the testing that was done on the simulation system we were able to expedite testing in the plant, minimizing the operational downtime of their system as a whole.
One of the challenges that we faced along the way was the inability to test what we were developing on the reactor plant. To solve this problem, we built out an entire replica set of the hardware that we would later use in the plant. This set of hardware was configured to be identical to what was going to run in the plant. In addition to this set of replica test hardware, we developed a new system called the Reactor Plant Emulator (RPE). This system was designed to be a stand-in for the plant so that we could test the new software against the plant months before the deployment and acceptance testing. This system was built to have the same IO connections as the actual plant as well as simulate the reactor behavior, the behavior of the nuclear instruments and the behavior of the surrounding safety systems and auxiliary utilities. Additionally, we provided an interface to the RPE that allowed for control of the simulated system in many of the same ways that the TREAT reactor is controlled. This included a virtual control room, virtual access to the various modes of plant operation as well as the ability to set the virtual safety hardware to match physical test boundaries to simulate edge case scenarios of testing in the plant. Because of the RPE, we were able to not only test that the software would behave as expected, but that the procedures being written to use the software were correct. The greatest benefit of building a simulation system came when it was time to install the new system in the plant. Because of all of the testing that was done on the simulation system we were able to expedite testing in the plant, minimizing the operational downtime of their system as a whole.
Results
As of early 2020 the ARCS replacement has been controlling transients successfully for the better part of a year. Due to the new systems design and with the collaboration of the engineers at Idaho National Labs we have been able to pinpoint the cause of and solve a number of legacy issues with the new system and the ability to iterate quickly against the development system. We have also vastly improved quality of life for the new engineers and operators who use the systems by updating the points of interaction with the system, offering higher resolution output data, live feedback of what every part of the system is doing during a transient as well as offering new tools and utilities that accelerate the experiment cycle at TREAT.
Benefits
- Valid upgrade paths with modern components
- Vastly improved workflow
- Modern user interfaces that give real-time feedback on what the system is doing
- Modern and flexible data collection
- Valid upgrade paths with modern components
- Vastly improved workflow
- Modern user interfaces that give real-time feedback on what the system is doing
- Modern and flexible data collection
Hardware
- NI PXIe-8880 (x4)
- NI PXIe-6363
- NI PXIe-6535
- NI PXIe-6738
- NI PXIe-7846R
- NI PXI-6683H
- NI PXIe-8880 (x4)
- NI PXIe-6363
- NI PXIe-6535
- NI PXIe-6738
- NI PXIe-7846R
- NI PXI-6683H
Software Modules
- LabVIEW 2017-2019
- NI-DAQmx
- NI-Multifunction RIO
- NI-Serial
- NI-Sync
- NI-Switch
- NI-VISA
- NI Phar Lap Realtime Operating System
- LabVIEW 2017-2019
- NI-DAQmx
- NI-Multifunction RIO
- NI-Serial
- NI-Sync
- NI-Switch
- NI-VISA
- NI Phar Lap Realtime Operating System