SoftGrid: A Testing Toolkit for Evaluating Substation Cybersecurity Solutions

Background

Although a number of security mechanisms, such as security-enhanced substation gateways, application-layer firewalls, intrusion detection systems(IDSes), etc., have been proposed it remains a challenge to evaluate such substation cybersecurity solutions in a realistic environment. SoftGrid is designed and implemented for facilitating evaluation of effectiveness, performance, and interoperability of the developed security solutions and thereby accelerating research and development in related areas.

Design Goals

Real-world interoperability: It is highly desirable if the testbed can be used to assess products or prototype solutions that are meant for actual deployment. Also, our goal is to evaluate a substation cybersecurity solution in a setting that resembles a real-world environment. To achieve so, it requires the testbed modules that interface with the substation cybersecurity solution under test to be able to communicate directly with those real-world solutions or devices. Since we focus on security solutions that are to be deployed at or around the substation gateways, the testbed needs to have a control center module and IED modules that can communicate with the solution under test using standard protocols.

Flexible and scalable power grid configuration: A substation cybersecurity solution needs to be tested against a variety of power grid configurations. Hence, the testbed needs to be flexible enough to allow users to conveniently define different power grid settings. Besides flexibility, it is highly desirable for a testbed solution to support settings that resemble large-scale power grid systems. In fact, scalability is a major limitation of hardware-based approaches. A solution may perform very differently in a system with a few dozen buses, as compared to a larger system with hundreds or even thousands of buses. In this sense, software-based approach is desirable.

Tight cyber-physical synchronization: To be used for testing of substation cybersecurity solutions, it is important that both cyber infrastructure and physical power system in the testbed can work in a tightly synchronized manner. For instance, from the perspective of a substation gateway device under test, the cyber side and physical side should behave just like that in a real environment. More specifically, once some incoming remote control commands are received and executed, their impact on the physical systems, as the outcome of the control commands, should be reflected in real-time to the subsequent interrogation commands issued to the substation gateway.

Advanced Evaluation Support: The testbed should offer features for monitoring and logging power grid status for both online and offline analysis and evaluation. The key metrics to be monitored depend on the purpose of the experiments. For example, they may include the amount of power flow on each transmission line, status of circuit breakers, generation and/or load amount, voltage on each bus, power grid frequency, etc. In addition, threshold-based violation detection based on these measurements is also desired. Moreover, some grid stability issues, such as short-term frequency deviation, may only be visible only during transient state. Thus, to complement steady state simulation, the integration of transient stability analysis is highly desired for detailed impact assessment. Additionally, feasibility to manage and repeat/reproduce experiments is a general requirement for a testbed, which we also must implement.

Turn-key Solution: Besides meeting the requirements discussed above, we aim at a testbed that is easy to set up and operate. For users (e.g., grid operators and security solution vendors), some of which might not be IT experts, to quickly construct the testing environment, a turn-key solution that requires minimal configuration (e.g., just by defining power grid for simulation and plugging in a security solution to be tested) is highly demanded.

System Architecture and Implementation

The figure below shows the architecture of the SoftGrid testbed. Shaded components show examples of possible security solutions/devices that can be evaluated, such as security-enhanced gateways, firewalls, and IDSes. In our design, they can be plugged into the SoftGrid testbed easily. Note that the substation security solutions under test often require the support from some external system component, such as an attack detection system deployed at the control center.

Standard-compliant Control Center and IED: To meet the real-world interoperability goal, it is crucial that the components that interact with the security solution under test, namely the control center and IEDs, must support standardized protocols that are supported by the solution. Our initial implementation supports IEC 60870-5-104 at the control center side and IEC 61850 MMS at the IED side. This is a common configuration of many power grid systems in Europe and Asia. To implement such standard protocols, we employed an open-source library called OpenMUC. On top of OpenMUC, SoftGrid's control center module implements a command-line user interface for sending interrogation and control commands to substations, and a text-based console to display interrogation results. Likewise, IEDs that are compliant with IEC 61850 MMS are implemented using OpenMUC. In our current implementation, each IED is modeled as a simple communication endpoint that receives interrogation and control commands and sends back responses. This is because the current focus of SoftGrid testbed is to assess solutions for securing the remote control interface of substations. We should also note that default ICD (IED Capability Description) files and SCD (Substation Configuration Description) file required for IEC61850 are generated automatically by SoftGrid according to model definitions of a simulated power grid (i.e., a PowerWorld case file, to be explained later). Therefore, basic experiments can be started immediately after loading a designed PowerWorld case file, satisfying the turn-key property.

Cyber-physical Integration: To support flexible and scalable power grid configuration, we employed the PowerWorld simulator, a high-fidelity power system simulator widely used by academia as well as industry users. PowerWorld allows users to arbitrary define power grid topology and detailed configuration of each physical component. With that, SoftGrid can simulate different power grid settings by loading different PowerWorld case files. We also tested SoftGrid with large power grid systems, e.g., the 2000-bus system, and found that PowerWorld can simulate such systems on commodity PCs with very low processing latency. To achieve our third design goal on tight cyber-physical synchronization, we keep a PowerWorld instance running throughout the experiment, and bridge our IED modules with PowerWorld using PowerWorld's COM APIs. To ensure up-to-date status of a simulated power grid is available to the testbed, Transient Clock module is responsible for periodically requesting PowerWorld for recalculation (e.g., every 20ms in our current implementation) to reflect changes made by control from IEDs. In this way, from the perspective of the security solution under test, the simulated power grid behaves similar to a real system.

Monitoring and Evaluation Support: Various logs are generated on the control center and IEDs. The control center records history of control or interrogation commands it sends out as well as responses received from IEDs. Through the interrogation response, at anytime during the simulation, the control center can learn the status of power grid, such as bus voltage, amount of power flow on each transmission line, and so forth. On the other hand, IEDs record received commands along with timestamp. In addition, each IED periodically (e.g., every 20ms) polls status information from a physical component on a power grid simulator and logs them on Monitoring Engine, which is used not only for a variety of evaluation regarding impact on grid stability but also for visualization on Monitoring Window. As can be seen on the figure below, Monitoring Window implements an SQL-like front end (see the top pane of Monitoring Windows in the figure) so that a user can select log data to be displayed. Moreover, data logged by Monitoring Engine (e.g., timestamp and payload of received commands) are also used to generate an input file, called an AUX file, for PowerWorld's transient stability analysis, which is passed to a separate instance of PowerWorld simulator via PowerWorld COM API for transient state simulation. Monitoring Engine can also log and count typical violations in voltage, frequency, etc.