- A healthy sensor and a full asset inventory do not prove that controller-to-HMI traffic is being captured.
- SPAN source selection and SPAN capacity need to be reviewed together. Fixing one without the other leaves gaps.
- Mirror configuration belongs under security change control and should be re-verified after network changes.
I worked on a passive Operational Technology (OT) monitoring deployment at a discrete manufacturing plant that, by every dashboard metric, was going well. The sensor reported healthy, traffic was coming in, and the asset inventory kept growing as controllers were discovered.
The problem showed up when I reviewed the communication map the platform had built. For a working production line, it was thin. The Programmable Logic Controllers (PLCs) were in the inventory, but there was very little traffic between them and the Human-Machine Interfaces (HMIs) the operators use. That traffic matters because it is where operators read process values and change setpoints. Engineering workstations, which handle logic downloads, often connect through the same local switches.
As Robert M. Lee, CEO and Founder of Dragos, stated during the SANS Keynote at RSAC 2026 Conference, “It is very realistic that you could have outages that we don’t know how to recover in a reasonable amount of time.” His point was that working out root cause in modern OT is getting harder, not easier. That stuck with me, because nobody can reconstruct an incident from traffic the sensor never saw. PLCs generally can’t run security agents, so the network is often the only record of what touched them.
Weak visibility near the controllers isn’t unique to this plant. In Fortinet’s 2026 State of OT and Cybersecurity Report, only 14% of respondents said they had full visibility into their OT systems, and almost a quarter could see only about half of their environment. By the dashboard’s measure, our plant would have counted itself in that 14%. The joint CISA guidance on Zero Trust for OT published in April 2026 is clear about how this kind of monitoring works: passive OT monitoring usually depends on SPAN ports or network taps. It also tells owners to design the monitoring architecture with resilience in mind. Our mirror had never been looked at that way.
The Answer Was in the Switch, Not the Platform
The sensor relied on switch mirroring (SPAN). When I checked the configuration, the production VLAN where the PLCs and HMIs sat was not in the source list. The sensor was only receiving a copy of the uplink to the distribution layer.
On an access switch, unicast traffic between two local ports is switched directly and never crosses the uplink. The sensor saw the controllers whenever they talked to upstream systems such as the historian, which is why the inventory looked complete. It missed most of the local PLC-to-HMI conversations. An engineering laptop plugged into the same cabinet switch as a PLC could connect, make a change and disconnect without that session ever reaching the sensor.
When I raised the gap, the first response was that switch configuration belonged to the network team and wasn’t really a security issue. I understood where that came from. But if a switch setting decides what the security sensor sees, it is a security issue, whoever owns the switch.
Adding the VLAN Was Only Half the Fix
Adding the production VLAN was the obvious step, and we did it. The same mirror session was also copying IT and office traffic that had nothing to do with the control system. A SPAN destination has limited capacity, and Cisco’s SPAN documentation notes that an oversubscribed destination can drop mirrored packets while production traffic carries on normally. Depending on the switch platform, interface counters and SPAN diagnostics can help show when that is happening. Adding sources without removing the noise would likely have created a different gap.
So, we made both changes together: we added the production VLAN and removed the unrelated IT and office traffic from the mirror sources.
Testing It, and Being Clear About What the Test Proved
This time I didn’t want to take the dashboard’s word for it. We ran an approved engineering session against a controller, and the platform captured and identified it correctly.
I recorded that result narrowly. It showed that this type of session was visible through this collection point after the change. It did not prove coverage of every controller on site, and it did not measure how much packet loss we had removed. Equipment behind OEM-managed networks or serial gateways needs its own evidence.
What We Changed Afterward
The bigger lesson for me was that the mirror configuration is part of the detection system, even though it lives on a network switch. A routine change, such as moving an uplink to a different port, can leave the mirror pointing at the old port with no alert and a sensor that still shows healthy. Failover paths raise the same question.
We made three process changes:
- Mirror configuration changes now go through security review.
- After network work on those switches, representative traffic is checked on the platform.
- Go-live validation includes a known engineering session test. If the platform can’t capture and identify it, the deployment isn’t finished.
For teams running passive OT monitoring, a practical starting point is one critical conversation, such as an engineering workstation talking to a key PLC. Trace its path, confirm which mirror source copies it, and test it. As Lee put it, it is getting harder to answer one question after an OT failure: “Was it a cyberattack or just a mishap?” The only way to answer it is with a record of what actually crossed the wire, and that kind of visibility has to be proven one conversation at a time. Our gap surfaced because someone looked at which conversations were actually there, not because the health indicator changed.