The following is a lightly anonymised account of a real DFIR engagement conducted in coordination with government partners. Client identifiers and geographic specifics have been altered. All technical findings, artifacts, and commands are presented as they occurred. We are publishing this case — with appropriate review — because it illustrates something critical infrastructure operators systematically underestimate: an air gap that has never been verified is not an air gap. It is a hope.
| Parameter | Detail |
|---|---|
| Case Type | Nation-State APT — OT/ICS Pre-Positioning (VOLTZITE/Volt Typhoon TTP cluster) |
| Industry | Critical Infrastructure — Municipal Water Treatment (anonymized region) |
| Duration | 31 days on-site + ongoing monitoring engagement |
| Year | Year 7 of operations |
| Notoriety | The one where we found a 7-month-old implant and the client said ‘huh, neat’ |
The Engagement
How We Were Called In
The referral came through a government threat sharing partner with the kind of carefully understated language that intelligence community contacts use when they don’t want to commit anything alarming to written communication: ‘potential indicators of unauthorized access to operational technology systems at a water-related critical infrastructure site.’ We translated this to English: someone may be in a water treatment plant’s control network, and we would very much like to know who, why, and for how long.
The facility treated drinking water for a mid-sized municipality. The IT/OT architecture was, charitably, organic. Thirty years of incremental additions had produced: a SCADA system running on Windows Server 2003 (not a typo), a Historian database from a vendor that had since been acquired, split, and partially discontinued, a corporate IT network connected to the OT network via a device the plant manager described as ‘the box that keeps them separate’ — which turned out to be an unmanaged switch. Not a firewall. Not a data diode. A switch.
The air gap wasn’t an air gap. It was a switch with optimistic labelling and a handwritten ‘DO NOT TOUCH’ sticker.
Initial Deployment — Passive Monitoring
We deployed passive network monitoring sensors (SecurityOnion on ruggedized hardware) on both the corporate IT segment and the OT segment. In OT environments, active scanning is prohibited — a port scan against a live PLC can cause a process disruption. Everything had to be passive. We captured traffic, sat on it, and let the anomalies surface.
It took six hours.
Finding 1: The Unknown Device That Shouldn’t Exist
Path: Sensor deployment: SPAN port on OT-segment core switch | Zeek conn.log, dns.log, http.log, ssl.log
- Zeek conn.log captures:
ts,orig_h,resp_h,resp_p,proto,service,duration,orig_bytes,resp_bytes - OT baseline: known-good traffic should be almost entirely Modbus/502, DNP3/20000, SCADA historian ports
- Unknown IP or MAC with internet-routable destination = immediate escalation in OT environment
- ICS-specific Zeek scripts:
zeek-plugin-s7comm,zeek-plugin-modbus,zeek-plugin-dnp3(available via GitHub)
# Zeek conn.log analysis — identify hosts communicating with internet-routable IPs # (OT networks should have ZERO internet-routable outbound connections) # Using zeek-cut and awk on OT segment conn.log: cat /nsm/zeek/logs/current/conn.log | zeek-cut orig_h resp_h resp_p proto service duration | \ awk '$2 !~ /^(10\.|172\.(1[6-9]|2[0-9]|3[01])\.|192\.168\.|127\.|0\.0\.0\.0)/ \ && $2 != "255.255.255.255" {print}' | \ sort -k1,2 | uniq -c | sort -rn | head -20 # OUTPUT (after 6 hours of capture): # count orig_h resp_h resp_p proto service 847 10.20.0.xxx 103.27.xx.xx 443 tcp ssl 847 10.20.0.xxx 103.27.xx.xx 443 tcp ssl # 10.20.0.xxx is NOT in the plant's asset register. # 103.27.xx.xx resolves to a Singaporean VPS — flagged in threat intel feed # (confirmed association: VOLTZITE/Volt Typhoon operational relay infrastructure) # MAC address lookup for unknown host: arp -n | grep 10.20.0.xxx # 10.20.0.xxx ether dc:a6:32:xx:xx:xx C eth0 <-- DC:A6:32 = Raspberry Pi Trading Ltd OUI
The Organizational Unique Identifier (OUI) prefix dc:a6:32 is registered to Raspberry Pi Trading Ltd. Every Raspberry Pi 3 and 4 manufactured after 2018 carries this MAC prefix. In a water treatment plant OT network, a Raspberry Pi is not a normal device. We had our target.
Physical inspection of the network closet identified the device within twenty minutes: a Raspberry Pi 3B+, painted grey to match the patch panel, zip-tied to the back of it, connected via a short blue Ethernet cable to a port labelled ‘SPARE — DO NOT USE.’ It had been there a while. The dust pattern around it was undisturbed.
Raspberry Pi Implant Discovered After 6 Hours of Passive Monitoring — MAC OUI Unmistakable
Zeek passive capture on the OT segment identified an undocumented host (10.20.0.xxx) beaconing to 103.27.xx.xx (confirmed VOLTZITE/Volt Typhoon relay infrastructure) on port 443 at regular intervals. The MAC OUI confirmed a Raspberry Pi. Physical inspection located the device zip-tied to the back of a patch panel in the OT network closet — painted grey, unregistered, and undisturbed for an estimated 7 months based on dust accumulation and filesystem timestamps.
Finding 2: Implant Forensics — Raspberry Pi Filesystem Analysis
Path: SD card: SanDisk 32GB microSD | Filesystem: ext4 (Linux) | Imaged with: dd + sha256sum verification
- Image with
dd if=/dev/sdX of=rpi_image.dd bs=4M status=progress— confirm SHA-256 pre/post - Mount read-only:
mount -o ro,loop,offset=272629760 rpi_image.dd /mnt/rpi_root - offset = partition 2 start sector * 512 (find via
fdisk -l rpi_image.dd) - Key paths:
/home/pi/,/etc/crontab,/etc/rc.local,/root/.bash_history,/var/log/
# Raspberry Pi forensic acquisition — dd image, partition mount, mactime timeline generation # Acquire SD card image (device powered down — removed from implant) dd if=/dev/sdb of=/evidence/rpi_implant.dd bs=4M status=progress conv=noerror,sync sha256sum /evidence/rpi_implant.dd > /evidence/rpi_implant.dd.sha256 # Partition table: fdisk -l /evidence/rpi_implant.dd # Device Boot Start End Sectors Size Type # rpi_implant.dd1 8192 532479 524288 256M W95 FAT32 (boot) # rpi_implant.dd2 532480 62521343 61988864 29.6G Linux # Mount root filesystem read-only: sudo mount -o ro,loop,offset=$((532480*512)) /evidence/rpi_implant.dd /mnt/rpi_root # Timeline of file modifications (mactime from The Sleuth Kit): fls -r -m '/' /evidence/rpi_implant.dd -o 532480 > /tmp/rpi_bodyfile.txt mactime -b /tmp/rpi_bodyfile.txt -d 2023-01-01 > /tmp/rpi_timeline.csv # First filesystem write after initial SD card preparation: # 2024-xx-xx 03:14:22 — /home/pi/.config/autostart/update-helper.desktop (CREATED) # This date = 214 days before discovery = ~7 months
The filesystem told a careful story. The device had been minimally configured to avoid detection: no unnecessary services, no swap file (minimising write amplification to the SD card), and a persistence mechanism using a desktop autostart file that would survive reboots but not appear in the standard /etc/crontab or rc.local locations a basic audit might check.
# Implant binary analysis — custom ARM ELF beacon, statically linked, 0/61 AV detections # Autostart persistence mechanism (desktop environment autostart — unusual for headless Pi): cat /mnt/rpi_root/home/pi/.config/autostart/update-helper.desktop [Desktop Entry] Type=Application Name=System Update Helper Exec=/home/pi/.local/bin/netman Hidden=false NoDisplay=true X-GNOME-Autostart-enabled=true # Payload binary: file /mnt/rpi_root/home/pi/.local/bin/netman # netman: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, stripped # Strings analysis (abbreviated): strings /mnt/rpi_root/home/pi/.local/bin/netman | grep -E '(http|domain|sleep|interval|beacon)' # beacon_interval=300 <-- 5-minute beacon cadence # jitter_pct=15 # c2_primary=103.27.xx.xx # c2_fallback=185.176.xx.xx # useragent=Mozilla/5.0 (Linux; Android 9) # staging_dir=/tmp/.cache/ # SHA-256 of netman binary: sha256sum /mnt/rpi_root/home/pi/.local/bin/netman # a3fxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx <-- submitted to VirusTotal: 0/61 detections
Custom ARM Implant — 0/61 AV Detections, 5-Min Beacon, Dual C2 Fallback, 214 Days Active
The implant binary (netman) was a custom-compiled ARM ELF executable — statically linked and stripped. Zero antivirus detections on VirusTotal. Configured for a 5-minute beacon with 15% jitter to a primary C2 (103.27.xx.xx) and a fallback (185.176.xx.xx), both confirmed as VOLTZITE relay infrastructure. Filesystem timestamps confirmed 214 days of continuous operation. The persistence mechanism used GNOME autostart to evade basic crontab/rc.local audits.
Finding 3: Lateral Movement — SCADA Historian Access
Path: Historian host Windows event logs | Zeek conn.log (PI port 5450 = historian query port)
- OSIsoft PI System uses port
5450(PI-NET) and5460(PI Buffer Subsystem) for client queries - Legitimate historian access: engineering workstations and HMI hosts only
- Windows Security Event ID 4624 LogonType 3 (network) from unexpected source = lateral access
- PI Audit trail (if enabled): PI SMT → Security → Audit Trail — records tag reads by user/source
# Zeek conn.log — identify connections to historian on PI-NET port 5450 # Expected sources: 10.30.0.10 (HMI-1), 10.30.0.11 (HMI-2), 10.30.0.50 (Engineering WS) cat /nsm/zeek/logs/current/conn.log | zeek-cut ts orig_h resp_h resp_p | \ awk '$3 == "10.30.0.20" && $4 == 5450 {print}' | \ sort -u | head -30 # EXPECTED legitimate sources: # 10.30.0.10 -> 10.30.0.20:5450 <-- HMI-1 (legitimate) # 10.30.0.11 -> 10.30.0.20:5450 <-- HMI-2 (legitimate) # ANOMALOUS source: # 10.20.0.xxx -> 10.30.0.20:5450 <-- The Raspberry Pi. Not registered. Not expected. # Historian Windows Security log — lateral logon from implant IP: # Event ID 4624, LogonType 3, TargetUserName: HISTORIAN$ # Source IP: 10.20.0.xxx (implant) Time: multiple sessions over 30 days # Zeek — calculate total bytes read from historian by implant: cat conn.log | zeek-cut orig_h resp_h resp_p orig_bytes | \ awk '$1=="10.20.0.xxx" && $2=="10.30.0.20" && $3==5450 {sum+=$4} END{printf "%.1f MB\n",sum/1048576}' # Output: 847.3 MB <-- ~30 days of historian data at 5-min polling cadence
847 MB of historian data over 30 days represents approximately 6 months of 5-minute polling interval process data for a mid-sized water treatment plant: flow rates, chemical dosing readings, pump states, tank levels, turbidity measurements, and chlorination levels. This is the data an adversary needs to understand the operational baseline of a plant — what is ‘normal’ — before deciding whether, and how, to interfere.
Implant Queried SCADA Historian Directly — 847 MB of Process Data Exfiltrated Over 30 Days
Zeek traffic analysis and Historian Windows event logs confirmed the implant had established lateral connections to the OSIsoft PI Historian on port 5450 — a port with no legitimate path from the corporate IT segment. 847 MB of operational process data (flow rates, chemical dosing, pump states) was retrieved over 30 days. The Historian had no access controls preventing unauthenticated LAN connections — PI security was entirely perimeter-based, and the perimeter was a switch.
Finding 4: HMI Workstation Access — Engineering Files Staged
Path: HMI host: Windows 7 Embedded (end-of-life) | Staging dir on implant: /tmp/.cache/
- Windows 7 Embedded: no WMI, PowerShell v2, limited audit policy — telemetry is minimal
- Event ID 4624 LogonType 3 from implant IP = network lateral logon to HMI
- SMB lateral movement: look for Event ID 5140 (network share access) and 5145 (file access)
- Staging dir on implant SD card: recovered files confirm what was exfiltrated before wipe
# HMI lateral logon events + staging directory — SCADA project file and OT network diagram exfiltrated # Windows Security log — lateral logon to HMI workstation from implant # (Pulled from HMI-1 Windows event log via direct USB collection) Get-WinEvent -Path .\Security_HMI1.evtx -FilterXPath \ "*[System[EventID=4624] and EventData[Data[@Name='LogonType']='3']]" | ForEach-Object { $xml = [xml]$_.ToXml() $ip = ($xml.Event.EventData.Data | Where-Object Name -eq 'IpAddress').'#text' if ($ip -eq '10.20.0.xxx') { [PSCustomObject]@{ Time=$_.TimeCreated; SrcIP=$ip; User=($xml.Event.EventData.Data | Where Name -eq 'TargetUserName').'#text' } } } # 47 lateral logon events from implant IP over 30-day window — using null session (anonymous) # Files recovered from implant /tmp/.cache/ staging directory: ls -la /mnt/rpi_root/tmp/.cache/ # -rw-r--r-- pi pi 2.4M SCADA_Project_Master_v3.7.rss <-- Wonderware InTouch project file # -rw-r--r-- pi pi 1.1M network_topology_OT_2022.vsd <-- OT network Visio diagram # -rw-r--r-- pi pi 890K chemical_dosing_schedule_2024.xlsx <-- Operational schedule # -rw-r--r-- pi pi 34M historian_export_sample.csv <-- 30 days of process data # CRITICAL: These files had been staged for exfiltration via the C2 channel. # The SCADA project file (.rss) contains the complete HMI display logic, # tag database, and control setpoints — a blueprint for disruption.
SCADA Project File, OT Network Topology, and Chemical Dosing Schedule Staged for Exfiltration
47 lateral logon events confirmed the implant accessed HMI-1 via null session (unauthenticated SMB). The implant’s staging directory contained the SCADA project file (complete HMI display logic, tag database, and control setpoints), the OT network topology Visio diagram, the chemical dosing schedule, and a 34 MB historian data export. These files represent a complete operational blueprint of the treatment plant. Write access to PLCs or control interfaces was not confirmed — no control commands were observed. The intent was intelligence collection, not disruption. This time.
Finding 5: C2 Infrastructure — VOLTZITE Attribution
Path: C2 IPs: 103.27.xx.xx, 185.176.xx.xx | TLP:AMBER — shared with government partner
- Attribution conducted in coordination with government threat intelligence partner
- Infrastructure overlap with publicly documented VOLTZITE/Volt Typhoon activity cluster
- TTPs consistent with CISA Advisory AA24-038A (Volt Typhoon pre-positioning on US CI)
- OSINT: Shodan, Censys, Passive DNS, VirusTotal graph — do not attribute from IOCs alone
# C2 infrastructure pivot and VOLTZITE TTP overlap mapping # Passive DNS and infrastructure pivot (OSINT — reproducible via SecurityTrails/VirusTotal) # C2 primary: 103.27.xx.xx # ASN: AS132203 (Tencent Cloud / Singapore VPS) <-- common VOLTZITE relay provider # Passive DNS: 3 hostnames resolved to this IP in past 90 days (all NXD now) # update-svc[.]network (registered xx/2024) # svc-telemetry[.]io (registered xx/2024) # cdn-relay[.]net (registered xx/2024) # Beacon traffic SSL certificate (extracted from Zeek ssl.log): # Subject: CN=update-svc.network # Issuer: Let's Encrypt R3 # NotBefore: 2024-xx-xx <-- within 1 week of estimated implant deployment # JA3 fingerprint: 769,47-53-5-10-49195-49199-49196-49200-49161-49162... (Raspberry Pi OpenSSL 1.1.1) # TTP overlap with CISA AA24-038A (Volt Typhoon): # [+] Living off the land — no commercial C2 framework detected # [+] Small-footprint custom implant, statically linked # [+] SOHO/VPS relay infrastructure rotation # [+] Pre-positioning without disruption — intelligence collection mode # [+] Targeting: critical infrastructure, water/energy sector # [-] Hands-on-keyboard activity not confirmed (all traffic appeared automated)
C2 Infrastructure and TTPs Consistent with VOLTZITE/Volt Typhoon Attribution Cluster
Network indicators (ASN patterns, certificate timing, relay infrastructure rotation) and TTPs (small-footprint custom ARM implant, living-off-the-land, OT intelligence collection without disruption) are consistent with publicly documented VOLTZITE/Volt Typhoon activity as described in CISA Advisory AA24-038A. Attribution was conducted in coordination with a government threat intelligence partner and shared under TLP:AMBER with relevant sector agencies. The pre-positioning doctrine — access without disruption, available for activation during a geopolitical flashpoint — matches the stated assessment of Volt Typhoon’s strategic intent.
Reconstructed Intrusion Timeline
| Timestamp | Artifact Source | Event |
|---|---|---|
T-214 days |
Implant filesystem (mactime) | Raspberry Pi implant first connected and configured. update-helper.desktop autostart created. Beacon begins. |
T-214 to T-184 days |
Zeek conn.log (reconstructed) | Initial beacon-only phase. No lateral movement. C2 communications established. |
T-184 to T-30 days |
Zeek conn.log / Historian logs | Lateral movement begins. Implant begins querying PI Historian via port 5450. Process data collection. |
T-30 days |
HMI event log (4624) | Lateral logon to HMI-1. SCADA project file accessed via null session SMB. |
T-30 to T-0 |
HMI event logs | 47 lateral logon events. SCADA project file, network topology, dosing schedule staged in /tmp/.cache/ |
T-30 to T-0 |
Implant staging dir | 34 MB historian export compiled and staged. Total staged: ~38 MB — exfiltrated via C2 over beacon sessions. |
Day 0, hour 6 |
Zeek passive capture | Unknown host beaconing to flagged IP detected. MAC OUI = Raspberry Pi. Physical inspection begins. |
Day 0, hour 8 |
Physical inspection | Implant located — zip-tied to patch panel, painted grey. Photographed in situ, then acquired. |
Day 1 |
Forensic image | SD card imaged (dd). SHA-256 verified. Filesystem analysis begins. |
Day 3 |
Government partner | C2 infrastructure confirmed as VOLTZITE relay. TLP:AMBER notification issued to sector agencies. |
Day 31 onwards |
Ongoing monitoring | OT network re-segmented. Hardware data diodes deployed. Passive monitoring permanent. SCADA upgraded roadmap initiated. |
What This Engagement Teaches Us
For OT/ICS Incident Responders
- Never scan actively in an OT environment without explicit confirmation from plant engineering that it is safe to do so. A SYN scan against an unpatched PLC can cause a process fault. Passive monitoring with SecurityOnion/Zeek is the correct initial approach and will surface anomalies if tuned for the expected OT protocol baseline.
- Physical asset inventory is a forensic prerequisite in OT. An asset register that has never been verified against physical inspection is fiction. Budget one day per site for a physical walk-through — it is the most effective threat hunt in any industrial environment.
- OT Windows hosts are rarely patchable on the same cycle as IT. The compensating control is network segmentation: a Windows XP SCADA host is substantially less dangerous if it cannot be reached from anything except the known HMI workstations. Segment aggressively and document the segments.
- Zeek OUI lookups are your fastest triage for unknown devices:
dc:a6:32is Raspberry Pi,b8:27:ebis older Pi,e4:5f:01is Pi 4. Build an OUI reference into your OT hunting playbooks.
For Critical Infrastructure Operators & Security Engineers
- The CISA advisory on Volt Typhoon pre-positioning (AA24-038A) is not abstract threat intelligence. It describes the exact methodology used in this case: small-footprint implant, living off the land, OT intelligence collection, no disruption. Read it. Then audit your OT network for devices that should not be there.
- A ‘data diode’ that is actually an unmanaged switch provides zero isolation. Hardware unidirectional gateways (Waterfall Security, Owl Cyber Defense, etc.) are purpose-built for IT/OT boundary enforcement. The cost difference between a proper data diode and an incident like this is not a close comparison.
- PI Historian and similar OT data historians must have access controls at the application layer, not just network perimeter controls. Unauthenticated LAN access to a historian is a design flaw — enforce PI trust store certificates and restrict the allowed host list at the PI server configuration level.
- If you are a water, energy, or transportation operator: CISA’s free ICS cybersecurity assessments are available and staffed by practitioners who understand OT environments. The barrier to obtaining expert assessment is lower than most operators realize.
Mjolnir Security — OT/ICS Security & Incident Response
Mjolnir Security provides 24/7 incident response, threat hunting, and security architecture services to critical infrastructure operators. Our DFIR team has responded to nation-state intrusions across water, energy, transportation, and manufacturing sectors.
mjolnirsecurity.com — 24/7 Incident Response Hotline: +1 833 403 5875
Written by Mjolnir Security DFIR team
Published August 2024 · DFIR Engagement Series · TLP:WHITE
Case #331 · Skuggaheimar · Mjolnir Security · All client details anonymized · TLP:WHITE