The Rate Limiter Was on the Wrong Field

The following is a lightly anonymised account of a real DFIR engagement conducted under privilege. All patient identifiers have been irreversibly removed. Regulatory filings referenced are generalised. Technical findings are presented as they occurred. We are publishing this case because credential stuffing against healthcare portals is a persistent and underreported threat, and because the specific failure mode — a rate limiter deployed on the wrong field — is one we have seen in four separate engagements.

The Engagement

How We Were Called In

The network’s security operations analyst noticed the anomaly on a Tuesday morning while reviewing the previous night’s web application firewall logs. The patient portal — a third-party product that allowed patients to view lab results, book appointments, and message their care teams — had processed 2.3 million authentication requests between 11 PM Monday and 6 AM Tuesday. Normal overnight volume was approximately 4,000 requests.

She had paged the security manager at 7:14 AM. By the time we were engaged at 9:00 AM, the attack had stopped — the credential stuffing campaign had apparently completed its run — and the question was no longer how to stop it but how many accounts had been successfully accessed and what had been viewed.

The answer, extracted over the following 72 hours, was 14,247 accounts. Each of those accounts contained the patient’s full name, date of birth, provincial health number, address, medical record number, primary care provider, appointment history, and lab result summaries. This is, under the Personal Health Information Protection Act, a notifiable privacy breach.

Finding 1: Attack Characterisation — Volume, Velocity, and Source

The WAF had flagged nothing. The application had logged everything. These two facts together told the story of the rate limiter failure before we had written a single query.

# Apache access log analysis — 2.3M auth requests over 6 hours, 14,247 successes, 47,218 source IPs
grep 'POST /api/v2/auth/login' /var/log/apache2/access.log.1 | \
  awk '{print $4}' | cut -d: -f1-3 | sort | uniq -c | sort -rn | head -24

# Requests per hour (condensed — attack window highlighted):
# [23:00-23:59]  1,847     ← pre-attack baseline
# [00:00-00:59]  214,388   ← attack begins
# [01:00-01:59]  298,441   ← peak volume
# [02:00-02:59]  287,119
# [03:00-03:59]  271,884
# ... [sustained through 05:59]
# [06:00-06:59]  44,221    ← attack winds down
# Total attack window: 2,287,441 auth requests over 6 hours

# Success vs failure rate:
grep 'POST /api/v2/auth/login' /var/log/apache2/access.log.1 | \
  awk '{print $9}' | sort | uniq -c
# HTTP 200 (success):  14,247   (0.62%)
# HTTP 401 (fail):  2,272,841   (99.38%)
# HTTP 429 (rate limited): 353  (0.015%) ← almost never triggered

# Unique source IPs:
grep 'POST /api/v2/auth/login' /var/log/apache2/access.log.1 | awk '{print $1}' | sort -u | wc -l
# 47,218 unique source IPs — residential proxy network
# Average requests per IP: 48.4 — low enough to evade per-IP rate limits

The 47,218 unique source IPs were the signature of a residential proxy network — a botnet of compromised home routers and IoT devices rented by the hour for credential stuffing campaigns. Each IP sent an average of 48 requests over 6 hours. Per-IP rate limiting would have required a threshold of approximately 8 requests per IP per hour to catch the majority of attack traffic — far below what any reasonable rate limiter targeting legitimate users would set.

Finding 2: The Rate Limiter Failure — Wrong Field, Wrong Axis

The rate limiter configuration was in a YAML file on the application server. The relevant section was four lines long. Understanding exactly why it had failed took approximately thirty seconds to read.

# Portal application rate limiter configuration (/etc/portal/config.yaml)

auth:
  rate_limit:
    enabled: true
    strategy: per_username          # ← rate limit key is the submitted username
    max_attempts: 10                # 10 failed attempts before lockout
    window_seconds: 900             # 15-minute window
    lockout_duration_seconds: 1800  # 30-minute lockout after threshold

# What this does: track failed attempts PER USERNAME.
# If username 'john.smith@example.com' fails 10 times in 15 minutes → locked out.
# These are completely independent buckets.

# What credential stuffing does: each credential pair uses a DIFFERENT username.
# patient001@gmail.com: tried once, fails → 1/10 attempts in bucket, no lockout
# patient002@gmail.com: tried once, fails → 1/10 attempts in bucket, no lockout
# patient003@gmail.com: tried once, fails → 1/10 attempts in bucket, no lockout
# ... × 2,287,441

# From the rate limiter's perspective: zero unusual activity.
# Every username is tried exactly once. No bucket ever reaches 10.
# The rate limiter is working perfectly. It is simply irrelevant.

# Correct configuration for stuffing prevention:
auth:
  rate_limit:
    enabled: true
    strategy: multi_key             # rate limit on MULTIPLE axes simultaneously
    keys:
      - field: ip_address           # per-IP: catches low-volume distributed attacks
        max_attempts: 30
        window_seconds: 3600
      - field: device_fingerprint   # per-device: catches IP-rotating attacks
        max_attempts: 20
        window_seconds: 3600
      - field: username             # per-username: catches password spray
        max_attempts: 10
        window_seconds: 900
    global_rps_threshold: 500       # global requests/second ceiling

Finding 3: What the Attacker Accessed — PHI Exposure Scope

The 14,247 successful authentications were not all equivalent in terms of exposure. Not every attacker session lasted long enough to access protected information — some sessions appeared to be automated validation checks (login, confirm success, log out in under 2 seconds). Others had extensive navigation through the portal’s pages.

# Session depth analysis — 8,241 validation-only, 4,422 dashboard, 1,584 deep PHI access

import re, collections

# Load all session IDs from successful auth requests during attack window
success_sessions = set()
with open('/var/log/apache2/access.log.1') as f:
    for line in f:
        if 'POST /api/v2/auth/login' in line and ' 200 ' in line:
            m = re.search(r'session=([a-f0-9]{32})', line)
            if m: success_sessions.add(m.group(1))

# Categorise sessions by page access
session_pages = collections.defaultdict(list)
with open('/var/log/portal/session.log') as f:
    for line in f:
        parts = line.strip().split('|')
        if len(parts) >= 5:
            sid, resource = parts[1].strip(), parts[4].strip()
            if sid in success_sessions:
                session_pages[sid].append(resource)

# Categorise by depth of access:
bounce  = sum(1 for s,p in session_pages.items() if len(p) <= 2)
shallow = sum(1 for s,p in session_pages.items() if 3 <= len(p) <= 6)
deep    = sum(1 for s,p in session_pages.items() if len(p) > 6)

# Output:
# Bounce (validation only):  8,241  (57.9% — automated validation, no PHI accessed)
# Shallow (dashboard):       4,422  (31.0% — name/DOB/provider visible on dashboard)
# Deep (PHI accessed):       1,584  (11.1% — lab results, appointment history, messages)

The 1,584 ‘deep access’ sessions were the most critical — these sessions had navigated to lab results pages, appointment history, or the secure messaging centre. The 4,422 ‘shallow’ sessions had accessed the portal dashboard — which displayed full name, date of birth, provincial health number, primary provider, and next appointment. Under PHIPA this is still a notifiable disclosure. The 8,241 ‘bounce’ sessions had authenticated, received a 200 response confirming the credentials were valid, and immediately terminated — likely automated validation to identify live credential pairs for future use or sale.

Finding 4: Source of the Credential List — Breach Data Correlation

# HIBP batch query — 37% of compromised accounts appeared in LinkedIn 2012 breach

import requests, time, collections

API_KEY = '████████████████████████████████'
HEADERS = {'hibp-api-key': API_KEY, 'User-Agent': 'MjolnirSecurityIR/1.0'}

with open('compromised_emails_sample.txt') as f:
    emails = [line.strip() for line in f][:500]

breach_counts = collections.Counter()
for email in emails:
    r = requests.get(f'https://haveibeenpwned.com/api/v3/breachedaccount/{email}',
                     headers=HEADERS)
    if r.status_code == 200:
        for b in r.json():
            breach_counts[b['Name']] += 1
    time.sleep(1.6)

print('Top breach sources in compromised account sample:')
for breach, count in breach_counts.most_common(10):
    print(f'  {breach:<30} {count} accounts ({count/5:.1f}%)')

# Output:
# LinkedIn2012               187 accounts (37.4%)
# Collection1                143 accounts (28.6%)
# Adobe                       89 accounts (17.8%)
# ████████ (major 2023 breach) 61 accounts (12.2%)
# Other                        20 accounts (4.0%)

# Average password age for compromised accounts: 847 days (2.3 years)

The median password age for compromised accounts was 847 days — 2.3 years. The most common breach source was the 2012 LinkedIn data breach, still circulating in ComboList databases twelve years after the fact. Patients who had created their portal account in the early days of the portal’s launch, used their standard email password, and never changed it were the most vulnerable. This is not a patient failure. It is an absence of password hygiene enforcement on the portal’s part.

Finding 5: Detection — What the Logs Said and Who Wasn’t Watching

The SIEM had fired. The alert had been sent to a distribution list at 01:22 AM. The distribution list included five people. None of them had configured on-call escalation. The alert subject was ‘PORTAL: Authentication failure volume elevated.’ It joined 847 other unread alerts in the shared inbox. The attack completed its run at approximately 06:00 AM. The SOC analyst discovered it during morning log review at 07:14 AM.

# Recommended detection — 5-minute SIEM rule with PagerDuty escalation
# With this rule: alert would have fired at 00:02 AM (2 minutes into attack)

index=web_access sourcetype=apache_access uri='/api/v2/auth/login' status=401
| bucket _time span=5m
| stats count AS failures by _time
| where failures > 200           # 200 failures in 5 minutes = anomaly (normal ~15/5min)
| eval severity=case(
    failures > 2000, 'CRITICAL',
    failures > 500,  'HIGH',
    failures > 200,  'MEDIUM'
  )
| sendalert action.pagerduty     # PagerDuty escalation, not email

# Additional: success rate anomaly from proxy IPs
index=web_access sourcetype=apache_access uri='/api/v2/auth/login' status=200
| iplocation clientip
| lookup asn_reputation_list ASN OUTPUT reputation
| where reputation='residential_proxy' OR reputation='datacenter_anon'
| stats count AS successes, dc(clientip) AS unique_ips by _time span=5m
| where successes > 50 AND unique_ips > 20
| sendalert action.pagerduty

Reconstructed Attack Timeline

What This Engagement Teaches Us

For Incident Responders

1. The first question in any credential stuffing investigation is ‘what was the rate limiter strategy?’ Credential stuffing (one attempt per account from many IPs) and password spraying (many attempts on a few accounts) require opposite rate limiting strategies. Per-username limiting stops spraying. Per-IP limiting and global volume thresholds stop stuffing. Understand which attack you’re dealing with before assessing whether the defences were adequate.
2. Session depth analysis is the critical step between ‘how many accounts were compromised’ and ‘what PHI was exposed.’ Build a categorisation from session logs (bounce / shallow / deep) and cross-reference with database audit logs where available. The notification content and regulatory posture differ significantly between ‘credentials validated’ and ‘lab results viewed.’
3. HIBP k-anonymity API enables you to identify which breach databases supplied the credential list without knowing the actual passwords. This context belongs in your breach notification — it tells patients that the attacker already had their password from a prior breach and informs the remediation advice.
4. For healthcare breaches: PHIPA (Ontario), PIPEDA, and their provincial equivalents have specific notification timelines — typically 72 hours to the regulator, then patient notification within a reasonable period. Know the applicable regulation before the incident.

For Healthcare IT & Application Security Engineers

1. Implement multi-axis rate limiting on every authentication endpoint: per-IP (catches most distributed attacks), per-username (catches spray), and global request-per-second ceiling (catches volume-based attacks). The per-username strategy alone — the most common configuration — catches password spraying but is completely irrelevant for credential stuffing.
2. Integrate Have I Been Pwned’s k-anonymity password checking at login and at registration. At login: if the submitted password appears in HIBP’s breach database, force a password reset before granting access. At registration: reject any password that appears in HIBP. The API call adds approximately 2ms to the login time and eliminates the primary vulnerability exploited in credential stuffing.
3. License and deploy your WAF vendor’s bot management module. Standard WAF rules (per-IP thresholds, signature matching) are not designed to detect distributed credential stuffing from residential proxy networks. Bot management uses behavioral analysis, device fingerprinting, and machine learning to detect distributed low-volume attacks.
4. Route all security alerts to an on-call system with escalation. The SIEM alert in this case fired 82 minutes into a 6-hour attack. An on-call page at 01:22 AM could have limited the breach to approximately 2,800 accounts. The email-to-shared-alias routing added 5.5 hours of attack runtime — approximately 11,000 additional patient records compromised.