Detecting AI-Generated Phishing at Scale

Signature-based email security has quietly become obsolete. For the past two decades, phishing detection engines relied on a fairly stable assumption: attackers write badly. Broken grammar, inconsistent tone, and templated boilerplate gave Bayesian filters and regex-based heuristics something reliable to grip onto. Large language models have removed that grip entirely. A modern threat actor no longer hand-crafts a spear-phishing email; they prompt a model with scraped LinkedIn data, internal org charts leaked from a previous breach, and a target’s writing style, and receive a grammatically flawless, contextually accurate message in under a second. This is the core architectural challenge behind AI-generated phishing detection: the linguistic and structural tells that legacy filters depend on have been engineered out of the attack surface.

This article breaks down a production-grade detection pipeline capable of flagging AI-generated phishing content and the fake infrastructure (domains, TLS certificates, cloned brand assets) that supports it. It is not a checklist for end users spotting typos — it is the systems architecture required to catch attacks that no longer contain typos.

#The Problem: Generative Models Have Broken Static Signature Detection

Traditional secure email gateways (SEGs) score messages against three primary vectors: known-bad sender reputation, keyword/regex matching, and structural anomalies (mismatched display names, spoofed headers). Generative AI defeats all three simultaneously:

• Sender reputation decay — attackers now spin up disposable, freshly registered domains with valid DKIM/SPF/DMARC alignment via legitimate cloud mail providers, so the infrastructure itself looks clean at time of send.
• Keyword matching failure — LLM output avoids the stilted phrasing (‘Dear Valued Customer, kindly verify’) that regex libraries were tuned against.
• Structural anomaly masking — AI-assisted domain generation now produces homoglyph and combosquat domains that pass basic string-distance checks (Levenshtein thresholds tuned for human typo patterns, not algorithmic generation).

The result is a detection gap where the payload (the malicious link, the credential harvesting form) is unchanged, but the delivery wrapper is statistically indistinguishable from legitimate correspondence using first-generation heuristics. Effective AI-generated phishing detection requires shifting the detection surface away from ‘is this text bad’ toward ‘is this text statistically anomalous relative to human baseline distributions’, combined with independent verification of the infrastructure serving the link.

#Architectural Breakdown of an AI-Generated Phishing Detection Pipeline

A robust pipeline needs four independent, loosely-coupled analysis layers feeding a single ensemble scoring engine. No single layer should be authoritative — this mirrors defence-in-depth architectural patterns used elsewhere in security tooling, where a single compromised signal (e.g., a spoofed sender domain) cannot alone clear or condemn a message.

#Layer 1: Mail Transfer Agent Ingestion and Header Forensics

Every inbound message is intercepted at the MTA via a milter (Postfix) or transport rule (Exchange Online) before reaching the mailbox. At this stage we extract Received-SPF, DKIM-Signature validity, and DMARC alignment, but critically we also pull Authentication-Results arc chains to detect relay hops that break DMARC inheritance — a common technique when phishing infrastructure is fronted through legitimate transactional email services (SendGrid, Mailgun) to inherit their sender reputation.

#Layer 2: Linguistic Feature Extraction — Perplexity and Burstiness Scoring

This is the layer that specifically targets generative text. LLM output exhibits measurably lower perplexity (a language model’s confidence in predicting the next token) than human writing, and near-uniform burstiness (variance in sentence length and structure) compared to natural human prose, which fluctuates. We run body text through a lightweight local scoring model (GPT-2 medium is sufficient — no need for an expensive frontier model) purely to compute perplexity, not to generate anything.

1from transformers import GPT2LMHeadModel, GPT2TokenizerFast
2import torch
3
4model_id = "gpt2-medium"
5tokenizer = GPT2TokenizerFast.from_pretrained(model_id)
6model = GPT2LMHeadModel.from_pretrained(model_id).eval()
7
8def perplexity_score(text: str) -> float:
9    encodings = tokenizer(text, return_tensors="pt")
10    with torch.no_grad():
11        outputs = model(**encodings, labels=encodings["input_ids"])
12    return torch.exp(outputs.loss).item()
13
14# Baseline: human corporate email corpus averages 45-90 perplexity.
15# AI-generated phishing typically scores 15-30 (unnaturally low variance).
16sample = "Your account requires immediate verification to avoid suspension."
17score = perplexity_score(sample)
18if score < 30:
19    flag_reason = "low_perplexity_ai_suspect"

This is scored alongside burstiness (standard deviation of sentence-length tokens) because perplexity alone produces false positives on legal boilerplate and templated marketing, which is also low-perplexity by design.

#Layer 3: URL and Domain Entropy Analysis

AI-generated phishing kits programmatically produce domain permutations at a rate no human typosquatter can match — often hundreds per campaign, registered within minutes of each other via automated registrar APIs. Shannon entropy calculated over the domain label, combined with a Unicode confusables check, catches both algorithmically generated random subdomains and homoglyph substitution (Cyrillic ‘а’ replacing Latin ‘a’).

1import math
2import unicodedata
3from collections import Counter
4
5def shannon_entropy(label: str) -> float:
6    counts = Counter(label)
7    length = len(label)
8    return -sum((c / length) * math.log2(c / length) for c in counts.values())
9
10def has_confusable_chars(domain: str) -> bool:
11    return any(unicodedata.name(ch, "").find("LATIN") == -1
12               and ch.isalpha() for ch in domain)
13
14domain = "secure-login-paypa1-verify.com"
15label = domain.split(".")[0]
16
17entropy = shannon_entropy(label)         # Human-registered brand domains: 2.8-3.4
18confusable = has_confusable_chars(domain) # Random algorithmic strings: >4.0
19
20if entropy > 3.8 or confusable:
21    flag_reason = "high_entropy_or_homoglyph_domain"

Domain age (via WHOIS/RDAP) and certificate transparency log lookups are cross-referenced here too — a domain registered less than 72 hours ago with a freshly issued DV certificate is a strong independent signal regardless of text content.

#Layer 4: Visual Brand Impersonation via Computer Vision

Because AI-generated phishing pages now clone corporate login screens pixel-for-pixel (scraped and reconstructed via vision models), text-based detection on the landing page is close to useless. The link is detonated in an isolated headless browser, a screenshot is taken, and the render is embedded via a CLIP-style vision model, then compared against a vector store of known brand login page embeddings using cosine similarity.

1npx playwright screenshot n  --browser=chromium n  --isolated n  --viewport-size=1280,800 n  --wait-for-selector="body" n  "https://secure-login-paypa1-verify.com" n  /sandbox/render_$(uuidgen).png

A cosine similarity above 0.92 against a known brand embedding, on a domain that is not the brand’s registered infrastructure, is treated as a near-certain impersonation event and triggers automatic blocklisting upstream at the DNS resolver / secure web gateway layer.

#Implementation Logic: Wiring the Pipeline Together

The four layers above are independent workers behind a message queue (Kafka or SQS), feeding a scoring service. Each worker publishes a normalised feature vector; the scoring service consumes all four asynchronously with a timeout budget, since visual detonation is the slowest path (typically 800ms–2.5s per URL) and should never block delivery of low-risk internal mail.

1. MTA milter tags the message with a correlation ID and pushes header/body metadata to the ingestion topic.
2. Linguistic and URL-entropy workers consume immediately (sub-100ms latency budget) and return synchronously where possible.
3. Any URL scoring above a low-confidence threshold (e.g., entropy > 3.2 OR perplexity < 35) is queued for asynchronous visual detonation rather than blocking the mail flow.
4. The ensemble scorer applies a weighted logistic model; weights are retrained monthly against labelled incident data to counter model drift.
5. Verdicts above the quarantine threshold trigger a SOAR playbook: mailbox purge, sender domain blocklist push, and a ticket into the SIEM correlating against MITRE ATT&CK T1566 (Phishing).

A minimal ensemble configuration looks like this:

1detection_pipeline:
2  ensemble:
3    weights:
4      header_forensics: 0.20
5      linguistic_perplexity: 0.25
6      domain_entropy: 0.25
7      visual_similarity: 0.30
8    thresholds:
9      quarantine: 0.75
10      soft_flag: 0.45
11  detonation:
12    max_concurrent_sessions: 40
13    sandbox_network_egress: allowlist
14    dns_allowlist:
15      - "*.internal.corp"
16    timeout_ms: 4000
17  retrain_cadence_days: 30

#Failure Modes and Edge Cases in AI-Generated Phishing Detection

No layer here is unbreakable, and attackers adapt within weeks of detection methods becoming public knowledge. Known failure modes include:

• Perplexity evasion via noise injection — attackers deliberately prompt models to introduce mild grammatical irregularities, artificially raising perplexity to mimic human variance. This is already observed in the wild and reduces linguistic-layer accuracy by roughly 15-20%.
• Sandbox fingerprinting — malicious pages detect headless browser signatures (missing WebGL renderer, absent mouse-jitter, automation flags in navigator.webdriver) and serve a benign placeholder instead of the credential harvester, defeating visual detonation entirely.
• False positives on legitimate AI copywriting — marketing platforms increasingly use LLMs for outbound campaigns, meaning low perplexity alone is not sufficient; the linguistic layer must always be weighted below 0.3 in the ensemble to avoid quarantining legitimate business mail.
• Certificate transparency lag — some CAs batch CT log submissions, creating a window of several minutes where a malicious domain has a valid certificate but no CT record yet, defeating age-based heuristics.
• Visual model staleness — brand login pages are redesigned periodically; if the embedding vector store isn’t refreshed, similarity scoring drifts and both false negatives (missed clones of the new design) and false positives (flagging the brand’s own updated real page) increase.

#Scaling and Security Trade-offs

Deploying this architecture at enterprise mail volume (100k+ messages/hour) forces explicit trade-off decisions rather than a single ‘correct’ configuration:

• Synchronous vs asynchronous detonation — blocking delivery on visual analysis guarantees zero exposure window but adds 1-3 seconds of latency per external link; asynchronous detonation with post-delivery retraction is faster but tolerates a short exposure window before quarantine fires.
• GPU cost vs detection depth — CLIP-based visual similarity scoring is meaningfully more accurate than perceptual hashing (pHash) but requires GPU inference nodes; at scale this is the single largest cost line in the pipeline, often justifying a hybrid approach where pHash pre-filters obvious negatives before GPU inference runs on the remainder.
• Model retraining cadence vs drift risk — retraining the ensemble weekly reduces drift but risks overfitting to recent, narrow attack campaigns; monthly retraining with a rolling 90-day labelled dataset is the more stable production default.
• Sandbox isolation strictness vs coverage — a fully egress-restricted detonation sandbox is safer against callback-based malware but may fail to trigger multi-stage payloads that require legitimate DNS resolution, reducing detection fidelity for advanced kits.
• Centralised vs federated brand embedding stores — a single shared vector store across tenants improves detection coverage for widely-cloned brands but introduces a data-residency and multi-tenancy isolation concern in regulated environments (finance, healthcare).

None of these trade-offs are resolved by adding more compute; they are governance decisions that determine the acceptable latency, cost, and residual-risk profile for a given organisation’s threat model. The pipeline described here is the current practical ceiling for AI-generated phishing detection at the network edge — it does not eliminate risk, it compresses the exposure window and shifts the cost of evasion back onto the attacker, forcing continual, expensive adaptation on their side rather than yours.