Generative AI has substantially changed the economics of phishing. Campaigns that previously required skilled social engineers to craft convincing lures now deploy LLM-generated content at scale, producing grammatically polished emails tailored to the target organisation's industry and writing style. Standard heuristics (grammar errors, urgency phrases, suspicious links) perform poorly against this new generation of lures. This report documents the construction and evaluation of an ML classifier specifically designed to detect AI-generated phishing content, achieving F1 = 0.94 on a held-out validation set from 2025-2026 campaigns.
//Dataset Construction
Labelling AI-generated phishing content accurately is non-trivial. Self-reported generation (e.g. inferring from LLM API usage patterns in related C2 infrastructure) is the most reliable signal, but only available for a subset of samples. For the remaining samples, a combination of statistical text analysis and known LLM output patterns was used to generate candidate labels, followed by manual review of a 10% stratified sample.
AI-Generated Sample Sources
AI-generated samples were collected from three sources: phishing kits recovered during incident response that contained LLM prompt templates and batch generation scripts; VirusTotal submissions where the associated campaign infrastructure showed API calls to commercial LLM endpoints; and underground forum offerings selling "AI phishing-as-a-service" with sample outputs that were verified as AI-generated via stylometric analysis.
# Example prompt template recovered from phishing kit # (redacted to remove targeting specifics) SYSTEM: You are a professional business communications writer. Generate a convincing email from [SPOOFED_SENDER] to [TARGET_ORG] employees. The email should: - Request urgent action on [LURE_TYPE] (invoice/IT/HR) - Include plausible internal references to [ORG_CONTEXT] - Have a natural, non-suspicious tone - End with a call to action linking to [PHISHING_URL] OUTPUT FORMAT: Subject line on first line, email body below.
//Feature Engineering
Three feature categories are used. Linguistic features capture statistical patterns in word choice and sentence structure. Structural features capture email formatting patterns. Metadata features capture signals from headers and sending infrastructure.
Linguistic Features
AI-generated phishing exhibits measurable differences from human-written phishing in several statistical dimensions. Perplexity under a reference language model (GPT-2 medium) is lower for AI-generated text - it is "too grammatical". Vocabulary diversity (type-token ratio) is higher. Sentence length variance is lower (AI tends toward consistent sentence length). These features are combined with TF-IDF weighted unigram/bigram features extracted from the email subject and body.
# Feature extraction (simplified)
import numpy as np
from transformers import GPT2LMHeadModel, GPT2Tokenizer
def compute_perplexity(text, model, tokenizer, max_len=512):
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=max_len)
with torch.no_grad():
loss = model(**inputs, labels=inputs["input_ids"]).loss
return float(torch.exp(loss))
def type_token_ratio(text):
tokens = text.lower().split()
return len(set(tokens)) / max(len(tokens), 1)
def sentence_length_variance(text):
import re
sentences = re.split(r'[.!?]+', text)
lengths = [len(s.split()) for s in sentences if s.strip()]
return np.var(lengths) if lengths else 0.0Structural Features
AI-generated phishing emails show distinct structural patterns: they tend to have more consistent paragraph lengths, more frequent use of bullet-point-style formatting (even in plain text), and less variation in salutation/closing patterns. HTML phishing emails generated by AI show higher template regularity scores when compared against a corpus of known clean email HTML.
//Model Architecture
The final classifier is a gradient boosted ensemble (XGBoost) trained on the combined feature set. A fine-tuned DeBERTa-v3-base model was evaluated as an alternative but showed only marginal accuracy improvements at 40x the inference cost, making it unsuitable for real-time gateway deployment. The XGBoost model runs in under 2ms per email on CPU, suitable for inline SMTP gateway integration.
# Model training (simplified)
import xgboost as xgb
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import f1_score
features = np.hstack([
linguistic_features, # perplexity, TTR, sentence var, TF-IDF (512 dims)
structural_features, # para length dist, bullet score, template regularity (8 dims)
metadata_features, # SPF/DKIM align, domain age, ASN reputation (12 dims)
])
model = xgb.XGBClassifier(
n_estimators=400,
max_depth=6,
learning_rate=0.05,
subsample=0.8,
colsample_bytree=0.8,
use_label_encoder=False,
eval_metric="logloss",
)
# 5-fold cross-validation
cv_scores = cross_val_score(model, features, labels, cv=StratifiedKFold(5), scoring="f1")
print(f"CV F1: {cv_scores.mean():.3f} +/- {cv_scores.std():.3f}")
# Result: CV F1: 0.941 +/- 0.008//Evaluation Results
Validation Set Results (n=8,400): Precision: 0.946 Recall: 0.934 F1: 0.940 AUC-ROC: 0.978 False Positive Analysis: FP rate on legitimate marketing email: 3.2% FP rate on security awareness test emails: 7.8% (Security awareness test emails mimic phishing and trigger AI detection) Ablation Study (feature group removal): - Without linguistic features: F1 = 0.831 - Without structural features: F1 = 0.912 - Without metadata features: F1 = 0.907 - All features: F1 = 0.940
//Deployment Considerations
At SMTP gateway deployment, the classifier runs on the email body after attachment stripping. A confidence threshold of 0.75 (rather than 0.5) is recommended to reduce false positives at the cost of slightly lower recall - this is appropriate for a quarantine workflow where false positives cause user friction. For a monitoring-only deployment (log and alert, no quarantine), 0.5 is acceptable.