Playbooks · Coding tasks
Fingerprinting AI coding agents from 33,580 PRs
Written November 6, 2026
A study fingerprints AI coding agents from GitHub PRs. What the method can and cannot prove about who — or what — wrote the code.
When a developer uses an AI agent to write code and submits the pull request under their own account, the PR looks human. A study from Trent University asks whether the agent still leaves a trace — a behavioural fingerprint in commit messages, PR structure, and code — and finds that it does, well enough for a classifier to name the agent. The result is useful for maintainers and researchers. It is not proof of authorship, and the paper says so in its own limits.
What the study did
The paper, Fingerprinting AI Coding Agents on GitHub by Taher A. Ghaleb, analyzes 33,580 pull requests from five agents — OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code — drawn from the AIDev-pop dataset. The mix is lopsided: Codex accounts for 21,793 PRs (64.9%), Copilot 4,967 (14.8%), Devin 4,822 (14.4%), Cursor 1,540 (4.6%), and Claude Code just 458 (1.4%).
From each PR the study extracts 53 features across five groups — commit message patterns, PR structure, code changes, patch-level code characteristics, and timing — then trims them to 41 by dropping features that duplicate each other. Two tree-based classifiers, XGBoost and Random Forest, are trained with 5-fold stratified cross-validation and scored only on held-out folds. XGBoost wins with a 97.2% macro F1-score, against 94.9% for Random Forest.
What the fingerprints look like
The strongest signals are not in the code. They are in how the change is communicated. Globally, the multiline commit-message ratio carries 44.7% of the feature importance, followed by change concentration at 10.1% and average commit message length at 2.3%. Agents write similar code but differ in version-control habits.
| Agent | Strongest fingerprint | Per-agent accuracy |
|---|---|---|
| OpenAI Codex | Multiline commit messages (67.5%) | 99% precision, 99% recall |
| GitHub Copilot | Longer PR descriptions (38.4%), concentrated changes (24.9%) | 99% precision, 98% recall |
| Devin | Multiline commits (48.9%), changes spread across files (8.2%) | 93% precision, 96% recall |
| Cursor | Bullet points (17.2%) and hyperlinks (12.8%) in PR bodies | 88% precision, 83% recall |
| Claude Code | Conditional density (27.2%), comment density (19.8%) | 82% precision, 57% recall |
Claude Code is the weak spot. With only 458 samples, the classifier spots most Claude Code PRs only 57% of the time — predictions of Claude Code are usually right, but many of its PRs get classified as Devin (137 cases) or Codex (28). The headline 97.2% is a macro average across all five agents; the smallest class is much harder to catch.
What it cannot prove
A fingerprint is a pattern match, not proof of authorship. The paper's own validity section sets the limits:
- Fingerprints drift. The dataset is a single snapshot. Agent developers can change commit formats, PR templates, and code style at any time, intentionally or not, and the study cannot detect that drift. A fingerprint that works today needs retraining as agents evolve.
- It was tested on public GitHub only. Private repositories and other platforms such as GitLab were not tested, and agent behaviour there may differ under stricter style guides.
- Adversaries can mimic humans. An agent instructed to vary its commit structure and PR format would be harder to fingerprint. The paper notes this would take deliberate effort and detailed knowledge of which features matter.
- The "human" baseline is contaminated. The AIDev dataset labels 6,618 PRs as human, but developers may have used agents to write them. Any study that treats those labels as ground truth inherits the noise.
For a maintainer, the honest use is triage: flag a PR as likely agent-written and review it more closely, or use the signatures as interpretable cues alongside other evidence. A fingerprint alone should never be presented as proof that a specific person did or did not write the code — and it should never be the sole basis for accusing a contributor of undisclosed agent use.
What a maintainer can do with it
Run the classifier's idea, not its verdict: check whether PRs in your repository carry the signatures — long multiline commits, bullet-heavy descriptions, dense comments — and treat a cluster of them as a prompt to ask about AI use, not as a finding. If your project requires disclosure of AI-generated code, a fingerprint can make that policy enforceable by flagging PRs for review before merge. And if you build datasets for research, apply fingerprinting to check your labels first: the paper's warning is that "human" datasets assembled after agents became common are probably already mixed.
Limits
- This page is written from the paper. No PRs were classified here, and no fingerprint was tested against a live repository.
- The 97.2% figure is the paper's result on its own dataset, with its own class mix. It is not a guarantee for your repositories.
- Claude Code's 57% recall is the number that matters most for small classes: a fingerprint that misses nearly half of one agent's PRs is a screening aid, not a detector.
Related playbooks
- Fifty agents built a bureaucracy; four beat the game
- Check where your coding agents store API keys on your own machine
- GitAgent: gated commit messages and PRs, locally
- Letterpress-1: a growth agent that files PRs
Sources
- Fingerprinting AI Coding Agents on GitHub — arXiv: abstract, method, results, and validity threats (Taher A. Ghaleb, Trent University)
- Run your own agent bake-off on one real task — Agent Upstart: comparing agents on the same task, where measurement discipline matters
- Review a pull request with a coding agent — Agent Upstart: reviewing PRs, where a fingerprint would surface
Back to: Coding playbooks hub · Also: Run your own agent bake-off on one real task · Review a pull request with a coding agent · 948 Linux exploits: what the pipeline actually did