Agent Upstart

Cool things people are building with agents — and practical help getting the most out of them.

Playbooks · Coding tasks

Fingerprinting AI coding agents from 33,580 PRs

Written November 6, 2026

A study fingerprints AI coding agents from GitHub PRs. What the method can and cannot prove about who — or what — wrote the code.

When a developer uses an AI agent to write code and submits the pull request under their own account, the PR looks human. A study from Trent University asks whether the agent still leaves a trace — a behavioural fingerprint in commit messages, PR structure, and code — and finds that it does, well enough for a classifier to name the agent. The result is useful for maintainers and researchers. It is not proof of authorship, and the paper says so in its own limits.

What the study did

The paper, Fingerprinting AI Coding Agents on GitHub by Taher A. Ghaleb, analyzes 33,580 pull requests from five agents — OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code — drawn from the AIDev-pop dataset. The mix is lopsided: Codex accounts for 21,793 PRs (64.9%), Copilot 4,967 (14.8%), Devin 4,822 (14.4%), Cursor 1,540 (4.6%), and Claude Code just 458 (1.4%).

From each PR the study extracts 53 features across five groups — commit message patterns, PR structure, code changes, patch-level code characteristics, and timing — then trims them to 41 by dropping features that duplicate each other. Two tree-based classifiers, XGBoost and Random Forest, are trained with 5-fold stratified cross-validation and scored only on held-out folds. XGBoost wins with a 97.2% macro F1-score, against 94.9% for Random Forest.

What the fingerprints look like

The strongest signals are not in the code. They are in how the change is communicated. Globally, the multiline commit-message ratio carries 44.7% of the feature importance, followed by change concentration at 10.1% and average commit message length at 2.3%. Agents write similar code but differ in version-control habits.

AgentStrongest fingerprintPer-agent accuracy
OpenAI CodexMultiline commit messages (67.5%)99% precision, 99% recall
GitHub CopilotLonger PR descriptions (38.4%), concentrated changes (24.9%)99% precision, 98% recall
DevinMultiline commits (48.9%), changes spread across files (8.2%)93% precision, 96% recall
CursorBullet points (17.2%) and hyperlinks (12.8%) in PR bodies88% precision, 83% recall
Claude CodeConditional density (27.2%), comment density (19.8%)82% precision, 57% recall

Claude Code is the weak spot. With only 458 samples, the classifier spots most Claude Code PRs only 57% of the time — predictions of Claude Code are usually right, but many of its PRs get classified as Devin (137 cases) or Codex (28). The headline 97.2% is a macro average across all five agents; the smallest class is much harder to catch.

What it cannot prove

A fingerprint is a pattern match, not proof of authorship. The paper's own validity section sets the limits:

For a maintainer, the honest use is triage: flag a PR as likely agent-written and review it more closely, or use the signatures as interpretable cues alongside other evidence. A fingerprint alone should never be presented as proof that a specific person did or did not write the code — and it should never be the sole basis for accusing a contributor of undisclosed agent use.

What a maintainer can do with it

Run the classifier's idea, not its verdict: check whether PRs in your repository carry the signatures — long multiline commits, bullet-heavy descriptions, dense comments — and treat a cluster of them as a prompt to ask about AI use, not as a finding. If your project requires disclosure of AI-generated code, a fingerprint can make that policy enforceable by flagging PRs for review before merge. And if you build datasets for research, apply fingerprinting to check your labels first: the paper's warning is that "human" datasets assembled after agents became common are probably already mixed.

Limits

Related playbooks

Sources

Back to: Coding playbooks hub · Also: Run your own agent bake-off on one real task · Review a pull request with a coding agent · 948 Linux exploits: what the pipeline actually did