GoKawiilA rule like "run the full test suite before committing" looks simple until an AI coding agent edits a source file after the last test run and then calls git commit . The kernel sees an ordinary process writing a commit object, while the harness sees one more tool call, yet the decision depends on which test result is still fresh, which edit invalidated it, and whether this commit is allowed now.
The ActPlane paper measures the gap between the behavioral rules developers write and the subset a system can actually check. Its statement-level analysis of 2,116 instructions shows that developers are not short of rules; the difficulty lies in turning natural-language requirements into state that a system can observe and evaluate over time. Many rules concern files, processes, or network activity but still depend on repository structure, task progress, or prior events, so a single OS hook can cover only part of the policy set.
Developers Have Already Written the Policies
Most discussions of AI agent safety start from threat models or attack surfaces. ActPlane starts from a different question: what do developers already tell their agents to do and not do, and what would it take to enforce those instructions?
The study examines 64 popular repositories containing CLAUDE.md and AGENTS.md files (median 20K GitHub stars, snapshot from 2026-05-23), covering 84 instruction files and 2,116 individual statements. Unlike prior work that analyzed instruction files at the file or section-heading level, ActPlane classifies every statement independently. The study asks three questions: are instruction files primarily behavioral policies or descriptive context? Which policies require OS-level enforcement, and what kinds of OS-level checks do they need? What context is needed to instantiate these policies into concrete, enforceable rules?
Statements were extracted through a two-pass LLM agent-assisted pipeline that recorded source line ranges and four labels per statement: content type, topic, enforcement level, and context requirement. A validation script verified full source coverage and verbatim span matching, then two independent agents (Claude and Codex) cross-checked the results. A stratified sample of 100 statements went through independent human review, which confirmed the labels were correct.
Across those 2,116 statements, 64% are policies that require, forbid, or condition a specific agent action. The remaining 36% are descriptive context, such as architecture notes or project background. Policy density varies widely across repositories, from 0% to 97%, with 70.1% of repositories containing more policy statements than descriptive ones. File- or heading-level studies do not report this statement-level distribution, which is why the finer classification matters.
To understand how policies distribute across concerns, the study assigns each statement to one of 12 topic categories adapted from prior instruction-file research, applied at statement granularity rather than file granularity. Development Process and Implementation Details dominate the policy landscape at 87% and 85% respectively. Architecture is mostly descriptive at 23% because directory layouts and design summaries make up the bulk of those sections. The imported source figures call policy statements directives and call the system-observable policy subset system-level directives. The prose follows the paper's policy and system-observable terminology.
Five real statements from the dataset illustrate the range of enforcement requirements:
Statement Enforcement level Context S4: "Never push to main directly." per-event self-contained S5: "Never modify upstream source code." per-event project S6: "Run the full test suite before committing." cross-event project S7: "Data read from .env must not reach the network." cross-event project S8: "Do not update dependencies without approval." per-event task
... continue reading