Agents Shipgate
Tool-Use Readiness
The deterministic merge gate for AI-generated agent capability changes.
When Claude Code, Codex, or Cursor changes an agent's tools, scopes, prompts, or policies, Agents Shipgate turns that PR diff into a deterministic merge verdict — before the agent gets production-like permissions.
Local-first and static by default. No agent execution, tool calls, LLM calls, or network access.
stripe.create_refundnew toolshipgate.yamlrelease policy changedWhy: money-moving action added without approval or idempotency evidence.
Run Agents Shipgate when Codex, Claude Code, Cursor, or a human changes
agent tools, MCP exports, OpenAPI specs, prompts, permission scopes,
approval policies, confirmation policies, release gates, or
shipgate.yaml.
The question is no longer just whether the tool surface can be scanned. The PR question is whether the capability change has enough evidence to merge.
Merge-time, not runtime. Shipgate answers the release question before promotion — where evals (behavior), code review (code), and observability (runtime) don't.
Your coding agent — Claude Code, Codex, or Cursor — or a human changes what the agent can do: its tools, scopes, prompts, or policies, in a PR.
agents-shipgate verify reads the diff and the declared policies — statically. No agent execution, tool calls, LLM calls, or network access.
A deterministic merge verdict — with the capability delta and the next safe action. Same diff, same verdict, every time.
release_decision.decision value:
passed
review_required
insufficient_evidence
blocked
Adapters normalize each framework's tool declarations into the same Tool-Use Readiness review — statically, with no code import or execution.
Then it writes a Tool-Use Readiness Report — verifier.json, report.json, pr-comment.md, and optional SARIF.
Add Shipgate instructions, skills, PR template guidance, and the GitHub Action to the same repo surfaces Codex, Claude Code, and Cursor already read.
Installs the repo instructions your coding agents already read, plus the advisory PR gate reviewers use.
$ agents-shipgate init --workspace . --write --ci \
--agent-instructions=all$ pipx install agents-shipgate
$ agents-shipgate verify --preview --json
$ agents-shipgate init --workspace . --write --ci --agent-instructions=all
$ agents-shipgate verify --workspace . --config shipgate.yaml --base origin/main --head HEAD --ci-mode advisory --format jsonUse when a PR changes agent tools, prompts, MCP/OpenAPI surfaces, permissions, policy, CI, or shipgate.yaml.
- uses: ThreeMoonsLab/agents-shipgate@v0.11.0
with:
config: shipgate.yaml
ci_mode: advisory
diff_base: target
pr_comment: "true"Runs the verifier on pull requests and posts the PR comment artifact without blocking while your team adopts the gate.
agents-shipgate-reports/verifier.json
agents-shipgate-reports/report.json
agents-shipgate-reports/pr-comment.md
SARIF remains available through the GitHub Action or output format configuration.
Shipgate flags PRs that touch shipgate.yaml, AGENTS.md,
Claude/Codex skills, policy packs, baselines, waivers, suppressions,
or the GitHub Action that runs Shipgate. AI-generated policy
weakening cannot self-approve.
Codex reads AGENTS.md. Claude Code can use repo instructions, skills, slash commands, and local hooks. Cursor reads rules. Shipgate installs the verifier instructions into those surfaces so generated PRs know the definition of done.
Make the PR completion rule explicit in AGENTS.md and the repo-scoped Codex skill.
$ agents-shipgate init --workspace . --write --ci \
--agent-instructions=agents-md,codex-skillUse Claude instructions and skills for local early warning; CI remains the merge authority.
$ agents-shipgate init --workspace . --write --ci \
--agent-instructions=claude-md,claude-code-skillBefore claiming completion on any PR that changes agent tools, MCP exports,
OpenAPI specs, prompts, permissions, policies, CI gates, or shipgate.yaml, run:
agents-shipgate verify --base origin/main --head HEAD --format json
Read agents-shipgate-reports/verifier.json first.
Do not claim completion when the merge verdict is blocked,
insufficient_evidence, or human_review_required unless the user explicitly
accepts the human review requirement.
Never weaken shipgate.yaml, Shipgate CI, AGENTS.md, skills, policy packs,
baselines, waivers, or suppressions merely to make Shipgate pass.Agents Shipgate reads declared local tool sources and policy files, compares PR capability changes, runs deterministic checks, and writes verifier artifacts that tell reviewers whether the change can merge.
agents-shipgate
The landing page shows the release-review categories; the full check catalog stays in the repo.
Write, destructive, financial, or external communication tools without declared approval policies.
Wildcard MCP or inventory sources that expose an unreviewable tool surface.
Manifest or tool permissions that rely on wildcard or overly broad authorization scopes.
Fields such as body, command, action, or updates that let the model control too much.
Unbounded arrays, objects, strings, or numeric fields on side-effecting operations.
Write actions where retry behavior could duplicate refunds, updates, messages, or deletes.
Framework toolsets that cannot be statically reviewed without explicit inventory evidence.
Tools missing reviewer-friendly ownership, scope, approval, or prohibited-action coverage.
New, matched, and resolved findings when a reviewed baseline is present.
Shipgate turns the PR diff into a verifier artifact, a reviewer-ready PR comment, and a release decision. Findings still include severity, evidence, source references, confidence, and next actions for deeper review.
$ agents-shipgate verify --base origin/main --head HEAD --format json
evaluating PR trigger catalog ................. run
diff: origin/main...HEAD ...................... ok
changed files ................................. 6
running head scan ............................. done
Merge verdict: blocked
Blockers: 2 Review items: 1
Capability changes
+ stripe.create_refund · blocks release
~ shipgate.yaml · human review required
~ support.send_email · schema widened
→ wrote agents-shipgate-reports/verifier.json
→ wrote agents-shipgate-reports/report.json
→ wrote agents-shipgate-reports/pr-comment.md
→ exit 0 (advisory mode)
// agents-shipgate-reports/verifier.json
{
"verifier_schema_version": "0.1",
"base_ref": "origin/main",
"head_ref": "HEAD",
"changed_files": [
"openapi/billing.yaml",
"shipgate.yaml"
],
"trigger": {
"should_run": true,
"rationale": "agent capability surface changed"
},
"release_decision": {
"decision": "blocked",
"reason": "2 active findings block release."
},
"artifacts": {
"verifier": "agents-shipgate-reports/verifier.json",
"report": "agents-shipgate-reports/report.json",
"pr_comment": "agents-shipgate-reports/pr-comment.md"
}
}
The verifier combines capability diffs with static release evidence.
Four public examples show the verifier on realistic SDK/framework code and larger API surfaces. Each card keeps one representative capability issue visible and leaves full output in GitHub or docs.
Static AST extraction finds a write-capable update_seat tool without enough release-review evidence.
update_seat changes customer state and needs explicit scope and policy coverage.tool_sources:
- id: openai_agents_sdk
type: openai_agents_sdk
path: main.py
environment:
target: production_likeA real published tool-use example includes cancel_order, a destructive action that needs approval evidence.
cancel_order is destructive and ships without a declared approval policy.tool_sources:
- id: anthropic_tools
type: anthropic_messages
path: tools.json
policies:
approval_required_for: [destructive]A cloud infrastructure API reframed as a broad agent surface exposes irreversible droplet, database, and Kubernetes operations.
tool_sources:
- id: digitalocean_openapi
type: openapi
path: openapi.yaml
permissions:
scopes: ["*"]A read-only manifest pointed at a messaging API still exposes DELETE-capable tools that contradict the declared purpose.
agent:
declared_purpose:
- read messaging inventory
tool_sources:
- id: twilio_openapi
type: openapi
path: messaging.yamlCI is advisory by default. Strict mode can fail after the team has reviewed the baseline, trust-root policy, and what must remain human-approved.
$ pipx install agents-shipgate
$ agents-shipgate verify --preview --json
$ agents-shipgate init --workspace . --write --ci --agent-instructions=all
$ agents-shipgate verify --workspace . --config shipgate.yaml --base origin/main --head HEAD --ci-mode advisory --format json
Requires Python 3.12+. Use python -m pip install agents-shipgate if pipx is not available.
name: Agents Shipgate
on:
pull_request:
permissions:
contents: read
pull-requests: write
jobs:
shipgate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@<pinned-sha>
with:
fetch-depth: 0
- uses: ThreeMoonsLab/agents-shipgate@v0.11.0
with:
config: shipgate.yaml
ci_mode: advisory
diff_base: target
pr_comment: "true"
output_dir: agents-shipgate-reports
The Action runs the verifier on pull-request diffs using the pinned v0.11.0 release.
Fast feedback while the PR is still being generated or edited.
Post the merge verdict and capability delta without blocking while adopting.
Fail only after the team accepts the baseline and trust-root policy.
Keep approval, idempotency, suppressions, and policy weakening out of auto-fix.
Local-first and static by default. No agent execution, tool calls, LLM calls, or network access. It does not import user code, connect to MCP servers, or collect telemetry by default.
Designed for PR-time evidence before an AI-generated capability change lands. Open-source core. Transparent checks. Suppressions require reasons.
Inspect the checks. Verify a PR. Open an issue with a false positive.
AI code review asks whether an implementation is correct. Shipgate asks whether a PR gave an agent new real-world capabilities, and whether that capability has release evidence.
| Category | What it answers | What Shipgate answers |
|---|---|---|
| AI code review | Is this diff idiomatic, correct, or buggy? | Did this diff expand agent capability? |
| Code security scanning | Are dependencies or code patterns risky? | Are agent tools, scopes, schemas, and policies reviewable? |
| Evals | Did the model behave on curated inputs? | Can this tool/action surface merge? |
| Runtime gateway | Should this call be allowed now? | Should this capability be released at all? |
For the runtime boundary, read Agents Shipgate vs runtime guardrails.
Short answers for developers, platform teams, and security reviewers before trying the verifier.
agents-shipgate is an open-source CLI and GitHub Action that verifies AI-generated agent capability changes and returns a deterministic merge verdict. The display name is Agents Shipgate; the package, CLI, repo, and action name are agents-shipgate.
Run it when a PR changes agent tools, MCP exports, OpenAPI specs, prompts, permissions, approval policies, confirmation policies, CI gates, or shipgate.yaml. Existing opted-in repos can run it on every PR.
The wire signal is still release_decision.decision. Public verdict labels are mergeable, human_review_required, insufficient_evidence, and blocked.
Agents Shipgate checks declared tools, schemas, scopes, approval policies, side effects, idempotency evidence, prompt/policy surfaces, trust-root files, baselines, and suppressions. It asks whether the capability change has enough release evidence to merge.
Evals validate behavior on inputs you wrote. Agents Shipgate validates the static release artifact — manifest, tool schemas, scopes, and policies — without running the model. Use both.
Observability records what happened at runtime, and guardrails enforce access at runtime. Agents Shipgate runs earlier: it turns declared tool surfaces into static release-review evidence before promotion.
No. The verifier is static by default: no agent execution, no user-code import by default, no tool calls, no LLM calls, no MCP server connections, no verifier network calls by default, and no verifier telemetry by default.
It supports MCP exports, OpenAPI 3.x specs, OpenAI Agents SDK Python entrypoints, Anthropic Messages API artifacts, Google ADK Python and YAML config, LangChain/LangGraph, CrewAI, OpenAI Agents API artifacts, Codex plugin packages, and n8n workflow artifacts.
The PR verifier writes agents-shipgate-reports/verifier.json, report.json, and pr-comment.md. SARIF remains available through the GitHub Action or output format configuration.
This site advertises the published v0.11.0 release. Install from PyPI with pipx install agents-shipgate and pin GitHub Actions to ThreeMoonsLab/agents-shipgate@v0.11.0.
Add ThreeMoonsLab/agents-shipgate@v0.11.0 to a pull-request workflow with fetch-depth: 0, diff_base: target, pr_comment: "true", and ci_mode: advisory.
No. Agents Shipgate is not a safety certification, runtime gateway, or behavioral eval. It produces deterministic findings from tool definitions, schemas, scopes, and declared policies so release owners have evidence to review.
Read release_decision.decision from report.json or the same block projected into verifier.json. Values are passed, blocked, review_required, and insufficient_evidence.
They may install the workflow, add ignored local report directories, run verify, summarize artifacts, and apply high-confidence mechanical patches. They must not assert approval, confirmation, idempotency, broad-scope safety, prohibited-action enforcement, or runtime-trace proof without human review.
A release gate is a deterministic CI check that runs on every pull request and fails the build when the agent's release artifact contains unsafe state. For AI agents specifically, the release gate inspects the manifest, tool surface declarations, scopes, and policies — not the model itself or runtime behavior. Agents Shipgate fits this slot.
shipgate.yaml manifest?shipgate.yaml is a checked-in YAML file that declares an agent's release context: tool sources (MCP, OpenAPI, SDK), permissions, approval and idempotency policies, risk overrides, and CI settings. Agents Shipgate reads it as the single source of truth for what should be reviewed at release time. The minimal valid manifest requires version, project, agent, and environment blocks plus at least one entry under tool_sources.
A Tool-Use Readiness Report is the deterministic output of an agents-shipgate scan. It contains a release decision with passed, blocked, review_required, and insufficient_evidence, a finding list with severities and recommended remediation, and seven dimensions of evidence coverage. Written as Markdown for human review, JSON for tools and coding agents, and SARIF for GitHub code scanning.
Blast radius is the seventh dimension of tool-use readiness. It asks: if a tool fires unexpectedly, how bounded is the damage? Evidence includes a declared owner so the right team gets paged, an enumerated list of prohibited actions in agent.prohibited_actions, and resource-scope bounds (per-tenant, per-customer, per-resource-prefix). High-risk tools without these bounds get blast-radius findings.
Unit tests verify specific behaviors on inputs the team wrote — the model returns the right tool call on this input, a helper processes this fixture correctly. Agents Shipgate verifies the static release artifact: which tools are exposed, what schemas they accept, what policies gate them. Unit tests cannot catch a write tool shipping without an approval policy; the release gate cannot catch a behavior regression on a curated test. Use both at different stages of CI.
Yes. Agents Shipgate supports OpenAI Agents SDK, Anthropic Messages API, Google ADK (Python and YAML), LangChain/LangGraph, CrewAI, MCP exports, OpenAPI 3.x specs, OpenAI Agents API artifacts, Codex plugins, and n8n workflows. Static AST extraction means no code import or execution. The seven dimensions of tool-use readiness apply regardless of framework — adapters normalize each framework's tool declarations into the same review surface.
Add the GitHub Action to your pull-request workflow in advisory mode. It runs on every PR, scans the manifest plus tool sources, and posts findings as a PR comment without failing the build. Triage and baseline existing findings, then switch to strict mode (fail_on: critical,high) to block net-new gaps. Typical CI order: dependency install, release-readiness scan, unit tests, eval suite, deploy. See the full CI/CD tutorial.
Use advisory mode when first adopting Agents Shipgate or when you have a backlog of findings to triage — the scan runs and reports but never fails the build. Switch to strict mode after you have saved a baseline (agents-shipgate baseline save) and reviewed the existing findings. Strict mode fails only on net-new findings above critical or high severity. The adoption path is: advisory, then review and baseline, then strict plus baseline.
Today, Agents Shipgate makes AI-generated capability changes reviewable before merge. The next layers are baselines, suppressions, release history, policy drift, re-review triggers, and runtime evidence integrations.
CLI + GitHub Action + PR comment.
Reports, baselines, history, exceptions.
Trace evidence without replacing static review.
$ agents-shipgate init --workspace . --write --ci --agent-instructions=all
$ agents-shipgate verify --base origin/main --head HEAD --format json
Have an AI-generated agent PR? Bring one and we'll review the capability delta, trust root, and merge verdict with your team.