MCP contract audit

MCP contract audit

Do FLOP's MCP tools return what they advertise? This read-only audit checks each safe tool's inputs, compares structured output with its declared schema and retains dated evidence. It also shows which trust material an attestation check still needs.

Quick start

Run from the workspace root with Node 22 and pnpm:

pnpm install --frozen-lockfile
pnpm --filter @flop-lab/mcpaudit build
node packages/mcpaudit/dist/cli.js run --tool flop_knowledge_get --json

The last command exits 1 when it reproduces the known contract violation. After the suite build, node packages/cli/dist/cli.js contracts run --json delegates to this same binary.

Commands and API

flop-mcpaudit run [--json] [--tool <name>] audits the hosted endpoint. Exit 1 means a contract violation was observed, even if its repeat could not complete. Exit 2 means the audit is incomplete or contains unverifiable or skipped checks. Exit 0 means every selected check passed. A missing output schema is never a pass.

import { runAudit, schemaFindings } from "@flop-lab/mcpaudit";

const report = await runAudit({ tool: "flop_knowledge_get" });
const violations = report.tools.filter((tool) => tool.status === "VIOLATION");
const errors = schemaFindings({ type: "integer" }, "1", "input");

The importable API exports source TypeScript for workspace consumers. The binary is built with tsup. Tests use raw responses recorded from https://mcp.flop.network/mcp.

Probe policy

Discovery runs once. The safe allowlist is independent of readOnlyHint. Unknown tools and the material-generation tool are skipped. All advertised tools still have a report row. No wallet, faucet or inference submission is used.

For each allowed tool, the runner validates a minimal request locally. It selects one small schema-valid edge when a different edge is available and sends one locally invalid input. Scalar fields use a small lower bound, empty value or enum endpoint. Object-only tools have no distinct scalar boundary. It does not exhaust every field combination. Known public settlement records seed account and channel lookups. When no seed exists, fixed zero identifiers test absence without inventing a resource.

AJV validates the advertised draft with formats enabled. It neither coerces values nor inserts defaults nor removes properties. additionalProperties is enforced exactly where the server declares it. Missing schemas remain unverifiable. A valid input that returns a domain error is not automatically a contract violation.

With structured output, at least one JSON text block must match it. A conflicting JSON text block is a violation. Plain explanatory text is not treated as JSON. The response schema and the text comparison are separate checks.

Each violation is repeated once within the same budget. confirmed means the repeat produced the same schema and instance paths. The complete raw reply remains in the fixture. Each observation links its request arguments, trace ids, UTC time, HTTP body size and duration.

Limits and recorded evidence

The hard ceiling is 149 physical HTTP attempts, including retries and discovery. Requests are sequential with a 500 ms minimum start gap. The shared MCP client backs off on 429 and 5xx with one extra attempt. Each attempt has a 15 second deadline. Bodies are capped at 2 MB. Body size means UTF-8 response body bytes after HTTP decompression. Latency covers the fetch and body read, not request spacing. The full-run duration includes spacing and retries.

There is no cross-run response cache. Its TTL is zero so confirmation requests are fresh. The website uses the committed recording and makes no audit requests on page load. Run the CLI for a new observation.

last-run.json was measured on 2026-11-05 at 21:44:20 UTC. It records 96 HTTP attempts and covers the 34 advertised tools: 2 tool rows with violations, 1 pass, 30 unverifiable and 1 skipped. These are tool-level results. Input rejection checks can pass on tools whose output remains unverifiable.

flop_knowledge_get reproduces network #87. flop_knowledge_manifest also returns undeclared fields, including root updatedAt, schema and specRecords. Its complete output was checked, including every concept entry.

Raw fixtures below the repository file limit are JSON. Larger responses are stored as rawGzipBase64 inside their JSON metadata. Gzip is lossless. test/fixtures.ts demonstrates decoding. pnpm --filter @flop-lab/mcpaudit record replaces the audit recording with a fresh bounded run. scripts/references.ts records Intel and CVE references separately from the MCP request budget.

Trust material

The inspector reports the trust-root version, snapshot hash, models, per-model TEE platform ids and dstackVkeys. It preserves numeric platform ids because the response provides no versioned name mapping. Empty keys cannot verify a dstack quote.

The recorded MCP TCB response cites INTEL-SA-00219 with release date 2025-12-15. Intel's advisory, fetched on 2026-11-05 at 21:44:19 UTC, says original release 2019-11-12 and last revision 2019-11-20. It concerns SGX with processor graphics. The Intel-assigned CVE-2019-0117 record links that same advisory. This mismatch is already tracked in network #36.

This TCB metadata is inconsistent with the cited advisory. It is not evidence of current platform security. The inspector compares the supplied id with a dated reference. It does not download a full current Intel PCS collateral set during each audit.

Checks

pnpm --filter @flop-lab/mcpaudit check runs strict types, lint, formatting and offline tests. Recorded output failures, invalid-input acceptance, empty discovery, malformed envelopes, conflicting text, budget exhaustion and Intel date parsing are covered. Live recordings are manual so the offline gate makes no public requests.

What this does not verify

This is a bounded sample of advertised contracts. It does not cover every argument combination or prove service uptime. No output schema means no output-contract verdict. A matching schema does not establish semantic truth, balances, payment finality, inference quality or cryptographic proof. The simulated quote verifier is not used as attestation evidence. Trust-root contents are not authenticated against a finalized chain state. The public hosted service does not expose its deployed source revision. One material-generation operation remains deliberately uncalled.