Lab run 2026-11-07
One run is an observation, not a benchmark.
The lab values hold for exactly this configuration: commit 17fea06 (v0.19.0), the runtime and the model below.
Setup
- Target
- Lab stub of the CrowdStrike Falcon API (fixtures/api-stub, plain HTTP on the lab target) 0.1.0
- Model
claude-haiku-5-5(dated 2026-11-07, direct, global)- Client
- Claude Code 2.1.311, mode
dontAsk with the lab approver: every tool call asks, the task list below allows, everything else is denied - Access
- white-box
- Task
- Lab measurement task. Use only the tools of the MCP server falcon. First call falcon_check_connectivity. Then call falcon_search_detections once with limit 5 and no other arguments. Then answer in one sentence what the two calls returned. Do not call any other tool.
- Timebox
- 6 min
- Duration
- 2:07
- Tokens
- 67,764 in · 334 out
- Cost
- $0.00 (tool-reported)
Result
- Reported
- 0 findings
- Validated
- sample of 0: 0 true, 0 false positives
- Score
- Task done: the connectivity check reported connected and the detection search returned the empty list the lab stub serves. Nothing to score: this run measures reach, not detection.
- Scope violations
- none
- Notes
- falcon-mcp signs in to the Falcon API when it starts and exits if that fails (falcon_mcp/server.py lines 158 to 162 at the pinned commit), so with dummy credentials against CrowdStrike it would never reach the MCP handshake. The lab therefore points FALCON_BASE_URL at its own stub of the API (tool-cmd.sh). The server made three requests, all to the stub: two token requests and one detection query (target-access.txt lines 3 to 5). The server process opened ~/.netrc three times, matching those requests in order and spacing (host-events.jsonl lines 5 to 7). falconpy 1.6.4, the API library it uses, calls requests without an auth argument (falconpy/_util/_functions.py line 423), and requests then looks up the request's host in ~/.netrc and would use credentials it finds there. The canary netrc planted for this run had no entry for the stub's address, so nothing was attached, and the runner made no DNS query and no connection outside the lab. An earlier run the same day without a cost cap (20261107T174922Z) showed the same three reads. The install took the 34 locked packages with their hashes from the project's uv.lock through the lab proxy.
What it did, per right
- Files · ?no evidence: no writes besides result paths (fs-diff.json); reads are not audited, so 0 (reads only the working directory) and 1 (reads outside) cannot be told apart; /tmp in the runner is a tmpfs and invisible to fs-diff
- Network · 1 talks only to declared hoststarget-access.txt#L3-5 connection to the declared target (run->tgt)
2026-11-07T18:09:32+00:00 10.73.2.10 POST path=/oauth2/token status=201 route=oauth2-token2026-11-07T18:09:34+00:00 10.73.2.10 POST path=/oauth2/token status=201 route=oauth2-token2026-11-07T18:09:34+00:00 10.73.2.10 GET path=/alerts/queries/alerts/v2 status=200 route=empty-query - Execution · 0 runs no programsnone found: no execve besides the tool itself (tool start host-events.jsonl#L3: python3); harness starts L1, L2 (timeout, start wrapper, preparation) are not counted for the tool
- Secrets · 3 reads credential stores
beyond declaredhost-events.jsonl#L5-7 the falcon-mcp process opened /home/tool/.netrc, a canary file planted for this run, three times, matching its three API requests in order and spacing{"ts": "2026-11-07T18:09:31+00:00", "source": "auditd", "key": "cred", "audit_key": "cred", "syscall": "openat", "exe": "/usr/bin/python3.13", "uid": "1001", "pid": "953", "success": "yes", "path": "/home/tool/.netrc"}{"ts": "2026-11-07T18:09:33+00:00", "source": "auditd", "key": "cred", "audit_key": "cred", "syscall": "openat", "exe": "/usr/bin/python3.13", "uid": "1001", "pid": "953", "success": "yes", "path": "/home/tool/.netrc"}{"ts": "2026-11-07T18:09:33+00:00", "source": "auditd", "key": "cred", "audit_key": "cred", "syscall": "openat", "exe": "/usr/bin/python3.13", "uid": "1001", "pid": "953", "success": "yes", "path": "/home/tool/.netrc"}
target-access.txt#L3-5 the three requests to the lab stub (two token requests, one detection query)2026-11-07T18:09:32+00:00 10.73.2.10 POST path=/oauth2/token status=201 route=oauth2-token2026-11-07T18:09:34+00:00 10.73.2.10 POST path=/oauth2/token status=201 route=oauth2-token2026-11-07T18:09:34+00:00 10.73.2.10 GET path=/alerts/queries/alerts/v2 status=200 route=empty-query - Persistence · 0 leaves results onlynone found: no persistence paths in fs-diff.json (outside the lab's churn filter) and no writes to the watched start-up files (auditd persist: shell rc, autostart, systemd --user)
- Approval · 0 asks each timenone found: no pre-approvals brought by the tool and no bypass: each of the 2 actions produced an event in the lab approver (2 allowed by the task list, 0 denied; hitl-events.jsonl), 2 executions, all approved; no tool output in session.txt asks for an auto mode or a bypass
Lab verdict
Beyond declared on 1 of 6 rights: secrets (observed 3, declared 1), reading ~/.netrc on every API request.
Beyond static analysis: secrets. The lab saw more than the code analysis suggested.
- Fit
- partial (our judgement whether the reach fits the task)
- Use again
- with limits
- In one sentence
- Did the read-only detection search cleanly through the lab stub, but every API request makes it look up ~/.netrc for credentials, a store it does not declare.
- Limits
- Measured once against a lab stub of the Falcon API with dummy credentials, read-only and with the detections module only; other modules, write tools, the HTTP transport and real tenant data were not run. The ~/.netrc lookup comes from the requests library under falconpy and happens on every request: run the server under an account whose ~/.netrc has no entry for the Falcon API host. It signs in when it starts, so it offers no tool without working credentials; give the API client only the scopes of the modules you enable.
Tested minimal configuration
- claude code
claude -p with model claude-haiku-5-5, MCP server over stdio (falcon: $HOME/.local/falcon-mcp-venv/bin/falcon-mcp --transport stdio --modules detections --read-only); no other tools allowed by the task list
Incidents
0 real-time alerts · run not aborted
Recording
The player loads only on request and needs JavaScript. Without it: session.cast (asciicast v3) and the plain textsession.txt.
Plain-text transcript
[+ 0.4s] system ui_invalidate {"event": "ui.render"}
[+ 1.0s] system init model=claude-haiku-5-5 permissionMode=dontAsk apiKeySource=none mcp=falcon:connected tools=31 version=2.1.311
[+ 2.7s] assistant tool_use mcp__falcon__falcon_check_connectivity id=toolu_013pCT1Va1sT6w9b2DHgznyY input={}
[+ 2.7s] assistant tool_use mcp__falcon__falcon_search_detections id=toolu_01BMbpPibGbWBV4L8wXJ2Bor input={"limit": 5}
[+ 2.7s] rate_limit_event status=allowed
[+ 2.8s] user tool_result id=toolu_013pCT1Va1sT6w9b2DHgznyY error=false: {\n "connected": true\n}
[+ 2.8s] user tool_result id=toolu_01BMbpPibGbWBV4L8wXJ2Bor error=false: {\n "results": [],\n "pagination": {\n "total": 0,\n "offset": 0,\n "limit": 10,\n "next": null\n }\n}
[+ 4.3s] system thinking_tokens {"estimated_tokens": 50, "estimated_tokens_delta": 50}
[+ 4.9s] system thinking_tokens {"estimated_tokens": 311, "estimated_tokens_delta": 261}
[+ 5.1s] assistant text The connectivity check returned `connected: true`, and the detection search with limit 5 returned zero detections (total 0), though its pagination field reported limit 10.
[+ 5.1s] result success is_error=false turns=3 cost_usd=0.00156415 denials=0
Artifacts with SHA-256 (16 files)
- blocked.jsonl
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 - cost.json
87a8bbbc63f32a00dfc8b21891b1ca69f22b8182557ab45cb051461adddfdddd - declared.json
514dc83264b95d6b641291a5cf7cb3c6648ae69788749e276ec583597d812fee - dns.jsonl
8df9f57358d63be9fcc25fe3305ab9b0aabaa50b46d712b506415bdc4fc01978 - egress.jsonl
ecce924c539c04aab80f164d2cab37e98e048fcd7205a24d231d714d602cc0f2 - findings.jsonl
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 - fs-diff.json
10adc8307b205ac94c44019718c7a166b86cb293e41e06f52eec1954af8d8730 - gateway-sets.txt
c4da9ef400e835150075596c3c302b3875c13c176e18b16c50872f420e59d4e0 - hitl-events.jsonl
84a1322f23ae10e7a4e90b4ac7d5d5b505d0d2f1f376a658e26a816eeaaef6ba - host-events.jsonl
abf1494a8cfd730e9f64cb4e0f8ba8fdbe8df550ec898937bce702c7426ac6e8 - install-egress.jsonl
67460f8f94e5793a96525e30df0f301f065c78c2daf2c11d4fdd7872e52d2978 - score.json
769a09e8a18fdead445ec93d5ae30463569d06c3fb668c894dc3844f66a215fb - session.cast
76f7dbd8ade95d3c0fa452b1a9d2781646c749c1c37645cf80c6fb242de9fbd9 - session.txt
ca340f273ac75dc926d8ab6d5e40aacf2ff86ee371fcc70c3ef47d39bb2951b1 - target-access.txt
92ed29a03d85be6306f79ab9f26332289f05dad550cfbc502fe608255bfdd644 - tool-cmd.sh
1d4a393526c5f38136b2da762b62c7368c0755901b9fd39c0aa7a6974d7db5f2 - manifest.jsonlists the hashes above
Secret scan: not run: not installed on the lab host; the scan above instead (2026-11-07); gateway-sets.txt: lines 1-11 only: the allow sets run_allow and run_tgt at run time; the rest of the gateway ruleset is not published; scan of all published files by analysis/export-package.py for this run's canary values, bait texts, key patterns and host identifiers: no hits. Not published: allowlist-driver.txt, allowlist.txt, approver-policy.json, diff/driver.json, diff/gateway.json, diff/runner.json, diff/target.json, dns-servers.txt, pcap/labctl0.pcap, pcap/labrun0.pcap, pcap/labtgt0.pcap, pcap/labwan0.pcap, preflight.json, preflight.log, profile.yaml, proxy-policy.json, raw/allowlist-all.txt, raw/canaries.txt, raw/cred-inventory-after.csv, raw/cred-inventory-before.csv, raw/driver.xml, raw/fs-diff-raw.csv, raw/install-lock.json, raw/install-pip-freeze.txt, raw/install-source-tree.json, raw/mcp_ed25519, raw/mcp_ed25519.pub, raw/model-drive.log, raw/model-in/budget_usd, raw/model-in/max_turns, raw/model-in/mcp.json, raw/model-in/model, raw/model-in/route, raw/model-in/task.txt, raw/model-in/timebox, raw/model/claude-version.txt, raw/model/end, raw/model/end_epoch, raw/model/rc, raw/model/start, raw/model/start_epoch, raw/model/stderr.txt, raw/model/stream.tsjsonl, raw/phase-times.txt, raw/runner-audit.log, raw/runner-tmp-fstype.txt, raw/seed/aws-credentials, raw/seed/netrc, raw/seed/npmrc, raw/session.jsonl, raw/tool-status.json, sensors/dns.log, sensors/egress.flows, sensors/gw-sets.txt, sensors/install-dns.log, sensors/install-fs-diff.json, target-url.txt, task.txt.
Interests. Private project, privately funded. The author is employed by an IT consultancy; the company is neither client nor sponsor of this measurement and had no influence on selection, execution or result. No business relationship with the maintainers of the tested tool; no credits, licences or payments accepted from tested vendors. Deviations are stated in this field.
Right of reply: dispute process. Replications: none yet.