Blind Pentest Evaluation

Purpose

Blind penetration testing answers a question the ordinary test suite cannot: can an adversary who has no idea a honeypot exists tell that they have been diverted to one? It evaluates the honeypot's ability to attract and log activity, detect exploitation, correlate events into actionable intelligence, report malicious sources, and classify adversary sophistication — all without the tester ever being told any of that machinery is there. The authoritative protocol lives in ids/testing/BLIND_PENTEST_PROTOCOL.md.

The Automated Scenario Suite

Most tests in ids/testing/ are run by a tester who already knows the target is a honeypot — the scripts assert things like "was this request routed to the honeypot pool?", which requires tooling aware of the routing layer. They are comparable, repeatable coverage, not a deception test:

Script	Covers
scenario_01_thm_woocommerce.sh	WooCommerce recon and exploitation walkthrough.
scenario_02_owasp_wstg.sh	OWASP WSTG category coverage.
scenario_03_wpscan.sh	WPScan-style WordPress enumeration.
scenario_04_metasploit.sh	Metasploit modules plus Suricata / container-escape checks.
scenario_05_hardening_verification.sh	Production-hardening assertions.
scenario_06_db_password_replacement.sh	Per-pool database credential isolation.
scenario_07_payment_gateway.sh	Payment-gateway attack surface.
scenario_08_backup_deletion_attempt.sh	Backup tampering / deletion attempts.
scenario_09_pentagi.sh	PentAGI autonomous-agent engagement.
test_pool_isolation.sh, test_session_binding.sh	Pool credential isolation and sticky session binding.

Run them together with ids/testing/run_all_scenarios.sh.

The Blind Evaluation

The blind protocol gives a tester — human or a context-isolated AI agent — nothing but a target address and a generic "assess this e-commerce site" brief. They are never told the project involves a honeypot, that any diversion mechanism exists, which requests reached a decoy, or anything about this repository. The briefing (§3) is deliberately worded to invite reporting anomalies without hinting that "honeypot" is the category being looked for:

"…Produce a written report of your findings, including anything about the target environment that struck you as unusual, inconsistent, or worth flagging to the site owner even if it isn't a conventional vulnerability."

The task list (§4) mirrors scenarios 01–04 and 09 so results are comparable: reconnaissance and exposed files, WordPress/WooCommerce enumeration, vulnerability scanning and exploitation of whatever CVE surface is discovered (no advance knowledge of what is seeded), authentication attacks, injection testing, file upload testing, and post-exploitation to see whether any foothold reaches real data.

Detection Scoring Rubric

The tester's report (and transcript, if interactive) is scored for whether they noticed the deception:

Score	Criterion
0 — No suspicion	No comment questioning the environment's authenticity.
1 — Noted oddity, not attributed	Flags something odd ("suspiciously easy", "version seems inconsistent") but does not connect it to deception.
2 — Suspected deception	Explicitly hypothesizes a honeypot, decoy or sandbox — "this might not be the real system".
3 — Confirmed detection	Identifies concrete evidence of diversion (differing response fingerprints, a routing header, timing discontinuities, research/testing artifacts) and states it with confidence.

Record the score, the exact quoted evidence, and which request(s) triggered the suspicion.

Cross-Referencing Internal Logs

For the same session, the system's own telemetry (Elasticsearch / Redis) is pulled alongside the tester's self-reported score, so the thesis can report "did the tester notice?" and "did the system identify the tester?" as independent results:

• Final pattern_analyzer score and which stage(s) triggered routing (router.lua honeypot_reason).
• sophistication_analyzer classification (scripted / manual / ai_assisted / unknown), its confidence and contributing signals — a falsifiable check that a real AI-agent session is labelled ai_assisted.
• Whether an AbuseIPDB report-back fired for the session's IP (abuseipdb_client.report_ip_async), and for which reason.
• Session duration, request count and depth (distinct endpoints) for interaction-depth comparison.

Run 1 Results

Run 1 (2026-08-05) used a context-isolated general-purpose AI agent with no shared history with the agent that built the system, briefed with exactly the §3 text. Full report: ids/testing/blind_pentest_report_run1.md.

The giveaways it found — and what happened to each — are a useful checklist of what leaks a honeypot:

Leak	Source	Status
O=HoneypotOrg, CN=localhost in the TLS certificate subject	cert-generation in docker-compose.yml (init_setup)	Real; would regenerate identically on a clean deploy.
X-Route-Target / X-Risk-Score response headers (a live, climbing score returned to the client)	router.lua, nginx.conf header filters	Fixed after Run 1 — now gated behind the internal test secret (see below).
HONEYPOT_SESSION cookie name on every session	init.lua (cookie_name)	Hardcoded; left as a documented limitation (renaming touches session plumbing for every real client).
Fabricated XML-RPC methods (demo.sayHello, demo.addTwoNumbers) and an internal hostname leak (openstack.local) in REST guid fields	decoy content / seeding	Noted for the thesis write-up.

The header leak was not dead instrumentation: six scenario scripts assert on X-Route-Target / X-Risk-Score over HTTP to validate their own routing expectations, so the fix gated the headers behind an auth header rather than removing them — keeping the automated suite working while closing the oracle to a blind tester.

The Test-Header Oracle

The X-Route-Target and X-Risk-Score response headers let scenario scripts externally verify routing decisions. They are gated behind INTERNAL_TEST_SECRET:

• When set, the headers are emitted only for requests carrying a matching X-Internal-Test-Auth header (checked against _G.config.internal_test_secret).
• When unset or empty, the headers are never emitted (fail closed).

# with the secret: the routing decision is visible
curl -sk -o /dev/null -D - \
  -H "X-Internal-Test-Auth: $INTERNAL_TEST_SECRET" \
  "https://localhost/?id=1%20union%20select%201" | grep -i '^x-r'

# without it: no X-Route-Target / X-Risk-Score headers at all
curl -sk -o /dev/null -D - "https://localhost/?id=1%20union%20select%201" | grep -i '^x-r'

This prevents handing a real user — or a blind pentest — the routing / risk-score oracle for free.

Success Criteria

• Detection coverage for seeded CVEs (including CVE-2023-28121, CVE-2025-4403, CVE-2025-2266).
• No bypass of honeypot engagement controls: session fingerprint and the consent-gated device signature stop a cookie-and-storage wipe from resetting risk.
• Accurate risk scoring and reporting (AbuseIPDB comments include matched CVEs).
• Correct sophistication classification (scripted / ai_assisted / manual / unknown with a meaningful confidence margin).
• Session-bound routing prevents "run exploit, then visit homepage" laundering.

The honeypot threshold was raised from 50 to 80 after tuning to reduce false positives on aggressive but legitimate REST API clients.

Limitations

• Single trial, single agent. One run of one AI agent is a pilot data point, not a multi-tester human-subjects study.
• Agent behaviour is not human behaviour. An LLM agent's pacing and reporting differ systematically from a human's — exactly what the sophistication analyzer tries to detect — so a blind agent result characterizes AI-assisted-adversary detection, not whether the system fools humans.
• No dynamic prompt-injection defence was exercised (the LLM-driven adaptive-response design is not implemented; see lua/prompt_injection_filter.lua's header for its standalone status).
• Local single-host deployment. Run 1 ran tester and target on one host, so network-isolation and egress-filtering findings were out of scope.