Testing

The testing strategy follows the architecture: pure rules can be proven without OpenResty, adapters are covered by scenario scripts against a running stack, and the blind pentest checks the whole thing from the outside. The suite is intentionally boring to run — a plain Lua interpreter and Python unittest.

Testing Layers

• Unit / Integration Tests: Lua module logic, dashboard generation, timeline capture. Pure Lua modules (*_rules.lua) are testable under a plain Lua interpreter without OpenResty.
• Functional Testing: End-to-end honeypot interaction and detection validation.
• Blind Pentest Evaluation: Controlled adversary assessments with internal-test secret gating.

Coverage Map

Unit Test Coverage

Test File	Module Under Test
test_pattern_rules.lua	pattern_rules.lua — URI/header/CVE/automation scoring
test_router_rules.lua	router_rules.lua — routing predicates, decay, decision trace
test_vulnerability_rules.lua	vulnerability_rules.lua — CVE pattern matching, exploit kit detection
test_suricata_rules.lua	suricata_rules.lua — alert parsing, severity scoring, decay
test_abuseipdb_rules.lua	abuseipdb_rules.lua — report eligibility, categories, comments
test_lua_pattern_utils.lua	lua_pattern_utils.lua — URL decode, pattern escape, install wizard URI
test_decay_policy.lua	decay_policy.lua — escalation-aware decay, permaflag
test_ip_rules.lua	ip_rules.lua — IPv4/IPv6 CIDR matching
test_upload_rules.lua	upload_rules.lua — content type, parameters, endpoints, bypass
test_admin_rules.lua	admin_rules.lua — login, patterns, brute force, credential stuffing
test_session_rules.lua	session_rules.lua — anomalies, honeypot routing
test_session_identity_rules.lua	session_identity_rules.lua — cookie signing, fingerprinting, rejections
test_honeytoken_rules.lua	honeytoken_rules.lua — token-in-corpus matching
test_prompt_injection_filter.lua	prompt_injection_filter.lua — prompt injection detection
test_pool_router_rules.lua	pool_router_rules.lua — pool assignment, health, parsing
test_router_decide_route.lua	router.lua — routing decision pipeline
test_gdpr_rules.lua	gdpr_rules.lua — timestamp, canonical chars, validation
test_pool_assignment.lua	pool_router.lua — pool assignment lifecycle
test_sophistication_analyzer.lua	sophistication_analyzer.lua — scripted / AI-assisted / manual / unknown classification
test_threat_intel.lua	threat_intel.lua — threat-intel lookups and scoring
test_shared_ip_rules.lua	shared_ip_rules.lua — shared/CGNAT reputation isolation per session
test_no_tarpit.lua	repo-wide scan — no request-path code may sleep (tarpit removal regression)
test_suspicious_patterns.lua	pattern_rules.lua — generic SUSPICIOUS_PATTERNS list, false-positive regression

Dashboard Tests

• dashboard/tests/test_kibana_dashboards.py — Validates NDJSON dashboard integrity against build_dashboards.py output.
• dashboard/tests/test_host_settings.py — Host settings validation.
• dashboard/tests/test_overview_stats.py — Overview stats validation.
• dashboard/tests/test_learning.py — Learning page validation.

Internal Test Secret

The INTERNAL_TEST_SECRET env var gates the X-Route-Target and X-Risk-Score response headers. When set, headers only emitted for requests with matching X-Internal-Test-Auth header. When unset, headers never emitted (fail closed). This allows scenario scripts to verify routing decisions without exposing oracles to real users or blind pentests.

Running Tests

# Lua unit tests (plain Lua interpreter, no OpenResty needed)
cd ids/reverse_proxy_enhanced/lua && lua tests/test_<module>.lua

# Dashboard tests
cd dashboard && python3 -m unittest discover -s tests -v

# Scenario tests
bash testing/scenario_*.sh

Validation Results

Two of the dashboard's report generators turn a live stack into evidence: the Segmentation Validation run proves per-session isolation, and the Exploit test-scenario run measures detection against the seeded CVEs. Both export a PDF with per-request scores, routing decisions, resource footprints and statistics. The figures below come from representative recorded runs; regenerate them from the dashboard to validate a given deployment.

Segmentation validation — verdict EXCLUSIVE on both layers

The run opens ten independent browser windows, each with its own cookie jar, User-Agent and language, so the proxy sees ten separate users. Each one attacks, is flagged, and is expected to land on a honeypot pool of its own. The run also injects random session tampering — cookie clearings plus hijack and forgery attempts — to confirm a window cannot be pushed onto, or steal, another window's pool.

Recorded run	Honeypot layer	Windows	Verdict
2026-10-24	WordPress (reverse-proxy) level	10, each on its own pool	EXCLUSIVE
2026-10-18	Database-proxy level	10, each on its own pool	EXCLUSIVE

• No shared pools below the resource ceiling. Every flagged session received an exclusive pool; round-robin reuse only happens once scaling hits POOL_MAX / the memory and CPU budget. In the 2026-10-24 run the limit (10 of POOL_MAX=10) was reached at 2:33, yet zero windows ended up sharing a pool — every later diversion still found a free spare.
• Tampering was rejected. Forged and hijacked session cookies were scored as tampering and refused; a window that cleared its cookies was re-recognised by the passive fingerprint rather than handed a clean slate. The run injected 5 cookie clearings (all 5 kept their session) and 2 forgeries — 1 browser-storage forgery ignored (the server never reads localStorage/sessionStorage) and 1 cookie hijack rejected (bad_signature, +50 scored as tampering, logged in Redis session_cookie_rejections). The real cookie is SERVERID = <session id>.<HMAC> and only the proxy holds the key, so a forged one cannot be signed.
• Reassignment after scale-down worked. After a forced idle scale-down to the three Compose-declared pools, all 7 windows whose pool was removed were reassigned to a surviving pool — HTTP 200, same session, no 502 from the gone container and no fall-through to production. Carts stayed intact except where a window had cleared its own cookies (expected: the cart lives only in the fh_cart cookie, so the server keeps no copy to restore).
• Both deception layers pass. The database-proxy run exercises only data-layer patterns and checks the shared filesystem at the end (see ids/db_proxy/COVERAGE.md), since it cannot isolate file or upload attacks.

Scaling and latency (measured, 2026-10-24 WordPress-proxy run)

The run opened 10 windows (3 of them attacking only after pools were pre-built for them) and took 5:12, plus 0:31 of clean-up after the reassignment check. pool_manager adds one honeypot pool — an eshop container plus its own database seeded from production — per session that finds no free pool, up to the resource limit; the figures come from its own event log and from the windows themselves (routing read after every step and every 2 s while waiting).

Measurement	Value
Resource ceiling	12,288 MB, 12 CPUs, POOL_MAX=10 (each pool reserves 768 MB / 0.75 CPU in the budget)
Pools built during the run	7 (on top of 3 Compose-declared) — build time median 27 s, max 40 s
Build phases (per pool)	copy template dominates (10–27 s); database start ~3 s; rotate password ~1 s; content sync 1–2 s; WordPress start ~6 s; then create/register
Borrow while scaling (case A)	4 windows borrowed a ready pool round-robin, then moved to their own after a median 34 s (max 43 s)
Share at the ceiling (case B)	0 windows — the limit was hit at 2:33 but a free spare was always available
Failed page loads	16 first-attempt failures, every one ERR_NETWORK_CHANGED while Docker was adding/removing a pool's network interface; all retried with back-off and loaded in the end

Pool memory footprint. The 10 pool containers peaked at a measured 4,687 MB (~469 MB per pool) — well under both the 768 MB budget charge (deliberate headroom) and the 1,536 MB container ceiling (1,024 eshop + 512 database). The host as a whole used 13,191 MB of 24,032 MB, because it also runs production, Redis, Suricata and the rest.

Page-load latency (the same product page, one window at a time, timed navigation→load finished) was effectively indistinguishable between production and a honeypot pool:

Served by	Median	p90	Max	n
Production	360 ms	382 ms	384 ms	12
Own honeypot pool	372 ms	469 ms	495 ms	16

The probe loads the same product page three times during the run (all windows on production, after the scale-up, at the end), so step loads made while ten windows render at once do not distort the comparison.

Forced scale-down. With the idle timeout (POOL_IDLE_TIMEOUT_SECONDS=1800 s) too long for a test, the run forced the scale-down: requested at 4:36, finished in 16.2 s, dropping from 10 pools to 3 (the Compose-declared 1, 2, 3 are only freed, never removed). Pool-container memory fell 4,687 MB → 1,515 MB (−3,172 MB) and running containers 20 → 6.

Scaling and latency (measured, 2026-10-18 database-proxy run)

On the database layer a pool is only a honeypot database cloned from production and scrubbed; the one shared shop connects a diverted session to it on every request. Pools are therefore much cheaper and faster to add. The run had 10 windows (3 delayed) and took 4:47, plus 0:22 of clean-up.

Measurement	Value
Pools built during the run	9 (on top of 1 Compose-declared) — build time median 15 s, max 17 s
Build phases (per pool)	database start 6–9 s; clone + scrub 7–9 s; query log and create/register under 1 s
Borrow while scaling (case A)	6 windows borrowed a pool, then moved to their own after a median 9 s (max 11 s)
Share at the ceiling (case B)	1 window shared a pool after the limit (10 of POOL_MAX=10) was reached at 2:27; every window still ended on a pool of its own
Pre-build for 3 delayed windows	3 free pools after 29 s (2 built for them, 1 spare already free)
Pool memory	peak 2,133 MB across 10 pools (~213 MB per pool, against a 512 MB database limit and a 768 MB budget charge)
Page-load latency	production median 362 ms (p90 405, n=13); own pool 320 ms (p90 370, n=17)
Forced scale-down	10 → 1 pools in 10.2 s (each removal ~0.3 s); pool memory 2,133 → 218 MB
Failed page loads	40 first-attempt failures, all while a pool was built or removed; every one loaded on retry

Because every pool's PHP runs in production_eshop on this layer, the diverted sessions' CPU load shows on production's container, not on the pool containers.

How these reports were made trustworthy — thirteen rounds of audits against pool_manager's event log and the fixes they led to — is described in Validation Rounds.

Exploit test-scenario — detection rate per attack class

Each seeded CVE is replayed from a freshly reset threat state (Redis and the proxy's shared dicts flushed) so its score is that exploit's own isolated value, read back from the diagnostic X-Risk-Score / X-Route-Target headers. Rates below are from a recorded WordPress-proxy-level run and use Wilson confidence intervals.

Attack class	Diverted to honeypot (any request)	Caught on the first request
CVE exploits	13 / 14 (93%)	13 / 14
Plugin / theme targets	14 / 16 (88%)	14 / 16
Generic attack patterns	15 / 18 (83%)	7 / 18 (rest on request 2–3)
WordPress-core probes	14 / 16 (88%)	6 / 16 (rest on request 2–3)
Backend-facing	26 / 29 (90%)	18 / 29

Service Start/Stop Reports

Starting or stopping a stack from the Workloads page records a services report: host CPU/RAM from /proc and per-container CPU/memory from the cgroup files every 0.25 s, container states from docker compose ps (~1.5 s) and Docker events, numbered event markers, Global Visualization screenshots before, during and after, and a Warnings section listing failed or unstarted containers, error lines logged during the recording, killed containers and missing host settings.

Recording	Window	Result
Edge start, 2026-10-17	58 s	all containers healthy at 19.2 s, every one-shot job exit 0, host CPU peak 35 %, no warnings
Edge stop, 2026-10-17	8 s	stack gone by 5.8 s, down done at 8.2 s, no container killed (24 s with five SIGKILLs before init: true)
Edge stop, database layer, 2026-10-18	8 s	no warnings, nothing killed
SIEM start, 2026-10-18	70 s	all healthy at ~58 s, host CPU peak 82 % at 56.8 s while Kibana starts, no warnings
Edge start, 2026-10-18	64 s	healthy, one warning: vector_outbound's first healthcheck was refused because the SIEM was still starting

A tile's red badge on Global Visualization counts every line with an error word since the dashboard's log tail started (including 50 lines of replayed scrollback); the report's Warnings count only lines logged during the recording, so the two numbers can differ.

Honest Assessment

Strengths: Pure/adapter split enables 23 unit test files. Comprehensive coverage of all pure modules. Dashboard validation catches NDJSON drift. Segmentation and exploit runs turn a live stack into reproducible, statistics-backed evidence on both deception layers.

Limitations: No end-to-end integration tests. Blind pentest requires manual setup. No CI/CD integration.