Validation Rounds

The Segmentation Validation and services start/stop reports were not written once and trusted. Between 2 and 18 October 2026 they went through thirteen rounds (the systematic issue audits ran from round 3, 16–18 October): run the stack, export the PDF, audit the PDF line by line against the system's own event logs, write every discrepancy into an issues file, fix it, and run again. This page summarises what those rounds found. The raw material is kept in the repository's reports/ folder, one numbered directory per round.

The Report-Driven Loop

1. Run. Start both stacks from the dashboard (each start and stop records its own services report), then run Segmentation Validation with 10 windows, some delayed so they attack only after pools were pre-built for them.
2. Audit. Read the exported PDF against independent sources — pool_manager's event log (honeypot_pool:events in Redis), the pool registry, container logs and Elasticsearch — and mark each issue confirmed (seen in a log or registry) or inferred (from the PDF alone).
3. Record. Write the issues file next to the PDF (pool_test_report_issues*.md, services_start_*.md, …), grouped into stack problems, data correctness, charts and layout, each with a severity and a suggested fix.
4. Fix and track. Fix each item with tests, and add a status table to the issues file that names the commit — or says plainly why an item was only surfaced and not changed.
5. Re-run. The next round's file opens with what the previous fixes changed, then lists only new issues.

Round by Round

Round	Reports	Main findings	Outcome
1	Exploit report, five-window pool tests, start-ups (2–7 Oct)	First scaling runs; under a memory cap (1,058 MB free, 2,560 MB needed) pools were shared round-robin — verdict SHARED EXPECTED	Baseline
2	Start/stop with and without purge, pool tests with 5 and 10 windows, idle browsing (9 Oct)	Five windows each on their own pool (EXCLUSIVE); first delayed-window runs	Material shared with the supervisor
3	Pool test, SIEM and edge start-ups	A diverted window escaped to production by clearing cookies (TLS GREASE); a new pool handed to an abandoned session; budget stopped at a quarter of real use; the edge shipped no logs; Kibana dashboards never seeded	~35 items fixed or surfaced
4	Start/stop of both stacks, operator notes on the pool test	Event times wrong (created ≠ started, removed ≠ exited), sample gaps drawn as lines, stale screenshots; slow 3-pool pre-build (213 s); CPU shown as % of one core	POOL_FAST_CLONE, CPU normalised to all cores, memory breakdown table
5	Pool test (verdict EXCLUSIVE)	ERR_NETWORK_CHANGED during builds; 502 after scale-down; post-run polling kept pools busy; reverse-proxy traces logged and shipped twice	Router fixed; report learned to detect 502s
6	Pool test	No 502, reassignment HTTP 200; but the reassignment check leaked into the run's statistics and rebuilt pools shown as "failed"	Separate observation phase, clean-up after it
7	Pool test, edge start-up	HTTP 4xx answers retried as failed loads, re-sending each attack up to ~67 times; short borrows missed by routing samples	Retry only on connection errors
8	Pool test, edge stop	Tamper schedule diverted a delayed window before its own attack; tables split before screenshot rows	One forgery per window, pagination fixes
9	Pool test, edge start and stop	Black screenshots of near-empty pages; five containers SIGKILLed after 10 s on stop; nginx [info] lines counted as errors	init: true, level-aware error matching
10	Pool test (7 delayed), edge start and stop	Header-only screenshots from small views; one-shot jobs too short to sample; stop now 8 s instead of 24 s	docker events for job timing
11	First database-proxy-level pool test, start, stop	Verdict EXCLUSIVE on the database layer; builds of 13–17 s	Second layer validated
12	Dashboard gap analyses (two rounds)	No Overview page while its data was collected and discarded; no native event feed, session drill-down, Suricata page, alerting or pool view; then alerting blind to pipeline/pool/Suricata health	Sidebar grew from 13 to 20 pages
13	Pool test, both start-ups	Re-validation of segmentation on the WordPress-proxy level	Verdict EXCLUSIVE

Measured Progress

Numbers taken from the reports of successive rounds, on the same development host (20 logical CPUs).

Measurement	Early rounds	Later rounds
Pools that fit the 12 GB budget	8 (each charged its 1,536 MB container limit)	10 = POOL_MAX (charged 768 MB via POOL_RESERVE_MEM_MB)
Measured memory per WordPress-level pool	~420–430 MB	~469 MB (round 13)
Measured memory per database-level pool	—	~213 MB (round 11; the database alone)
Pool build time	~60 s, of which ~50 s re-applying storefront settings via wp-cli	median 27 s (WordPress level), 15 s (database level)
Pre-build for 3 delayed windows	213 s	29 s (database level, 2 built + 1 spare)
Wait on a borrowed pool (case A)	63–99 s	median 34 s (WordPress level), 9 s (database level)
Edge stack down	24 s (five containers killed after the 10 s grace period)	8 s, nothing killed
Post-scale-down outcome	502 Bad Gateway for most windows	All 7 reassigned, HTTP 200, same session

What the Reports Found in the Stack

Session redirection

• Escape by clearing cookies. A window diverted at 1:20 cleared its site data and spent about four minutes on the real shop as a new session. Chromium (QtWebEngine) puts a random GREASE value in each TLS ClientHello, so the passive recovery fingerprint changed per connection. Fixed with session_identity_rules.strip_grease().
• Pools handed to sessions nobody uses. The hand-off took the oldest entry of the waiting queue, and a session whose client had vanished kept its place for the 24 h assignment TTL. A live window waited 99 s instead of ~18 s. Fixed: a waiting session gets a pool only if its client sent a request within POOL_WAITING_ACTIVE_SECONDS (default 60).
• 502 after scale-down. Every nginx worker cached session → pool for 300 s, and a pool with a missing or stale health status was treated as healthy, so requests went to a removed container. Fixed: a cached pool not in the ready list is a cache miss.
• Tamper score diverting the wrong window. A delayed window received three tamperings in a row; two rejected forgeries (+50 each) reached the threshold before its attack. The test now spreads tamper events across windows and forges each window's cookie at most once — tampering alone must not divert.
• The test's own setup scored as suspicious. The window that looked up product IDs through the Store API reached score 65 before attacking, close to the threshold.

Resource delegation

• Budget by reservation, not by use. Scaling stopped at 8 pools with ~3.4 GB used of a 12 GB budget. POOL_RESERVE_MEM_MB / POOL_RESERVE_CPUS now set the per-pool charge; the host-level checks still stop growth if the host really runs short.
• A spare built before the compose pools existed. Before pools 1–3 became healthy, pool_manager saw zero free pools and built a runtime spare, taking budget and adding a 60 s build to the busiest part of the start. Compose pools that are still starting now count as on their way.
• Scale-down left abandoned sessions' pools running. The forced scale-down released only each window's current session. It now releases every session a window held during the run.
• Slow builds. A runtime pool already clones the configured production database, then re-applied the same storefront settings via wp-cli. POOL_FAST_CLONE (experimental, off by default) skips that step; raising POOL_PARALLEL_BUILDS builds several delayed windows' pools in one wave.
• Network churn. Each build adds containers and veth interfaces, and Chromium cancels in-flight loads on every interface change (ERR_NETWORK_CHANGED). Every such load succeeds on retry; in round 10 all 21 failures fell inside the 7-pool pre-build while windows kept browsing.

Start-up and shutdown

• Kibana dashboards never seeded. Kibana's healthcheck accepted the 200 OK of its temporary preboot server, the import got 503, curl without --fail exited 0, and cleanup deleted the failed job. Fixed with a stricter healthcheck and an import job that waits, retries and verifies; failed one-shot jobs are kept visible.
• Kibana at its memory limit. Kibana sat at exactly 1,024 MiB; KIBANA_MEM_LIMIT is now 2 GiB.
• Host settings. Redis warned that background saves may fail without vm.overcommit_memory=1 — and a failed save makes Redis refuse writes, which would stop sessions, threat state and the pool registry together. The dashboard now checks this and Elasticsearch's vm.max_map_count on every host (core/host_settings.py).
• Containers killed on stop. backup_service, log_capper, honeypot_content_sync, pool_manager and session_store ran a shell or Python as PID 1, which ignores SIGTERM without a handler, so Docker killed them after 10 s. init: true and exec redis-server cut down from 24 s to 8 s.

Logging and the SIEM

• The edge shipped nothing. vector_outbound belongs to the elk profile, but the dashboard always started compose with --profile '*', so it ran against the placeholder VECTOR_HOST, failed DNS and dropped events while the SIEM held 0 honeypot-* documents. The dashboard now scales the shipper to 0 while shipping isn't configured. On one machine, point VECTOR_HOST at host.docker.internal.
• Request traces shipped twice. nginx writes info-level traces both to error.log and to stdout, and Vector read both, so every line was parsed and shipped twice. Vector also tagged every error.log line as an error; it now carries the line's own level.
• Post-run traffic. After a report finished, each window's cart poll kept sending ~3 requests per second, keeping pools busy so idle scale-down never fired. Polling now stops with the run.
• Pre-warm noise. The proxy logged its pre-warm pass ~160 times per start, and the word "failed" in "0 failed" raised false warnings. It now logs one summary line, and error matching trusts nginx's own level tags.

What the Reports Found in Themselves

Problem	Effect	Fix
HTTP error answers treated as failed loads	QtWebEngine reports a 4xx page as loadFinished(False), so a 404 attack was re-sent up to ~67 times, inflating scores and evidence	Retry only connection-level errors; never retry an attack after an HTTP answer
Timeline merged a window's sessions	A window that lost its session was credited with outcomes of its old one	Timelines and cases per (window, session)
Owner = last window seen	Designed borrowing reported as cross-traffic leaks	Owner = first holder; explained borrows marked as expected
Pools shared with an abandoned session drawn green	Verdict counted only test windows	Shared when the Redis owner count exceeds the windows on it
Reassignment check counted in the run	False "borrowed" findings and builds shown as failed	Run ends at the scale-down request; the check is its own phase
"Started" meant created, "stopped" meant removed	Start order and job durations untrustworthy	Times from docker events / docker inspect
One-shot jobs shorter than the ~1.5 s poll	Jobs silently missing from charts	Follow docker events; list jobs too brief to sample
Killed containers reported as clean	Exit code 137 gone once compose removed the container	Exit codes from die events
Sample gaps drawn as straight lines; lines ramping from 0 before a container existed	Invented load	Lines broken at gaps, start at the first real sample
Host CPU as % of one core; the RAM peak labelled as CPU peak	Misleading load figures	% of all logical CPUs, one decimal; peak per series
Screenshots: stale, black, or header-only	Evidence that did not show the claim	Wait for a painted frame; minimum view size during export

Lessons

• Fingerprints must survive the client's own randomisation. A recovery fingerprint that hashes GREASE is not a fingerprint.
• Every cache needs an invalidation path. The 502s came from a worker-local route cache that a pool removal never touched.
• Queues need liveness. A waiting list without a last-seen time hands resources to clients that have gone.
• Budget on what is used, cap on what is allowed. Charging container limits fitted a quarter of the pools the host could carry.
• Tell server answers from network failures. Retrying a 404 changes the experiment.
• PID 1 is special. Without init: true or exec, SIGTERM is ignored and every stop costs the full grace period.
• A test harness can be the adversary. Setup requests, post-run polling and tamper scheduling all changed scores or load until they were isolated.