Troubleshooting & Runbooks

Service Restart Runbook

1. Stop the affected service.
2. Clear stale state (queues, caches) if applicable.
3. Start the service.
4. Verify health endpoints and logs.
5. Confirm dashboards are operational.

Incident Response Runbook

• Detect: IDS alerts, high-risk IPs, reportable events.
• Triage: Correlate with session and pattern data from pattern_analyzer and router.lua decision traces.
• Contain: Apply rate limits / block lists via IP whitelist or AbuseIPDB reporting.
• Recover: Review logs and adjust rules.

Common Issues

• Suricata not producing alerts: Check rule reload and eve.json pipeline. suricata_rules.extract_alert() returns nil for non-alert events.
• Kibana dashboards stale: Re-run build_dashboards.py or export_dashboards.py.
• Timeline capture failures: Verify Kibana auth and embed configuration.
• IP reputation not gating (same-host testing): Suricata src_ip may be container-internal. Filter private IPs via is_private_ip().
• Fast.log vs eve.json: Do not regex-parse fast.log. Use structured eve.json.
• Session not recovering after cookie clear: Passive fingerprint binding (fp_session in Redis) recovers sessions. Check request_fingerprint() and bind_fingerprint_to_session().
• Decay not aging out: Check score_decay_half_life_seconds — if 0, decay returns stored peak unchanged. Check permaflag thresholds.
• False positives on legitimate traffic: Honeypot threshold raised from 50 to 80. Consider raising ip_reputation_cap above 50 only if clients don't share addresses.
• Pools not scaling or stuck building: The pool manager only grows within the budget (POOL_MAX, POOL_MAX_MEMORY_MB, POOL_MAX_CPUS) and above the free-memory floor (POOL_MIN_FREE_MEM_MB); at the ceiling new sessions share ready pools round-robin rather than failing. Check the manager's logs, the budget and POOL_PARALLEL_BUILDS. See Engine.
• Diagnostic headers missing during a test: X-Route-Target / X-Risk-Score are emitted only when INTERNAL_TEST_SECRET is set and the request carries a matching X-Internal-Test-Auth header (fail closed). This is intentional — see the test-header oracle.
• Host settings warning in a start report: Elasticsearch needs vm.max_map_count ≥ 262144 and Redis wants vm.overcommit_memory=1. core/host_settings.py checks both and the services report lists any that are missing; set them with sysctl -w (and persist them in /etc/sysctl.d/).
• Edge logs not reaching the SIEM: check that vector_outbound is running and healthy and points at the SIEM's Elasticsearch. A refused first healthcheck just after start is expected if the SIEM was started later.
• Kibana dashboards missing: the import runs only after Kibana leaves preboot. If dashboards are still missing, re-run the import job once Kibana answers, or build_dashboards.py.
• Pool builds slow: check POOL_PARALLEL_BUILDS and host memory headroom. Expect about 27 s per pool on the reverse-proxy layer and 15 s on the database layer; POOL_FAST_CLONE=1 is experimental.
• down slow or containers killed: a service without init: true ignores SIGTERM as PID 1 and is SIGKILLed after the grace period. Add init: true to the service.
• Failed loads or blank screenshots during a pool build: Chromium cancels requests on Docker network changes (ERR_NETWORK_CHANGED). Retry; this is not a routing fault. See Known Issues.
• Switching the honeypot layer: The reverse-proxy and database layers use different compose files (docker-compose.yml vs docker-compose.db-proxy.yml); the active layer is persisted in dashboard state. Bring the current stack down before switching. See Honeypot Layers.

Evidence to Collect

• Decision trace — the stage that fired and the accumulated score for the session.
• Suricata alert — the SID, signature, severity and source address, with the private-IP caveat in mind.
• Session state — whether the session was bound, which pool, and when the binding was written.
• Pool state — container health, the assigned database host and whether a build was in progress.
• Log window — the matching Nginx security events and the SIEM documents they became.

Commands that gather most of it, run from ids/:

# decision trace and security events for one client
docker compose logs --since 1h reverse_proxy | grep SECURITY_EVENT | grep '"ip":"203.0.113.7"'
docker compose exec reverse_proxy grep 203.0.113.7 /var/log/nginx/security.log | tail -20

# session and pool state
REDIS_PW="$(grep ^REDIS_PASSWORD= .env | cut -d= -f2-)"
docker compose exec -T session_store redis-cli -a "$REDIS_PW" --no-auth-warning GET honeypot_pool_session:<SID>
docker compose exec -T session_store redis-cli -a "$REDIS_PW" --no-auth-warning SMEMBERS honeypot_pool:ready
docker compose ps

# full research export (sessions, honeytokens, logs) as NDJSON/CSV
./scripts/collect_data.sh --compress

Escalation

Escalate to repository maintainers with logs, dashboard state, and alert context.