Home › Docs ›Observability › SIEM & Log Pipeline

SIEM & Log Pipeline

How logs are processed and analyzed using SIEM tools

The SIEM path is intentionally one-directional. The edge writes structured events and ships them out over mTLS; the SIEM host indexes and visualises them. Nothing in the request path waits for Elasticsearch or Kibana to answer.

Outbound only One direction The edge opens the connection and ships; the SIEM never connects into the edge network. vector_outbound → vector_inbound

Transport Mutual TLS A private CA signs both ends; the agent certificate is copied to the edge before shipping starts. gen_elk_certs.sh

VRL enrich Typed fields Security lines are parsed into real fields so dashboards can aggregate on route, score and decay state. per-type daily indices

Data Flow

reverse_proxy (Lua)  → SECURITY_EVENT lines in the Nginx error log
                     → security access log (decision_trace fields)
Suricata             → eve.json
        ↓ edge Vector shipper (vector_outbound), tags log_type
        ↓ mTLS
SIEM Vector aggregator (vector_inbound) → VRL enrich → Elasticsearch honeypot-{log_type}-YYYY.MM.DD
        ↓
Kibana dashboards (honeypot-* data view)

The request-side pipeline that produces these events is in Architecture → Request Pipeline. To check that the hop is actually delivering, use the dashboard's Log Pipeline page.

Transport & mTLS

The SIEM host (siem/) and the honeypot host (ids/) are a two-host split — see Architecture for why. The Vector hop between them is secured with mutual TLS:

siem/certs/root-ca/gen_elk_certs.sh creates a CA and signs certificates for es01, kibana, vector and vector-agent. The vector-agent client certificate is copied to the honeypot host's ids/vector/certs/ before its shipper can connect.
The vector server certificate carries a Subject Alternative Name list so the agent can connect by hostname (host.docker.internal for single-host testing) or by raw IP (VECTOR_HOST in a real two-host deployment).
Vector's Elasticsearch sink writes to honeypot-{log_type}-%Y.%m.%d, using the log_type field the edge shipper tags on every event, so each log type and day lands in its own index.
A VRL enrich transform in vector/vector.yaml parses the nginx_security lines into real, typed fields (route, risk_score, suspicious, the decision fields above) instead of one opaque message string.

The edge never blocks on the SIEM

Vector buffers locally and the sink retries, so an Elasticsearch or Kibana outage shows up as ingestion lag, never as a failed or slowed request in the honeypot's path.

Pipeline Contracts

Events are JSON lines. log_security_event() emits a single JSON object per line, prefixed so Vector can filter without parsing unrelated Nginx errors.
Arrays are arrays. Empty Lua tables are tagged with cjson.empty_array_mt, so they encode as [] instead of and do not trigger Elasticsearch mapping conflicts.
Retries never block the edge. Vector buffers locally; a SIEM outage shows up as lag, not as failed requests.
Dashboards are code. Saved objects are generated from Python and committed as NDJSON, which makes reviews and rollbacks possible.
Exports preserve manual work. If a panel was edited live, export it back to NDJSON instead of rebuilding over it (see Building and Exporting).

Security Access Log

Besides the JSON events, every request is written to /var/log/nginx/security.log with the routing decision attached. The format in nginx.conf:

log_format security '$remote_addr - $remote_user [$time_local] "$request" '
                   '$status $body_bytes_sent "$http_referer" '
                   '"$http_user_agent" "$http_x_forwarded_for" '
                   'session_id="$session_id" route="$route_decision" '
                   'risk_score="$risk_score" suspicious="$suspicious_activity" '
                   'request_score="$request_score" decayed_score="$decayed_score" '
                   'offenses="$offenses" decay_state="$decay_state" '
                   'uri_class="$uri_class" patterns="$threat_patterns" '
                   'reason="$route_reason" cache_status="$upstream_cache_status"';
access_log /var/log/nginx/security.log security;