The security data pipeline layer, shipped lean. A Vector hot path and an OpenTelemetry distribution move data at throughput; receivers, sinks, and the OCSF VRL transform framework form the core. Per-vendor capability arrives as versioned, OCSF-mapped App Store apps, so the core stays small and every source normalizes to the same nested OCSF shape. Open pipeline, OCSF-native, no per-GB tax.
A lean core, capability from apps
Caver-Collector no longer bakes every vendor integration into its core. The core carries the pipeline machinery: receivers, sinks, and the OCSF VRL transform framework, nothing else. Everything vendor-specific, the parser, the field mapping, the detection content, lives in an app you install from the Caver App Store. Adding a source means installing its app, not upgrading the collector.
Two backends, one normalization
The hot path runs on your choice of engine, and both share the same OCSF normalization library, so you pick by team familiarity, not by feature gap.
remap transform runs Caver's VRL standard library: OCSF normalization, ATT&CK tagging, threat-intel and CVE enrichment, HMAC tokenization.caver-otelcol) that speaks native OTLP and drops into infrastructure you already run. Caver OTTL functions bring the same enrichment and OCSF helpers to the transform processor.Receivers and inputs
The core exposes 21 receivers spanning push and pull: Splunk HEC and Universal Forwarder S2S, syslog, raw TCP and UDP, file tailing, generic webhooks, native OTLP, Kafka, Kinesis Firehose, and S3 via SQS. A single api_poll receiver drives 70+ SaaS adapters, and industrial protocol receivers decode Modbus, DNP3, EtherNet/IP, and S7comm directly (see Caver Industrial).
hec · uf_compat (S2S) · syslog · tcp_line · udp_packet · file_tail · webhook · otlp_http · kafka_consumer · s3_sqs · windows_event_log · api_poll · modbus_tcp · dnp3 · s7commNormalization to nested OCSF
Transforms are where raw vendor data becomes queryable security data. The VRL framework maps each source to nested OCSF, real nested structs rather than flattened columns, then routes the event at ingest into the right CLASS index by data model (endpoint, network, identity, cloud, saas_audit), preserving the original source as a sourcetype. That nested, consistent shape is what makes the lake compact to store and fast to query.
The same layer does the data-engineering work inline: PII detection and format-preserving tokenization (FPE + HMAC), schema validation against OCSF, conditional routing and schema-aware drops, dedupe, sampling, and IOC bloom-filter lookups. Trim and shape before anything is stored, so you are not paying to index noise.
Sinks and destinations
The pipeline is open at the far end too. Caver-native exporters write OCSF Parquet straight to S3 or MinIO for the Caver lake, or push to the indexer over HEC and S2S. The same events can fan out to Elastic, Loki, Kafka, any OTLP sink, syslog, or plain files, routed by class, tenant, or content, with no per-GB tax on where they land.
caver_parquet (S3/MinIO lake) · caver_hec · caver_search_peer · s2s (indexer) · elasticsearch · loki · kafka · otlp_http · s3 · syslog · fileThe App Store model
Each app is a versioned bundle: source config, the OCSF VRL mapping, a schema binding to OCSF classes and a CLASS index, and CAVERN detection content, all under one SemVer so upgrades are explicit and reversible. Because every app normalizes to OCSF, apps compose: two vendors in one category land in the same CLASS index with a consistent shape. A deployment-server subscription pipeline tracks which apps each collector subscribes to; when an app publishes a new version, subscribed collectors pull and apply it on their next check-in, so a mapping fix or a new detection reaches every collector running that source without hand-editing configs.
Fleet and install-service
The collector installs as a first-class background service on every platform: systemd on Linux, launchd on macOS, and a native Windows service. One command registers, enables, and starts it. From there the deployment server manages the fleet centrally, so app subscriptions, mapping updates, and content roll out to every collector on check-in rather than box by box.
/etc/systemd, auto-restart, journald integration.Reliability
Delivery is at-least-once with bounded backpressure: when a downstream sink slows, the pipeline pushes back through the buffer rather than dropping events or exhausting memory. A disk buffer with a write-ahead log survives restarts and outages, and anything undeliverable lands in a dead-letter queue instead of vanishing.
Data is yours from receiver to sink. Normalize once to nested OCSF, route anywhere, and pay for compute and storage rather than a licence on every gigabyte that moves. The lake stays open, so the collector never becomes the lock-in.