Lookups

Lookups are external data files used by the lookup search command to enrich events during a search.

Directory Structure

Lookup files are stored under the lookups directory inside CRYSTALLINE_EXTRA_DIR. The default extra directory is /var/lib/crystalline/extra, so the default lookup directory is:

/var/lib/crystalline/extra/lookups

For example:

extra/
  lookups/
    people.csv
    assets/
      hosts.csv

Loading

Lookup files are loaded when a search using the lookup command is prepared. For CSV lookups, the command path is resolved relative to the lookup directory:

| lookup csv("people.csv") name

Nested lookup files can be referenced with relative paths:

| lookup csv("assets/hosts.csv") host

Lookup paths cannot escape the configured lookup directory. If a path resolves outside CRYSTALLINE_EXTRA_DIR/lookups, the command fails and a warning is logged.

CSV Format

CSV lookups use the first row as column headers. The fields named in the lookup command are matched against these headers, and output fields are read from the remaining selected columns.

Duplicate CSV header names are rejected.