What is Crystalline?

Crystalline is a log aggregation and search server with a flexible query language that aims to provide the search experience of a SIEM solution without the management, license, or resource overhead of solutions like Splunk, ELK, Humio/LogScale, etc. It's designed to be easy to deploy and use for networks small enough to not need horizontal scaling.

Features

• A pipe-based query language that supports:
  • Selecting events from multiple indices at once by keywords
  • Dynamically extract fields from structured and unstructured data by parsing json or using regular expressions
  • Filter and aggregate events by field values
  • Perform aggregations on dynamically extracted data
• Time-series indices with configurable retention policies and tiered storage support
• Fast parallel indexing with zstd compression thanks to Tantivy
• Highly multithreaded with an async actor-like processing model
• Customizable HTTP api for ingesting and querying logs
• planned Scheduled searches and alerts
• planned Summary searches to perform aggregations or risk based alerting on large amounts of data
• planned Native chart interface to visualize search results
• planned Lookups against reference data such as IP hostnames

Example search aggregating SSH logins from multiple indices

select systemd(accepted ssh) syslog(accepted sshd)
| eval 
	host = lower(mvindex(split(if(HOSTNAME=*, HOSTNAME, host), "."), 0))
	message = if(MESSAGE=*, MESSAGE, trim(message))
| extract message = /^\w+\s(?<method>\w+)\s\w+\s(?<user>\w+)\s\w+\s(?<remote>[^\s]+)/
| stats count() values(host) unique(host) by user remote