What is Crystalline?

Crystalline is a log aggregation and search server with a flexible query language that aims to provide the search experience of a SIEM solution without the management, license, or resource overhead of solutions like Splunk, ELK, Humio/LogScale, etc. It's designed to be easy to deploy and use for networks small enough to not need horizontal scaling.

Features

• A pipe-based query language that supports:
  • Selecting events from multiple indices at once by keywords
  • Dynamically extract fields from structured and unstructured data by parsing json or using regular expressions
  • Filter and aggregate events by field values
  • Perform aggregations on dynamically extracted data
• Time-series indices with configurable retention policies and tiered storage support
• Fast parallel indexing with zstd compression thanks to Tantivy
• Highly multithreaded with an async actor-like processing model
• Customizable HTTP api for ingesting and querying logs
• Native chart interface to visualize search results
• Scheduled searches and alerting
  • Configured with a simple TOML schema for automation
• Summary searches to perform aggregations or risk based alerting on large amounts of data
• Support for loading an initial configuration to bootstrap deployment
• Lookups against reference data such as IP hostnames (planned)

Example search aggregating SSH logins from systemd journals

select systemd(accepted ssh)
| extract message = /^\w+\s(?<method>\w+)\s\w+\s(?<user>\w+)\s\w+\s(?<remote>[^\s]+)/
| stats 
	values(remote) as remotes
	values(method) as methods 
	count()  by host