How does it compare to other log search applications?

Crystalline specifically targets small networks and self hosted infrastructure. It's not designed to scale horizontally, but rather to be easy to deploy and use on a single machine.

It aims to provide the 80% of functionality that users of software such as the ELK stack or Splunk actually use while being deployable as a single small executable that starts up and is ready to use in under a second. It aims to use minimal resources to allow for deployments in lab environments or on resource constrained devices.

When should I use Crystalline?

If you want a SIEM-like search experience

Crystalline has a flexible pipe-based query language that allows you to perform complex searches on your logs. It's designed to be able to perform complex transformations on both structured and unstructured data, allowing you to extract fields from json, parse unstructured log messages, and filter/transform/aggregate events using a fully dynamic schema that is defined at search time.

Example search getting event logs for Nginx containers out of systemd journals

Example search Aggregating millions of nginx events to show the number of requests per IP

If you're looking for a low overhead logging solution

Crystalline is built to be trivial to deploy and use, it's designed to be used as a "network's first centralised logging" solution. It's not designed to be a full-featured SIEM solution that requires a team to maintain, but rather a lightweight alternative that can be used to aggregate and search logs from multiple sources that can be deployed in minutes and left to run in the background until you have a reason to search for something in your logs.

If you want a free logging solution that isn't based on cloud services or elasticsearch/opensearch

Crystalline is designed to be self-hosted, it's not based on cloud services and it doesn't require you to run elasticsearch or opensearch as it handles all the data storage and indexing itself. It's also free and open source under the AGPL 3.0 license, which means you can use it without paying for a subscription or licensing fee.

When shouldn't I use Crystalline?

If you're looking for a metrics solution

Crystalline is not designed to be a metrics solution, it's designed to be a log aggregation and search server. It's not suitable for storing purely numeric data as there are better solutions that use columnar storage that is much more efficient for this use case such as prometheus or influxdb.

It may be extended in the future to handle columnar storage but that isn't happening any time soon.

If you need horizontal scaling and fault tolerance

Crystalline is not designed to scale horizontally, so it's not suitable for large networks with very high ingest or query volume; it also won't provide high availability on it's own or handle application-level fault tolerance.

It has been stress tested on a single machine ingesting over 100MB/s of logs but at this volume (>8TB/day!) it would be strongly recommended to use a clustered application with fault tolerance as a network that organically produces this much telemetry is going to require high availability and redundancy.