Event ingestion

When ingested by an input, an event will be routed to an Index, which is responsible for indexing the event and storing it in a bucket. The index will filter events so that it only ingests new events that would be placed into a primary/hot bucket.

The input will parse all events in the request it received and handle timestamp extraction before routing the events. An input will always send events to the same index, and an index can have multiple inputs.

As buckets age out they will be moved from primary/hot to secondary/warm and then finally tertiary/cold storage based on the retention policy of the index.

Buckets for indices are stored under /data (or /var/lib/crystalline/data when not in a container).