Event ingestion

Event ingestion in Crystalline happens via the /api/ingest/* api endpoints.

These endpoints are populated via configured inputs which accept a range of parameters. Each input will send any events sent to it to the index specified in it's configuration, each input will send events to exactly one index; but an index may be configured with many inputs.

Inputs handle the following operations:

• Batch decoding - splitting a request into multiple events
• Timestamp extraction and parsing
• Retention policy filtering

When a request is received by an input, it will first be split into seperate events, and have the timestamp extracted based on the input configuration. after the timestamp for an event has been validated; it will be evaluated against the retention policy of the index to ensure that the stamp falls with the retention period for hot buckets. If the event doesn't fall into this period it will be dropped.

All retained events will then be forwarding to a pool of workers for the target index to be routed to appropriate buckets.