Filter and transform events with actions
You can use actions in a Grepr pipeline to decide what happens to log events at a specific point in the pipeline. With actions, you can:
- Drop events, or keep only the events that match a condition.
- Drop late-arriving events.
- Process all or some events with SQL, and send the results to a later pipeline step.
Conditions use familiar query syntaxes, such as the Datadog-like syntax or the New Relic log query-like syntax. To learn more about the supported query syntaxes, see Query log data in Grepr.
Where actions run in a pipeline
Actions are available at three locations in a pipeline. Each location is named for the pipeline step it runs before:
| Location | Runs before | CLI phase |
|---|---|---|
| Pre-Parsing Actions | Parsing | pre-parser |
| Pre-Warehouse Actions | Masking and the Data Lake | pre-warehouse |
| Pre-Reducer Actions | Exceptions and the Reducer | pre-exceptions |
Choose the earliest location that has the fields you need. An event dropped at Pre-Parsing Actions isn’t parsed, stored, or reduced. An event dropped at Pre-Warehouse Actions is parsed but isn’t stored in the data lake. An event dropped at Pre-Reducer Actions is stored in the data lake but isn’t reduced or sent to your sinks.
Pre-Warehouse Actions run before masking. Conditions and SQL at this location see the original values of any fields that your masking rules replace.
Add actions to a pipeline
- In the Grepr UI, go to the details page for your pipeline.
- In the left pane, select Pre-Parsing Actions, Pre-Warehouse Actions, or Pre-Reducer Actions.
- Click Add Action, and then select one of the following:
- Condition (If/Else): Sends events down different branches based on a query. See Drop or keep events with conditions.
- SQL: Processes events with SQL. See Process events with SQL.
- Drop late-arriving events: Drops events that arrive too long after their timestamp. See Drop late-arriving events.
- To save the change to your pipeline, click Apply Changes.
Before you apply a change, you can test it against live data with the Pipeline Log Viewer. See Test pipeline changes with live edit.
Drop or keep events with conditions
A condition sends each event down exactly one branch. The first branch whose query matches the event handles it, and an event that matches no query goes to the ELSE branch.
- To add a condition, enter a query in the IF row.
- To test events against more queries, add ELSE IF branches. Branches are checked from top to bottom, so you can reorder them to change which query handles an event first.
- Each branch ends with either Drop, which removes the events from the pipeline, or the next pipeline step, which sends the events on. To change how a branch ends, click its ending and select the other option.
- To process the events on one branch with SQL, insert a SQL action into that branch.
For example, to drop debug events and keep everything else, add a condition with the query status:debug, end the IF branch with Drop, and end the ELSE branch with the next pipeline step.
Process events with SQL
A SQL action processes events with ANSI standard SQL queries. Each SQL action contains one or more statements. Each statement has a name and a SQL query that reads input events from the logs table or from an earlier statement by its name.
To send the results of a statement to a pipeline step, connect the statement to that step. Depending on the location of the action, you can send results to steps such as the parsers, the Data Lake, the Reducer, or the pipeline sinks. Statements that aren’t connected to a step are available only to later statements.
A SQL action doesn’t remove events from the pipeline. Events continue to the rest of the branch, and the results of connected statements are sent to the connected steps in addition.
The results of a SQL statement contain only the fields you select. To keep all fields from the original event, include the wildcard character (*) in your SELECT statement. For example, to transform the severity while keeping all other fields: SELECT *, CASE WHEN severity > 16 THEN 21 ELSE severity END as severity FROM logs.
To learn more about using SQL in Grepr pipelines, see Transform events with SQL.
Drop late-arriving events
You can drop events whose received time is later than their event timestamp by more than a maximum duration. Enter the maximum duration in the ISO-8601 format, such as PT48H for 48 hours. This is useful when timely data is critical, and late-arriving events could affect analysis or reporting.
The late-data rule applies to every event at that location, before any condition is checked.