Transform data in a Grepr pipeline

A Grepr transform is a processing step that modifies, enriches, or restructures data as it flows through a pipeline. As records move through a pipeline, transforms allow you to parse structured and semi-structured data, filter records, and use a familiar SQL interface to perform more complex transformation of records as they move through a pipeline. Transforms are configured as pipeline steps that process data sequentially. Each transform receives input data, applies its logic, and passes the result to the next step in the pipeline.

Most transforms are available in the Grepr pipelines UI. Some transforms are available only in the Grepr REST API or provide more capabilities when used with the API, such as filters and branching the processing of records based on query matches.

The following sections describe the transforms available in Grepr pipelines.

Transform semi-structured records with the Grok parser

The Grok transform parses unstructured text using pattern matching with regular expressions and named capture groups. Use this transform to extract fields from log lines, error messages, and other text-based data where you need to identify specific patterns and convert them to structured fields.

See Parse and transform semi-structured data using the Grepr Grok parser.

Convert JSON messages into structured objects

The JSON parser transforms JSON strings into structured JSON objects. This transform handles nested objects, arrays, and various data types, making it useful for processing API responses, configuration data, and structured log entries.

See Transform JSON strings into JSON objects.

Restructure JSON objects with a remapper

The remapper transform moves or copies attributes to top-level fields or to tags in a JSON object. In the pipelines UI, a remapper with customizable default attributes is automatically added to a pipeline.

See Remap JSON attribute fields to top-level log event fields.

Transform records with SQL

The SQL operation allows you to use ANSI standard SQL queries to filter, transform, and enrich data records. The SQL operation provides a familiar SQL interface to perform a number of data processing and transformation tasks.

See Transform events with the SQL operation.

Filter records based on conditions

The Filter transform includes or excludes data records based on specified conditions. You can create filters using several query interfaces, such as the Datadog query syntax, Splunk’s Search Processing Language, or the New Relic query syntax.

In the Grepr pipelines UI, filters are located in predefined locations in the pipeline. To have more flexibility for where filters are included in a pipeline, use the REST API. See LogsFilter.

Transform and enrich log messages with the REST API

You can use the LogTransformAction in the Grepr REST API to specify actions that transform log messages, such as adding or removing attributes and tags. See LogTransformAction.

Route the processing of messages based on query matches

You can route log messages to be processed by different pipelines based on query matches. Because the pipelines UI doesn’t support branching, Grepr recommends creating multiple pipelines that ingest data using the same source, and then use filters to route data to the correct pipeline.

You can also use the REST API to create more complex pipeline configurations. See LogsBranch.