Manage Grepr pipelines with Terraform

You can use the Grepr Terraform provider to manage Grepr pipelines using infrastructure as code (IaC) principles. With the Grepr Terraform provider, you can define, deploy, and manage your pipelines in a consistent and repeatable manner.

Requirements

• The option to build the Terraform provider from source requires Go version 1.21 or later.
• The provider uses OAuth2 client credentials for authentication. To obtain the required credentials, contact support@grepr.ai. The support team will send the following credentials:
  • client ID: A unique identifier for your organization.
  • client secret: A secret key to authenticate your organization.

Install the Grepr Terraform provider

You can install the Grepr Terraform provider by building from source.

The following commands clone the Terraform provider repository, compile the provider, and install it to ~/.terraform.d/plugins/registry.terraform.io/grepr-io/grepr/<VERSION>/<OS_ARCH>/:

git clone https://github.com/grepr/grepr-terraform.git
cd grepr-terraform
make install

Configure the Grepr Terraform provider

After the installation completes, add the provider to your Terraform configuration. Ensure the version field matches the provider version you installed.

terraform {
  required_providers {
    grepr = {
      source  = "grepr-io/grepr"
      version = "0.0.1"
    }
  }
}

Authenticate the Grepr Terraform provider

The provider uses OAuth2 client credentials for authentication. To learn how to obtain the required credentials, see the Requirements.

To learn more about authenticating to the Grepr platform, see Authenticate to Grepr REST API.

provider "grepr" {
  host          = "https://<my-org>.app.grepr.ai/api"
  client_id     = var.<grepr-client-id>
  client_secret = var.<grepr-client-secret>
}

Replace:

• <my-org> with your organization name.
• <grepr-client-id> with the client ID provided by Grepr.
• <grepr-client-secret> with the client secret provided by Grepr.

You can also use environment variables for authentication:

• GREPR_HOST: The Grepr API host URL.
• GREPR_CLIENT_ID: Your OAuth client ID.
• GREPR_CLIENT_SECRET: Your OAuth client secret.

Import an existing pipeline

To use Terraform to manage an existing pipeline, use the Terraform import command:

terraform import grepr_pipeline.example <pipeline-id-or-name>

Alternatively, creating a grepr_pipeline resource in Terraform with the name of an existing pipeline is the equivalent of a Terraform import. Creating a grepr_pipeline resource with a duplicate name does not cause a failure, but instead brings the existing pipeline under Terraform management. You can also use this to recover from partial failures where a pipeline was created but not tracked in state.

For more information about creating pipelines programmatically, see Create and manage jobs with the Grepr REST APIs.

Example: create a pipeline

resource "grepr_pipeline" "example" {
  name = "my_pipeline"
 
  job_graph_json = jsonencode({
    vertices = [
      {
        type          = "datadog-log-agent-source"
        name          = "source"
        integrationId = var.datadog_integration_id
      },
      {
        type = "grok-parser"
        name = "parser"
        grokParsingRules = ["%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"]
      },
      {
        type      = "logs-iceberg-table-sink"
        name      = "sink"
        datasetId = var.dataset_id
      }
    ]
    edges = ["source -> parser", "parser -> sink"]
  })
 
  desired_state = "RUNNING"
 
  tags = {
    environment = "production"
    team        = "platform"
  }
}

Resource attributes

The grepr_pipeline resource supports these attributes: