Manage Grepr resources with Terraform

Use the Grepr Terraform provider to create and manage integrations, datasets, and pipelines from your infrastructure configuration.

Requirements

To use the Grepr Terraform provider, you must have:

• Terraform or OpenTofu CLI version 1.0 or later.
• OAuth2 client credentials for authentication. To obtain the required credentials, create a service account in the Grepr UI. See Manage Grepr service accounts.

Install the Grepr provider

The Grepr provider is published to the OpenTofu registry . To install it, declare it in the required_providers block of your configuration. To use the OpenTofu CLI, use the following configuration:

terraform {
  required_providers {
    grepr = {
      source  = "grepr/grepr"
      version = "~> 1.2"
    }
  }
}

To use the Terraform CLI, you must use the fully qualified source address registry.opentofu.org/grepr/grepr:

terraform {
  required_providers {
    grepr = {
      source  = "registry.opentofu.org/grepr/grepr"
      version = "~> 1.2"
    }
  }
}

Then initialize your working directory to download the provider:

terraform init # Or tofu init

Configure authentication for the Grepr Terraform provider

To configure the required OAuth2 credentials for authentication, add a provider block to your Terraform configuration with the host, client_id, and client_secret fields:

provider "grepr" {
  host          = "https://<your-org>.app.grepr.ai"
  client_id     = var.grepr_client_id
  client_secret = var.grepr_client_secret
}

Replace:

• <your-org> with your organization name.
• var.grepr_client_id with a variable containing your service account client ID.
• var.grepr_client_secret with a sensitive variable containing your service account client secret.

Alternatively, you can use environment variables to set the required credentials. The provider automatically reads these environment variables if you set them:

export GREPR_HOST=https://<your-org>.app.grepr.ai
export GREPR_CLIENT_ID=<your-client-id>
export GREPR_CLIENT_SECRET=<your-client-secret>

Manage integrations and datasets

The provider also manages the integrations and datasets your pipelines use, starting with provider version 1.2.0. Each integration type has its own resource:

Reference their IDs from a pipeline so Terraform creates them first:

variable "datadog_api_key" {
  type      = string
  sensitive = true
}
 
resource "grepr_datadog_integration" "datadog" {
  name    = "terraform_datadog"
  site    = "datadoghq.com"
  api_key = var.datadog_api_key
}
 
resource "grepr_data_warehouse_integration" "warehouse" {
  name = "terraform_warehouse"
}
 
resource "grepr_dataset" "logs" {
  name           = "terraform_logs"
  integration_id = grepr_data_warehouse_integration.warehouse.id
}
 
resource "grepr_pipeline" "logs" {
  name = "terraform_logs"
  job_graph_json = jsonencode({
    vertices = [
      {
        type          = "datadog-log-agent-source"
        name          = "source"
        integrationId = grepr_datadog_integration.datadog.id
      },
      {
        type      = "logs-iceberg-table-sink"
        name      = "sink"
        datasetId = grepr_dataset.logs.id
      }
    ]
    edges = ["source -> sink"]
  })
}

Keep in mind:

• Secrets such as API keys are sensitive attributes, but Terraform stores them in state. Protect your state backend with encryption and access controls.
• Grepr never returns secrets, so Terraform writes them but can’t read them back and doesn’t detect a secret changed outside Terraform. Removing a secret from your configuration stops managing it without revoking it.
• Write values the way Grepr stores them, such as durations in hours (PT24H rather than P1D) and lowercase encodings. Grepr rewrites other spellings, and the next plan shows the difference.
• Deleting a dataset drops its stored data. Changing its integration_id replaces it, so the new dataset starts without the old data or table settings such as its promotion policy. Add lifecycle { prevent_destroy = true } to guard against accidental deletion. Table settings made outside Terraform are left alone.
• Import integrations and datasets by ID, for example terraform import grepr_dataset.logs DATASET_ID. Import can’t recover secrets, so the next apply writes the secrets you configure. Writing a Datadog API key is refused while a pipeline uses the integration, so stop the pipeline first or leave the key out of the configuration.

See the provider documentation  for each resource’s attributes.

Import an existing pipeline

To bring an existing pipeline under Terraform management, use the terraform import command with the pipeline’s ID or name:

terraform import grepr_pipeline.example <pipeline-id-or-name>

To automatically import an existing pipeline without using the import command, create a grepr_pipeline resource with the same name as the existing pipeline. The provider will detect and import the existing pipeline.

Tags

Starting with provider version 1.3.0, the tags that you configure are the pipeline’s complete set of tags. When you change them, Grepr updates the pipeline’s tags without creating a new pipeline version or redeploying the pipeline. You can change a pipeline’s tags only while the pipeline is running or stopped, so apply a tag change after any earlier update to the pipeline finishes. Tags that you leave out of the configuration are removed from the pipeline. To keep editing a pipeline that you created in the Grepr UI, include grepr-ui-managed = "true" in its tags. To keep a pipeline’s existing tags, including for a pipeline that you import, leave tags unset.

A pipeline can have up to 10 tags. Each key can be up to 64 characters long, and each value can be up to 256 characters long. Tag keys that start with __grepr__ are reserved.

Example: create a pipeline

The following example shows a simple pipeline that uses a Datadog log agent source, a Grok parser, and an Iceberg table sink. To improve maintainability and avoid conflicts with Terraform reserved symbols or keywords, Grepr recommends defining your pipeline’s job graph in a separate JSON file. You then reference this JSON file in your Terraform configuration. This example demonstrates how to define a pipeline job graph in a JSON file and reference the file in the Terraform configuration.

A common development workflow is to create a pipeline in the Grepr UI or API, test it and make changes until it meets your requirements, then copy the generated JSON to a file for use with your Terraform configuration. If you choose to create pipelines through the API, see Create and manage jobs with the Grepr REST APIs.

After you’ve completed development of your pipeline, to find its JSON definition in the UI, click Jobs in the menu bar, click the three-dot menu in the Actions column for your pipeline, and click Edit JSON to view and copy the JSON. Save the copied JSON to a file in your Terraform project, for example, pipeline.json:

{
  "vertices": [
    {
      "type": "datadog-log-agent-source",
      "name": "source",
      "integrationId": "<your-integration-id>"
    },
    {
      "type": "grok-parser",
      "name": "parser",
      "grokParsingRules": [
        "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"
      ]
    },
    {
      "type": "logs-iceberg-table-sink",
      "name": "sink",
      "datasetId": "<your-dataset-id>"
    }
  ],
  "edges": ["source -> parser", "parser -> sink"]
}

<your-integration-id> and <your-dataset-id> are the identifiers of the Datadog integration and dataset used by this pipeline. If you copied the JSON from an existing pipeline, these fields are already set to the correct value.

Then reference the JSON file in your Terraform configuration:

resource "grepr_pipeline" "example" {
  name           = "my_pipeline"
  job_graph_json = file("${path.module}/pipeline.json")
  desired_state  = "RUNNING"
}

Use terraform init to download the provider, terraform plan to preview your changes, and terraform apply to apply the configuration and create the pipeline. With the OpenTofu CLI, use the equivalent tofu commands:

terraform init
terraform plan
terraform apply

You can also define the job graph directly in the Terraform configuration using jsonencode:

resource "grepr_pipeline" "example" {
  name = "my_pipeline"
 
  job_graph_json = jsonencode({
    vertices = [
      {
        type          = "datadog-log-agent-source"
        name          = "source"
        integrationId = var.datadog_integration_id
      },
      {
        type = "grok-parser"
        name = "parser"
        grokParsingRules = ["%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"]
      },
      {
        type      = "logs-iceberg-table-sink"
        name      = "sink"
        datasetId = var.dataset_id
      }
    ]
    edges = ["source -> parser", "parser -> sink"]
  })
}

Resource attributes

The grepr_pipeline resource supports these attributes:

Read-only attributes