Configure Datadog connections for integration with Grepr

The Grepr Datadog integration enables you to ingest logs into Grepr using a Grepr source, process them with a Grepr pipeline, and then send the processed logs to Datadog using a Grepr sink.

To send logs to a Grepr pipeline before forwarding the logs to Datadog, you change your collector configuration to send the logs to a Grepr endpoint URL instead of a Datadog endpoint. The Grepr Datadog integration supports several Datadog-compatible collectors, including the Datadog Agent, Fluent Bit, Fluentd, and the OpenTelemetry Collector.

You can use the Grepr UI or REST API to add a Datadog integration. To learn how to create a Datadog integration in the Grepr UI and configure collectors to send logs to Grepr, see Use a Datadog integration in a Grepr pipeline.

To use the REST API, see the Datadog Integration endpoint.

Although the Grepr Datadog integration supports all Datadog sites, when you use the Datadog US1 site in the us-east-1 region, the Grepr SaaS offering uses an AWS PrivateLink connection to Datadog.
Although this page describes using an integration to create a pipeline for sourcing and sending data with a single observability platform, your pipelines can use multiple integrations to source and sink data for different platforms. For example, you can create a pipeline with a Datadog integration that’s used to source data from Datadog agents, and a Splunk integration and sink that sends the pipeline output to Splunk.

Requirements

Creating and configuring a Datadog integration requires the following:

A Datadog API key for your account. You must use the same API key in both your Grepr integration and collector configuration.
(Optional) A Datadog application key for your account. An application key is required only to enable Grepr’s automatic exception parsing feature. See Exception Parsing. An application key also allows you to use the Datadog integration as a cloud source.

To learn more about Datadog keys, see API and Application Keys .

Use a Datadog integration in a Grepr pipeline

Use the following steps to create a Datadog integration in the Grepr UI and use it in a pipeline to source and sink data.

Step 1: Create a Datadog integration

To create a Datadog integration in the Grepr UI:

Go to the Integrations page in the Grepr UI and click Add new next to Observability Vendors.
In the Add Observability Vendor dialog:
- In the Type menu, select Datadog.
- In the Name field, enter a name for the integration.
- In the Site menu, select the parameter matching your Datadog site.
- In the API Key field, enter your Datadog API key.
- (Optional) In the Application Key field, enter your Datadog Application key. The application key is required only to import exceptions from Datadog or to use it as a cloud source.
Click Create.

Step 2: Add a source to your pipeline

To add a source to your pipeline that uses the Datadog integration:

In the Grepr UI, go to the overview page for an existing pipeline or create a new pipeline from the homepage.
In the left-hand navigation menu on the pipeline overview page, click Sources.
In the Add source dialog, select the Datadog integration in the Source menu.
(Optional) In the Name field, to override the Grepr-generated name, enter a new name for the source.
Click Submit.

Step 3: Configure log forwarding to Grepr

To send data to Grepr, you configure your collector or agent to send the data to the ingestion URL for your Grepr integration, rather than sending it to your observability vendor’s platform. To find the ingestion URL when using the Grepr UI to configure your pipeline, go to the details view for the pipeline, click Sources in the left pane, and copy the Ingest URL.

Accessing the Grepr ingestion URL in the UI.

When you use the API, you can construct the ingestion URL using the following format:

https://<integration-id>-<org-id>.ingest.grepr.ai

Replace <integration-id> with the identifier of the vendor integration used by your pipeline, and replace <org-id> with the identifier for your organization.

To see more information and configuration examples, select the tab for your preferred log collector:

Datadog Agent

Forwarding logs to Grepr from Datadog agents requires adding or updating settings in the agent configuration. How you configure these settings depends on how your agents are deployed:

Standalone agent: Update the datadog.yaml file with the following settings:
- Set logs_config.logs_dd_url to the ingestion URL.
- Set logs_config.force_use_http to true.
- Set logs_enabled to true.
- Set api_key to your Datadog API key.
- (Optional) Set container_collect_all to true to collect logs from all containers.
Docker agent: Set or pass the following environment variables:
- Set DD_LOGS_CONFIG_LOGS_DD_URL to the ingestion URL
- Set DD_LOGS_CONFIG_USE_HTTP to true.
- Set DD_LOGS_ENABLED to true.
- Set DD_API_KEY to your Datadog API key.
- (Optional) Set DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL to true to collect logs from all containers.

The Grepr platform supports only HTTP and HTTPS. TCP is not supported. Setting the logs_config.force_use_http and DD_LOGS_CONFIG_USE_HTTP settings to true enforces HTTP/HTTPS transport for Datadog agents.

To learn more about Datadog support and configuration for agent transport, see Agent Transport for Logs

To learn more about standalone agent configuration, see Host Agent Log collection .

To learn more about configuring Docker and other container agents, see Container Monitoring .

Fluent Bit

Fluent Bit can forward logs using the Datadog output plugin. To configure the plugin, set the Host value using the ingestion URL from the integration and set the apikey value to your Datadog API key. The following is an example OUTPUT section from a Fluent Bit configuration file (fluent-bit.conf):


[OUTPUT]
   Name        datadog
   Match       *
   Host        <ingestion-url>
   TLS         On
   apikey      <dd-api-key>
   # Optional: Specify service, source, tags, host
   # dd_service
   # dd_source
   # dd_tags
   # dd_hostname

To learn more about configuration settings for Fluent Bit, see the documentation for the Fluent Bit Datadog output plugin .

Fluentd

FluentD can forward logs to Grepr using the fluent-plugin-datadog output plugin. To configure the plugin, add a match directive to the FluentD configuration. The following is an example match directive from a Fluentd configuration file (fluentd.conf):


<match your_tag.**> # Adjust your match pattern as needed
 @type datadog
 api_key <dd-api-key>
 host    <ingestion-url>
 
 # Optional: Specify service, source, or tags
 # service
 # source
 # tags
 
 # Buffer configuration is recommended for production
 <buffer>
   @type memory # Or file for persistence
   flush_interval # Adjust as needed
   chunk_limit_size # Adjust as needed
   # Add other buffer parameters as needed
 </buffer>
</match>

Replace:

your_tag.** with the tag of the logs you want to forward to Grepr.
<dd-api-key> with your Datadog API key.
<ingestion-url> with the ingestion URL from your Datadog integration.

To learn more about configuration settings for Fluentd, see the documentation for the Fluentd Datadog output plugin .

OpenTelemetry Collector

The OpenTelemetry Collector can forward logs to Grepr using the Datadog exporter. In the OpenTelemetry collector configuration (collector.yaml), set <ingestion-url> to the ingestion URL from your Datadog integration and <dd_api_key> to your Datadog API key:


exporters:
  datadog/exporter:
    api:
      key: <dd_api_key>
      fail_on_invalid_key: false
    logs:
      endpoint: <ingestion-url>
 
service:
  pipelines:
    logs:
      receivers: [otlp] # Replace with your actual receiver(s)
      processors: [batch] # Replace/add your actual processor(s)
      exporters: [datadog/exporter]

See Install and Configure the OpenTelemetry Collector .

Enable selective reduction with a Datadog integration

When you include a Datadog application key in your integration configuration, Grepr automatically tracks all log monitors configured in your Datadog account. By parsing queries used in these monitors and making them available as exceptions to the Log Reducer, Grepr can determine which messages should not be processed by the reducer.

To see a full list of exceptions from the parsed alerts, in the Observability vendors section of the Integrations page in the Grepr UI, click on the cloud sync icon under Actions.

List of Datadog parsed alerts

To learn more about Grepr log reduction, see How does Grepr reduce log volumes?.

Limitations

The Datadog integration supports only HTTP/HTTPS connections. TCP connections are not supported.
The Datadog integration does not support the use of Datadog log pipelines . Grepr processes and routes all logs before sending them to Datadog.