Splunk

A Splunk integration can be added via HEC in both the UI and the API.

Requirements

Here are the requirements for setting up a Splunk integration in Grepr.

1. HTTP Event Collector (HEC) Token: You'll need an HEC Token from your Splunk instance. See the instructions on how to create an HEC token in Splunk at https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector (opens in a new tab).

2. Collector Configuration: You need to configure your data collector to send logs to Grepr's HEC endpoint. We currently support Fluent Bit, Fluentd, OpenTelemetry Collector, and the Splunk OpenTelemetry Collector helm chart. You can find instructions for configuring these collectors below.

3. Splunk Enterprise Access: Grepr supports Splunk Cloud without requiring any further configurations. However, for Splunk Enterprise, Grepr SaaS requires access to your Splunk Enterprise instance. Contact us for help if you're running Splunk Enterprise.

Sending data to Grepr

To send data to Grepr from the Splunk collector, you need to configure the collector to use the ingestion URL. You can get this ingestion URL from the pipeline's detail view, by going to the "Sources" section as shown below.

The Grepr ingestion hostname is generally formatted as: <integration-id>-<org-id>.ingest.grepr.ai:443. The Grepr UI will format the full endpoint URL and make it available for you to copy when you set up the integration. Otherwise, the integration-id is the ID of the Splunk integration you wish to use (available via the API) and the org-id is your organization's ID.

Fluent Bit

Fluent Bit can forward logs using its native splunk output plugin. The plugin constructs the final endpoint path, so you only need to provide the hostname.

Configuration (fluent-bit.conf):

[OUTPUT]
   Name              splunk
   Match             *
   Host              <integration-id>-<org-id>.ingest.grepr.ai
   Port              443
   TLS               On
   TLS.Verify        On
   Splunk_Token      <hec-token>

For any additional information about the configuration file and other settings, please see https://docs.fluentbit.io/manual/pipeline/outputs/splunk (opens in a new tab).

Fluentd

Fluentd can send logs to Grepr using the splunk_hec output plugin. You will need to split the ingestion URL from the Grepr UI into its respective hec_host, hec_port, and hec_endpoint components.

<match your_tag.**>        # Adjust your match pattern as needed
 @type                     splunk_hec
 protocol                  https
 hec_host                  <integration-id>-<org-id>.ingest.grepr.ai
 hec_endpoint              services/collector/event
 hec_port                  443
 hec_token                 <hec-token>

 <buffer>                  # Buffer configuration is recommended for production
   @type memory            # Or file for persistence
   flush_interval          # Adjust as needed
   chunk_limit_size        # Adjust as needed
 </buffer>                 # Add other buffer parameters as needed
</match>

For any additional information about the configuration file and other settings, please see https://github.com/splunk/fluent-plugin-splunk-hec#readme (opens in a new tab).

OpenTelemetry Collector

The OpenTelemetry Collector can forward logs to Grepr using the splunk_hec exporter which is packaged in the otel/opentelemetry-collector-contrib distribution of the collector. The endpoint should be the full ingestion URL from the Grepr UI.

Configuration (collector.yaml):

exporters:
 splunk_hec:
   token: <hec-token>
   logs:
     endpoint: https://<integration-id>-<org-id>.ingest.grepr.ai:443/services/collector/event

service:
 pipelines:
   logs:
     receivers: [otlp] # Replace with your actual receiver(s)
     processors: [batch] # Replace/add your actual processor(s)
     exporters: [splunk_hec]

For any additional information about the configuration file and other settings, please see https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/splunkhecexporter#readme (opens in a new tab).

Splunk OpenTelemetry Collector helm chart configuration

To configure this collector:

1. Set splunkPlatform.endpoint to the Grepr ingest URL.
2. Set splunkPlatform.token to the HEC token.
3. Set splunkPlatform.logsEnabled to true.

splunkPlatform:
  endpoint: <Ingest URL>
  token: <HEC token>
  logsEnabled: true

Visit https://github.com/signalfx/splunk-otel-collector-chart (opens in a new tab) for more information on deploying Splunk collector using helm charts.

Creating a Splunk Integration in Grepr

To create a Splunk integration, follow the instructions below.

1. Go to the Integrations page in the Grepr UI and click on the Add new button in front of Observability Vendors.

2. Select Splunk from the list of vendors.

3. Enter a name for the integration.

4. Enter your Splunk HEC/S2S token.

5. REST API Search Bearer token (optional) – allows Grepr to run on-demand searches against your Splunk instance to fetch exceptions for pipeline configuration.

6. Filter query (optional) – narrow the scope of logs (e.g. service:abc).