Splunk

A Splunk integration can be added via HEC in both the UI and the API.

Requirements

Here are the requirements for setting up a Splunk integration in Grepr.

1. HTTP Event Collector (HEC) Token: You'll need an HEC Token from your Splunk instance. See the instructions on how to create an HEC token in Splunk at https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector (opens in a new tab).

2. Collector Configuration: You need to configure your data collector to send logs to Grepr's HEC endpoint. We currently support Fluent Bit, Fluentd, OpenTelemetry Collector, and the Splunk OpenTelemetry Collector helm chart. You can find instructions for configuring these collectors below.

3. Splunk Enterprise Access: Grepr supports Splunk Cloud without requiring any further configurations. However, for Splunk Enterprise, Grepr SaaS requires access to your Splunk Enterprise instance. Contact us for help if you're running Splunk Enterprise.

Sending data to Grepr

To send data to Grepr from the Splunk collector, you need to configure the collector to use the ingestion URL. You can get this ingestion URL from the pipeline's detail view, by going to the "Sources" section as shown below.

The Grepr ingestion hostname is generally formatted as: <integration-id>-<org-id>.ingest.grepr.ai:443. The Grepr UI will format the full endpoint URL and make it available for you to copy when you set up the integration. Otherwise, the integration-id is the ID of the Splunk integration you wish to use (available via the API) and the org-id is your organization's ID.

Fluent Bit

Fluent Bit can forward logs using its native splunk output plugin. The plugin constructs the final endpoint path, so you only need to provide the hostname.

Configuration (fluent-bit.conf):

[OUTPUT]
   Name              splunk
   Match             *
   Host              <integration-id>-<org-id>.ingest.grepr.ai
   Port              443
   TLS               On
   TLS.Verify        On
   Splunk_Token      <hec-token>

For any additional information about the configuration file and other settings, please see https://docs.fluentbit.io/manual/pipeline/outputs/splunk (opens in a new tab).

Fluentd

Fluentd can send logs to Grepr using the splunk_hec output plugin. You will need to split the ingestion URL from the Grepr UI into its respective hec_host, hec_port, and hec_endpoint components.

<match your_tag.**>        # Adjust your match pattern as needed
 @type                     splunk_hec
 protocol                  https
 hec_host                  <integration-id>-<org-id>.ingest.grepr.ai
 hec_endpoint              services/collector/event
 hec_port                  443
 hec_token                 <hec-token>

 <buffer>                  # Buffer configuration is recommended for production
   @type memory            # Or file for persistence
   flush_interval          # Adjust as needed
   chunk_limit_size        # Adjust as needed
 </buffer>                 # Add other buffer parameters as needed
</match>

For any additional information about the configuration file and other settings, please see https://github.com/splunk/fluent-plugin-splunk-hec#readme (opens in a new tab).

OpenTelemetry Collector

The OpenTelemetry Collector can forward logs to Grepr using the splunk_hec exporter which is packaged in the otel/opentelemetry-collector-contrib distribution of the collector. The endpoint should be the full ingestion URL from the Grepr UI.

Configuration (collector.yaml):

exporters:
 splunk_hec:
   token: <hec-token>
   logs:
     endpoint: https://<integration-id>-<org-id>.ingest.grepr.ai:443/services/collector/event

service:
 pipelines:
   logs:
     receivers: [otlp] # Replace with your actual receiver(s)
     processors: [batch] # Replace/add your actual processor(s)
     exporters: [splunk_hec]

For any additional information about the configuration file and other settings, please see https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/splunkhecexporter#readme (opens in a new tab).

Splunk OpenTelemetry Collector helm chart configuration

To configure this collector:

1. Set splunkPlatform.endpoint to the Grepr ingest URL.
2. Set splunkPlatform.token to the HEC token.
3. Set splunkPlatform.logsEnabled to true.

splunkPlatform:
  endpoint: <Ingest URL>
  token: <HEC token>
  logsEnabled: true

Visit https://github.com/signalfx/splunk-otel-collector-chart (opens in a new tab) for more information on deploying Splunk collector using helm charts.

Creating a Splunk Integration in Grepr

To create a Splunk integration, follow the instructions below.

1. Go to the Integrations page in the Grepr UI and click on the Add new button in front of Observability Vendors.

2. Select Splunk from the list of vendors.

3. Enter a name for the integration.

4. Enter your Splunk HEC/S2S token.

5. REST API Search Bearer token (optional) – allows Grepr to run on-demand searches against your Splunk instance to fetch exceptions for pipeline configuration.

6. Filter query (optional) – narrow the scope of logs (e.g. service:abc).

Setting up a Splunk Sink

Once you have created a Splunk integration, you can add a Splunk sink to your pipeline to send processed logs to your Splunk instance. To set up a Splunk sink:

Configuration Fields

1. Sink: Select the Splunk integration you want to use from the dropdown. This should be one of the Splunk integrations you've previously created.

2. Name: Enter a descriptive name for this sink. This name will be used to identify the sink within your pipeline configuration.

3. Additional tags: Specify comma-separated tags to add to all log messages sent through this sink. These tags can be used for filtering and organizing logs in Splunk. For example: processor:grepr,pipeline:my_pipeline,environment:production.

Index Configuration

The Index Configuration section allows you to control which Splunk index your logs are sent to. Grepr follows a specific index routing logic to determine the destination index for each log:

Index Routing Logic

1. Index override takes priority: If an 'Index override' is provided, all logs flowing to this sink will use this index, regardless of any other index information.

2. Log-level index field: If no 'Index override' is provided and an index field exists on the log (from an agent or transformations), that index will be used.

3. Sink default index: If no 'Index override' is provided and no index field exists on the log, the 'Default index' specified in this sink configuration will be used.

4. HEC token default: If none of the above conditions are met, the log will be routed to the default index configured on the HEC token in your Splunk instance.

Configuration Fields

4. Index override: Forces all logs flowing to this sink to use the specified index, overriding any index information present in the log data. Leave blank to allow index routing based on log content or the default index.

5. Default index: Serves as a fallback index when no index override is configured and no index field exists on the log data.