Configure integration with observability platforms and cloud storage
To support data ingestion and data output for your Grepr pipelines, you need to connect Grepr to external observability platforms and storage systems. To support these connections, the Grepr platform uses integrations. A Grepr integration provides the configuration details to connect to an external system, enabling data exchange between Grepr and that system.
After configuring an integration, you can use it in your Grepr pipelines to create sources to ingest data from agents and collectors, such as FluentD or the OpenTelemetry collector, and sinks to send data to an observability platform or storage.
Integration types
The Grepr platform includes two integration types:
- Observability platform integrations: Connections to monitoring, logging, and APM platforms, such as Datadog, Splunk, New Relic, and Sumo Logic.
- Cloud storage integrations: Connections to cloud storage. The Grepr data lake uses cloud storage to store your raw data and metadata.
Supported platforms and storage integrations
The Grepr platform includes the following integrations. For details on configuring and using an integration, refer to the linked documentation page. For a complete list of supported integrations, supported functionality, and details on vendor-specific capabilities with some integrations, see Grepr supported integrations and regions.
Datadog
The Datadog integration supports connecting to Datadog and sourcing data from several Datadog-compatible collectors.
Splunk
The Splunk integration provides connectivity to Splunk through Splunk’s HTTP Event Collector (HEC) or S2S over HTTP. This integration supports reading data from Splunk collectors and writing data to Splunk indexes. You can create both sources and sinks using this integration.
New Relic
The New Relic integration connects to New Relic’s observability platform. Use this integration to create sources that retrieve log data from New Relic collectors and sinks that send processed data to New Relic for monitoring and analysis.
See Connect Grepr to New Relic.
Sumo Logic
The Sumo Logic integration connects to Sumo Logic’s cloud-based log management and monitoring platform using a Sumo Logic HTTP Source. You can use the integration to create sources that ingest data from Sumo Logic collectors and sinks that send processed data to Sumo Logic.
See Connect Grepr to Sumo Logic.
Grafana Cloud
Grepr’s OpenTelemetry integration can be configured to connect with Grafana Cloud using the OpenTelemetry Protocol (OTLP). Use an OpenTelemetry integration to create sources that ingest data from OTLP-compatible collectors and sinks that send processed log data to Grafana Cloud.
See Connect Grepr to Grafana Cloud.
Google Cloud Platform (GCP)
The GCP integration enables you to stream logs from Google Cloud services to Grepr for processing and analysis, then forward the processed logs to any Grepr-supported observability platform or tool, such as Datadog, Splunk, or New Relic.
See Google Cloud Platform observability with Grepr.
Cloud storage
The Grepr data lake uses AWS S3 to store, manage, and support querying the raw data from your Grepr pipelines. Depending on your requirements, you can choose from two S3 integrations: a Grepr-hosted integration, which uses a Grepr-managed S3 bucket, and an integration that uses an S3 bucket in your account. Both integrations provide secure, managed, and efficient storage for your data, along with an optimized query interface.
To configure the Grepr-hosted integration, you only need to assign a name for the integration, and the Grepr platform deploys and manages the S3 bucket.
Using the integration with your own S3 bucket requires that you deploy and configure an S3 bucket in your AWS account, but it provides more control over your data. See Host a Grepr data lake with the AWS S3 integration.
Limitations
- Grepr drops incoming data from external integration sources with timestamps older than 48 hours.
- Grepr may split log events over 1MB, causing them to lose their original structured format. Each split event includes a common
grepr.splitOriginIdif you need to reconstruct the original message.