Indexed columns

Grepr stores each raw log’s attributes as one JSON document and its tags as a map. A search that filters on one attribute reads every log’s whole attribute document to find a match.

An indexed column removes that cost for one field. Grepr writes the field into its own column in the dataset’s data lake table, so a search that filters on the field reads that column and skips the files that can’t contain a match. See The Grepr data lake.

Searches don’t change. The same search returns the same results whether or not a field is indexed, and runs faster when it is.

What Grepr indexes

Each dataset’s raw log table indexes the fields on two lists:

Indexed paths: attribute paths, such as http.method or kube.pod.name. An indexed path has a column for its value, and a path that holds numbers can also have a numeric column, so that range filters skip files too.
Indexed tag keys: tag keys, such as region or pod_name. An indexed tag key has a column for its value.

Grepr indexes exactly the fields on these lists, and always indexes the service and host tags. A new dataset indexes only service and host.

A dataset whose indexing settings haven’t been saved since Grepr began indexing only listed fields can still add columns for fields that aren’t listed, up to the limits it was set up with. To stop that, save its settings: the lists show every field the table already has columns for, so the save keeps those columns.

Choose the fields to index

In the dataset’s settings, add attribute paths to Indexed paths and tag keys to Indexed tag keys, then click Save. The lists show every field the table has columns for, so a save keeps those columns unless you remove the field.

For attribute paths:

A path matches exactly and is case-sensitive. Write the path as it appears in the log, for example http.status_code. A path doesn’t cover the paths under it, and wildcards aren’t supported.
Select Numeric to give a path its numeric column when you save, for values that are numbers written as strings. Otherwise, a path gets its numeric column when Grepr finds a number there.

For tag keys:

A key matches exactly.
To partition a table on a tag key, the key must be on the list, and it stays there while the table is partitioned on it.

Your organization sets how many attribute columns and tag columns a table can have, 20 of each by default. service and host don’t count. A column counts until it’s removed from the table, including after you take its field off the list. An attribute path you list while the table is at the attribute limit stays on the list, but isn’t indexed until the columns of fields you remove are removed from the table. You can’t list a tag key that would take the table past the tag limit, so to replace a key at the limit, remove the old key and save, then add the new key after its column is removed.

When changes take effect

A field you add gets its columns when you save, while the table is under its limit, so you can partition on a tag key immediately. Running pipelines start writing the columns within about a minute, without a restart. Logs written before then don’t have values in the new columns, and stay searchable through their attributes and tags.

Stop indexing a field

To stop indexing a field, take it off its list and save. Grepr stops writing the field’s columns within about a minute, and searches read the field from the log’s attributes or tags instead, so they return the same results.

Grepr removes the columns from the table 6 hours after you save. While the pipeline is receiving logs, the columns are removed at that time or within about an hour after it. A pipeline that receives no logs removes them when logs arrive again. Until then the columns still count toward your organization’s limit. Logs written before the removal stay searchable through their attributes and tags, but searches on that field no longer skip files using the columns.

If you add the field back before then, Grepr goes back to writing the same columns, and everything already indexed in them stays. After the removal, adding the field back gives it new columns, which cover only logs written from then on.

Collect file statistics on every column

A search skips a data file on an indexed column by using the file’s statistics for that column: the smallest and largest values the column holds in the file, and how many of them are null. By default, a data lake table collects these statistics on its first 100 columns only, counting top-level columns before nested fields. On a table with many indexed columns, the columns and fields past the first 100 have no statistics, so searches that filter on them read every data file.

To collect statistics on every column of a dataset’s raw log table or raw span table, in the dataset’s settings, expand Advanced, turn on Raw logs table or Raw spans table, and click Save. Both are off for a new dataset. Turning one off returns the table to collecting statistics on its first 100 columns. The raw span table’s built-in span fields keep their statistics either way. Grepr pins their statistics mode when a pipeline writing the dataset starts, and keeps any mode already set on the table for one of those fields.

Statistics on every column add about 18 bytes of table metadata for each indexed column in each data file.

A change applies to the data files that pipelines write after their next restart, and earlier when the table gets a new indexed column. Data files already written keep the statistics they were written with.

Change indexing through the API

Reading a table’s configuration returns its indexing policy under promotion, and the columns the table has under promotedAttributes and promotedTags. To change what the table indexes, send a policy to the promotion endpoint. The policy replaces what’s stored, so send both of its groups:

attributes: set excludeAllByDefault to true and cap to your organization’s attribute column limit, and list each path to index under include as a {"path": ..., "numeric": ...} entry. A path you leave off stops being indexed.
tags: set cap to 0 and list each tag key to index under include. To stop indexing a tag key that has a column, list it under exclude.

Include the revision you read: if the policy changed since then, the request fails with a 409 error, and you read it again before you retry. Without a revision, the request replaces the stored policy whatever it is. See the Datasets specification.

To change how long Grepr waits before removing the columns of a field you stop indexing, set columnRemovalDelay to an ISO-8601 duration from PT2M (2 minutes) to P30D (30 days). Without it, the delay is 6 hours. A new delay applies to fields you stop indexing from then on.

To turn file statistics on every column on or off for the logs_raw or spans_raw table, send {"enabled": true} or {"enabled": false} to the table’s column metrics endpoint. Reading the endpoint returns whether the setting is on. See Collect file statistics on every column.

Frequently Asked Questions

What is an indexed column in Grepr?

An indexed column is a log attribute path or tag key that Grepr writes into its own column in the dataset's data lake table, instead of leaving it only in the log's attribute document or tags. Searches that filter on the field read that column and skip files that can't match, which makes them faster.

Which fields does Grepr index?

Exactly the attribute paths and tag keys on the dataset's Indexed paths and Indexed tag keys lists, plus the service and host tags, which are always indexed. Grepr doesn't add columns for fields you haven't listed. A dataset whose indexing settings haven't been saved since Grepr began indexing only listed fields can still add columns for other fields until you save its settings.

Do my searches change when a field is indexed?

No. The same search returns the same results whether or not a field is indexed, and runs faster when it is.

How many fields can I index?

Your organization sets how many attribute columns and tag columns a table can have, 20 of each by default. service and host don't count. A column counts until it's removed from the table, including after you take its field off the list.

Can I index every path under an attribute, such as kube?

No. Each path on the list matches exactly, so list each path under kube that you search on, such as kube.pod.name.

How do I stop Grepr from writing a column?

Take the field off its list and save. Grepr stops writing the column within about a minute and removes it from the table 6 hours later. Add the field back before then to keep the column. The data already written stays searchable.

When do indexing changes take effect?

A field you add gets its columns when you save, and running pipelines start writing them within about a minute, without a restart. A field you stop indexing stops being written within about a minute, and its columns are removed 6 hours later.

Can I stop indexing service and host?

No. Grepr always indexes the service and host tags.

Why do searches on some indexed columns read every data file?

By default, a data lake table collects file statistics on its first 100 columns only, and a search skips data files on a column only by using those statistics. To collect them on every column of the raw log or raw span table, turn on Raw logs table or Raw spans table under Advanced in the dataset's settings. The change applies to data files written after the pipelines writing the dataset restart.

What does marking an attribute path numeric do?

The path gets its numeric column when you save, for values that are numbers written as strings. Without it, the path gets its numeric column when Grepr finds a number there.