= grok
:type: processor
:status: stable
:categories: ["Parsing"]



////
     THIS FILE IS AUTOGENERATED!

     To make changes, edit the corresponding source file under:

     https://github.com/redpanda-data/connect/tree/main/internal/impl/<provider>.

     And:

     https://github.com/redpanda-data/connect/tree/main/cmd/tools/docs_gen/templates/plugin.adoc.tmpl
////

// © 2024 Redpanda Data Inc.


component_type_dropdown::[]


Parses messages into a structured format by attempting to apply a list of Grok expressions, the first expression to result in at least one value replaces the original message with a JSON object containing the values.


[tabs]
======
Common::
+
--

```yml
# Common config fields, showing default values
label: ""
grok:
  expressions: [] # No default (required)
  pattern_definitions: {}
  pattern_paths: []
```

--
Advanced::
+
--

```yml
# All config fields, showing default values
label: ""
grok:
  expressions: [] # No default (required)
  pattern_definitions: {}
  pattern_paths: []
  named_captures_only: true
  use_default_patterns: true
  remove_empty_values: true
```

--
======

Type hints within patterns are respected, therefore with the pattern `%\{WORD:first},%{INT:second:int}` and a payload of `foo,1` the resulting payload would be `\{"first":"foo","second":1}`.

== Performance

This processor currently uses the https://golang.org/s/re2syntax[Go RE2^] regular expression engine, which is guaranteed to run in time linear to the size of the input. However, this property often makes it less performant than PCRE based implementations of grok. For more information, see https://swtch.com/~rsc/regexp/regexp1.html.

== Examples

[tabs]
======
VPC Flow Logs::
+
--


Grok can be used to parse unstructured logs such as VPC flow logs that look like this:

```text
2 123456789010 eni-1235b8ca123456789 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK
```

Into structured objects that look like this:

```json
{"accountid":"123456789010","action":"ACCEPT","bytes":4249,"dstaddr":"172.31.16.21","dstport":22,"end":1418530070,"interfaceid":"eni-1235b8ca123456789","logstatus":"OK","packets":20,"protocol":6,"srcaddr":"172.31.16.139","srcport":20641,"start":1418530010,"version":2}
```

With the following config:

```yaml
pipeline:
  processors:
    - grok:
        expressions:
          - '%{VPCFLOWLOG}'
        pattern_definitions:
          VPCFLOWLOG: '%{NUMBER:version:int} %{NUMBER:accountid} %{NOTSPACE:interfaceid} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NOTSPACE:srcport:int} %{NOTSPACE:dstport:int} %{NOTSPACE:protocol:int} %{NOTSPACE:packets:int} %{NOTSPACE:bytes:int} %{NUMBER:start:int} %{NUMBER:end:int} %{NOTSPACE:action} %{NOTSPACE:logstatus}'
```

--
======

== Fields

=== `expressions`

One or more Grok expressions to attempt against incoming messages. The first expression to match at least one value will be used to form a result.


*Type*: `array`


=== `pattern_definitions`

A map of pattern definitions that can be referenced within `patterns`.


*Type*: `object`

*Default*: `{}`

=== `pattern_paths`

A list of paths to load Grok patterns from. This field supports wildcards, including super globs (double star).


*Type*: `array`

*Default*: `[]`

=== `named_captures_only`

Whether to only capture values from named patterns.


*Type*: `bool`

*Default*: `true`

=== `use_default_patterns`

Whether to use a <<default-patterns, default set of patterns>>.


*Type*: `bool`

*Default*: `true`

=== `remove_empty_values`

Whether to remove values that are empty from the resulting structure.


*Type*: `bool`

*Default*: `true`

== Default patterns

For summary of the default patterns on offer, see https://github.com/Jeffail/grok/blob/master/patterns.go#L5.