Changelog ========= All notable changes to this project will be documented in this file. ## 4.40.0 - TBD ### Added - New `pg_stream` input supporting change data capture (CDC) from PostgreSQL (@le-vlad) - Field `metadata_max_age` added to the `redpanda_migrator_offsets` output. (@mihaitodor) ### Changed - `snowflake_streaming` with `schema_evolution.enabled` set to true can now autocreate tables. - Fields `translate_schema_ids` and `schema_registry_output_resource` added to the `redpanda_migrator` output. (@mihaitodor) - Fields `backfill_dependencies` and `input_resource` added to the `schema_registry` output. (@mihaitodor) - The `schema_registry` input and output and the `schema_registry_encode` and `schema_registry_decode` processors now use the `github.com/twmb/franz-go/pkg/sr` SchemaRegistry client. (@mihaitodor) ## 4.39.0 - 2024-11-07 ### Added - New `timeplus` input. (@ye11ow) - New `snowflake_streaming` output. (@rockwotj) - Benthos will now use an optional `/etc/redpanda/connector_list.yaml` config to determine which connectors are available to run. (@Jeffail) - (Benthos) Field `follow_redirects` added to the `http` processor. (@ooesili) - New CLI flag `--secrets` added. (@Jeffail) - New CLI flag `--disable-telemetry` added. (@Jeffail) - New experimental `spicedb` watch input. (@simon0191) - New `redpanda_common` input and output. (@Jeffail) - New `redpanda` input and output. (@Jeffail) - New `snowflake_streaming` output. (@rockwotj) ### Fixed - The `kafka`, `kafka_franz` and `redpanda_migrator` outputs no longer waste CPU for large batches. (@rockwotj) ### Changed - The `aws_sqs` output field `url` now supports interpolation functions. (@rockwotj) - (Benthos) CLI `--set` flags can now mutate array values indexed from the end via negative integers. E.g. `--set 'foo.-1=meow'` would set the last index of the array `foo` to the value of `meow`. (@Jeffail) ## 4.38.0 - 2024-10-17 ### Added - Anonymous telemetry data is now sent by Connect instances after running for >5 mins. Details about which data is sent, when it is sent, and how to disable it can be found in the [telemetry README](./internal/telemetry/README.md). (@Jeffail) - Field `checksum_algorithm` added to the `aws_s3` output. (@dom-lee-naimuri) - Field `nkey` added to `nats`, `nats_jetstream`, `nats_kv` and `nats_stream` components. (@ye11ow) - Field `private_key` added to the `snowflake_put` output. (@mihaitodor) - New `azure_data_lake_gen2` output. (@ooesili) - New `timeplus` output. (@ye11ow) ### Fixed - The `elasticsearch` output now performs retries for HTTP status code `429` (Too Many Requests). (@kahoowkh) - The docs for the `collection` field of the `mongodb` output now specify support for interpolation functions. (@mihaitodor) ### Changed - All components with a default `path` field value (such as the `aws_s3` output) containing the deprecated function `count` have now been changed to use the new function `counter`. This could potentially change behaviour in cases where multiple components are executing a mapping with a `count` function sharing the same of the old default count, and these counters need to cascade. This is an extremely unlikely scenario, but for all users of these components it is recommended that your `path` is defined explicitly, and in a future major version we will be removing the defaults. ## 4.37.0 - 2024-09-26 ### Added - New experimental `gcp_vertex_ai_embeddings` processor. (@rockwotj) - New experimental `aws_bedrock_embeddings` processor. (@rockwotj) - New experimental `cohere_chat` and `cohere_embeddings` processors. (@rockwotj) - New experimental `questdb` output. (@sklarsa) - Field `metadata_max_age` added to the `kafka_franz` input. (@Scarjit) - Field `metadata_max_age` added to the `kafka_migrator` input. (@mihaitodor) - New experimental `cypher` output. (@rockwotj) - New experimental `couchbase` output. (@rockwotj) - Field `fetch_in_order` added to the `schema_registry` input. (@mihaitodor) ### Fixed - Fixed a bug with the `input_resource` field for the `kafka_migrator` output where new topics weren't created as expected. (@mihaitodor) - Fixed a bug in the `kafka_migrator` input which could lead to extra duplicate messages during a consumer group rebalance. (@mihaitodor) - `kafka_migrator`, `kafka_migrator_offsets` and `kafka_migrator_bundle` components renamed to `redpanda_migrator`, `redpanda_migrator_offsets` and `redpanda_migrator_bundle` (@mihaitodor) ### Fixed - Fixes a panic in the `parquet_encode` processor (@mihaitodor) ## 4.36.0 - 2024-09-11 ### Added - Fields `replication_factor` and `replication_factor_override` added to the `kafka_migrator` input and output. (@mihaitodor) ### Fixed - The `schema_registry_encode` and `schema_registry_decode` processors no longer unescape path separators in the schema name. (@Mizaro) - (Benthos) The `switch` output metrics now emit the case id as part of their labels. This is a regression introduced in v4.25.0. (@mihaitodor) - (Benthos) Fixed a bug where certain logs used the `%w` verb to print errors resulting in incorrect output. (@mihaitodor) - (Benthos) The logger no longer tries to replace Go fmt verbs in log messages. (@mihaitodor) ## 4.35.1 - 2024-09-06 ### Added - Azure and GCP components added to cloud builds. (@Jeffail) ### Fixed - The `kafka_migrator_bundle` input and output no longer require schema registry to be configured. (@mihaitodor) ## 4.35.0 - 2024-09-05 ### Added - Auth fields added to the `schema_registry` input and output. (@mihaitodor) - New experimental `kafka_migrator` and `kafka_migrator_bundle` inputs and outputs. (@mihaitodor) - New experimental `kafka_migrator_offsets` output. (@mihaitodor) - Field `job_project` added to the `gcp_bigquery` output. (@Roviluca) ## 4.34.0 - 2024-08-29 ### Fixed - The `schema_registry` output now allows pushing schemas if the target Schema Registry instance is in `IMPORT` mode. (@mihaitodor) - Fixed an issue where the `azure_blob_storage` input would fail to delete blobs when using `targets_input` with `delete_objects: true`. (@mihaitodor) - New experimental `gcp_vertex_ai_chat` processor. (@rockwotj) - New experimental `aws_bedrock_chat` processor. (@rockwotj) ## 4.33.0 - 2024-08-13 ### Added - Field `content_md5` added to the `aws_s3` output. (@dom-lee-naimuri) - Field `send_ack` added to the `nats` input. (@plejd-sebman) - New Bloblang method `vector`. (@rockwotj) - New experimental `ockam_kafka` input and output. (@mrinalwadhwa, @davide-baldo) - Field `credentials_json` added to all GCP components. (@tomasz-sadura) - (Benthos) The `list` subcommand now supports the format `jsonschema`. (@Jeffail) - New experimental `schema_registry` input and output. (@mihaitodor) - New experimental `qdrant` output. (@Anush008) - (Benthos) The `--set` run flag now supports structured values, e.g. `--set input={}`. (@Jeffail) ## 4.32.1 - 2024-07-24 ### Changed - The number of release build artifacts for the `community` and `cloud` flavours have been reduced due to Github Action Runner disk space limitations. ## 4.32.0 - 2024-07-24 ### Added - Field `app_name` added to the MongoDB components. (@mihaitodor) - New `openai_chat_completion` processor. (@rockwotj) - New `openai_embeddings` processor. (@rockwotj) - New `openai_image_generation` processor. (@rockwotj) - New `openai_speech` processor. (@rockwotj) - New `openai_transcription` processor. (@rockwotj) - New `openai_translation` processor. (@rockwotj) - New `ollama_chat` processor. (@rockwotj) - New `ollama_embeddings` processor. (@rockwotj) ### Changed - The `gcp_pubsub` output now rejects messages with metadata values which contain invalid UTF-8-encoded runes. (@AndreasBergmeier6176) - The `.goreleaser.yml` configuration has been set back to version 1. (@Jeffail) ## 4.31.0 - 2024-07-19 ### Added - The `splunk` input and `splunk_hec` output now support custom `tls` configuration. (@mihaitodor) - Field `timestamp` added to the `kafka` and `kafka_franz` outputs. (@mihaitodor) - (Benthos) Field `max_retries` added to the `retry` processor. (@mihaitodor) - (Benthos) Metadata fields `retry_count` and `backoff_duration` added to the `retry` processor. (@mihaitodor) - (Benthos) Parameter `escape_html` added to the `format_json()` Bloblang method. (@mihaitodor) - (Benthos) New `array` bloblang method. (@gramian) - (Benthos) Algorithm `fnv32` added to the `hash` bloblang method. (@CallMeMhz) - New experimental `redpanda_data_transform`. (@rockwotj) - New `-community` suffixed build included in release artifacts, containing only FOSS functionality. (@Jeffail) - New `-cloud` suffixed build included in release artifacts, containing components enabled in Redpanda Cloud. (@Jeffail) - Field `status_topic` added to the global `redpanda` config block. (@Jeffail) - New `pinecone` output. (@rockwotj) - (Benthos) The `/ready` endpoint in regular operation now provides a detailed summary of all inputs and outputs, including connection errors where applicable. (@Jeffail) ### Changed - (Benthos) All cli subcommands that previously relied on root-level flags (`streams`, `lint`, `test`, `echo`) now explicitly define those flags such that they appear in help-text and can be specified _after_ the subcommand itself. This means previous commands such as `connect -r ./foo.yaml streams ./bar.yaml` can now be more intuitively written as `connect streams -r ./foo.yaml ./bar.yaml` and so on. The old style will still work in order to preserve backwards compatibility, but the help-text for these root-level flags has been hidden. (@Jeffail) ## 4.30.1 - 2024-06-13 ### Fixed - AWS Lambda serverless build artifacts have been added back to official releases. ## 4.30.0 - 2024-06-13 ### Added - (Benthos) Field `omit_empty` added to the `lines` scanner. (@mihaitodor) - (Benthos) New scheme `gcm` added to the `encrypt_aes` and `decrypy_aes` Bloblang methods. (@abergmeier) - (Benthos) New Bloblang method `pow`. (@mfamador) - (Benthos) New `sin`, `cos`, `tan` and `pi` bloblang methods. (@mfamador) - (Benthos) Field `proxy_url` added to the `websocket` input and output. (@mihaitodor) - New experimental `splunk` input. (@mihaitodor) ### Fixed - The `sql_insert` and `sql_raw` components no longer fail when inserting large binary blobs into Oracle `BLOB` columns. (@mihaitodor) - (Benthos) The `websocket` input and output now obey the `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` environment variables. (@mihaitodor) ### Changed - The `splunk_hec` output is now implemented as a native Go component. (@mihaitodor) ## 4.29.0 - 2024-06-04 ### Added - Go API: New packages `public/bundle/free` and `public/bundle/enterprise` with explicit licensing for bundles of component imports. - Field `auth.oauth2.scope` added to the `pulsar` input and output. (@srenatus) - Field `subscription_initial_position` added to the `pulsar` input. (@srenatus) ### Fixed - The `pulsar` input and output should no longer ignore `auth.oauth2` fields. (@srenatus) - Creating builds using `make` no longer prints warnings when the repository does not contain a tag. (@mkysel) - Messages resulting from the `redis` processor are no longer invalid when using hash commands. (@mkysel) - The `nats_jetstream` input no longer fails to initialise when a stream is specified and a subject is not. (@maxarndt) ## 4.28.0 - 2024-05-30 ### Changed - The repository has been moved to `redpanda-data/connect` and no longer contains the core Benthos engine, which is now broken out into `redpanda-data/benthos`. ## 4.27.0 - 2024-04-23 ### Added - New `nats_kv` cache type. - The `nats_jetstream` input now supports `last_per_subject` and `new` deliver fallbacks. - Field `error_patterns` added to the `drop_on` output. - New `redis_scan` input type. - Field `auto_replay_nacks` added to all inputs that traditionally automatically retry nacked messages as a toggle for this behaviour. - New `retry` processor. - New `noop` cache. - Field `targets_input` added to the `azure_blob_storage` input. - New `reject_errored` output. - New `nats_request_reply` processor. - New `json_documents` scanner. ### Fixed - The `unarchive` processor no longer yields linting errors when the format `csv:x` is specified. This is a regression introduced in v4.25.0. - The `sftp` input will no longer consume files when the watcher cache returns an error. Instead, it will reattempt the file upon the next poll. - The `aws_sqs` input no longer logs error level logs for visibility timeout refreshing errors. - The `nats_kv` processor now allows [nats wildcards](https://docs.nats.io/nats-concepts/subjects#wildcards) for the `keys` operation. - The `nats_kv` processor `keys` operation now returns a single message with an array of found keys instead of a batch of messages. - The `nats_kv` processor `history` operation now returns a single message with an array of objects containing the record fields instead of a batch of messages. - Field `timeout` added to the `nats_kv` processor to specify the maximum period to wait on an operation before aborting and returning an error. - Bloblang comparison operators (`>`, `<`, `<=`, `>=`) now match the precision of the compared integers when applicable. - The `parse_form_url_encoded` Bloblang method no longer produces results with an unknown data type for repeated query parameters. - The `echo` CLI command no longer fails to sanitise configs when encountering an empty `password` field. - The `sql_insert` and `sql_raw` components no longer fail when inserting large binary blobs into Oracle `BLOB` columns. ### Changed - The log events from all inputs and outputs when they first connect have been made more consistent and no longer contain any information regarding the nature of their connections. - Splitting message batches with a `split` processor (or custom plugins) no longer results in downstream error handling loops around nacks. This was previously implemented as a feature to ensure unbounded expanded and split batches don't flood downstream services in the event of a minority of errors. However, introducing more clever origin tracking of errored messages has eliminated the need for this undocumented behaviour. ## 4.26.0 - 2024-03-18 ### Added - Field `credit` added to the `amqp_1` input to specify the maximum number of unacknowledged messages the sender can transmit. - Bloblang now supports root-level `if` statements. - New experimental `sql` cache. - Fields `batch_size`, `sort` and `limit` added to the `mongodb` input. - Field `idemponent_write` added to the `kafka` output. ### Changed - The default value of the `amqp_1.credit` input has changed from `1` to `64`. - The `mongodb` processor and output now support extended JSON in canonical form for document, filter and hint mappings. - The `open_telemetry_collector` tracer has had the `url` field of gRPC and HTTP collectors deprecated in favour of `address`, which more accurately describes the intended format of endpoints. The old style will continue to work, but eventually will have its default value removed and an explicit value will be required. ### Fixed - Resource config imports containing `%` characters were being incorrectly parsed during unit test execution. This was a regression introduced in v4.25.0. - Dynamic input and output config updates containing `%` characters were being incorrectly parsed. This was a regression introduced in v4.25.0. ## 4.25.1 - 2024-03-01 ### Fixed - Fixed a regression in v4.25.0 where [template based components](https://www.benthos.dev/docs/configuration/templating) were not parsing correctly from configs. ## 4.25.0 - 2024-03-01 ### Added - Field `address_cache` added to the `socket_server` input. - Field `read_header` added to the `amqp_1` input. - All inputs with a `codec` field now support a new field `scanner` to replace it. Scanners are more powerful as they are configured in a structured way similar to other component types rather than via a single string field, for more information [check out the scanners page](https://www.benthos.dev/docs/components/scanners/about). - New `diff` and `patch` Bloblang methods. - New `processors` processor. - Field `read_header` added to the `amqp_1` input. - A debug endpoint `/debug/pprof/allocs` has been added for profiling allocations. - New `cockroachdb_changefeed` input. - The `open_telemetry_collector` tracer now supports sampling. - The `aws_kinesis` input and output now support specifying ARNs as the stream target. - New `azure_cosmosdb` input, processor and output. - All `sql_*` components now support the `gocosmos` driver. - New `opensearch` output. ### Fixed - The `javascript` processor now handles module imports correctly. - Bloblang `if` statements now provide explicit errors when query expressions resolve to non-boolean values. - Some metadata fields from the `amqp_1` input were always empty due to type mismatch, this should no longer be the case. - The `zip` Bloblang method no longer fails when executed without arguments. - The `amqp_0_9` output no longer prints bogus exchange name when connecting to the server. - The `generate` input no longer adds an extra second to `interval: '@every x'` syntax. - The `nats_jetstream` input no longer fails to locate mirrored streams. - Fixed a rare panic in batching mechanisms with a specified `period`, where data arrives in low volumes and is sporadic. - Executing config unit tests should no longer fail due to output resources failing to connect. ### Changed - The `parse_parquet` Bloblang function, `parquet_decode`, `parquet_encode` processors and the `parquet` input have all been upgraded to the latest version of the underlying Parquet library. Since this underlying library is experimental it is likely that behaviour changes will result. One significant change is that encoding numerical values that are larger than the column type (`float64` into `FLOAT`, `int64` into `INT32`, etc) will no longer be automatically converted. - The `parse_log` processor field `codec` is now deprecated. - *WARNING*: Many components have had their underlying implementations moved onto newer internal APIs for defining and extracting their configuration fields. It's recommended that upgrades to this version are performed cautiously. - *WARNING*: All AWS components have been upgraded to the latest client libraries. Although lots of testing has been done, these libraries have the potential to differ in discrete ways in terms of how credentials are evaluated, cross-account connections are performed, and so on. It's recommended that upgrades to this version are performed cautiously. ## 4.24.0 - 2023-11-24 ### Added - Field `idempotent_write` added to the `kafka_franz` output. - Field `idle_timeout` added to the `read_until` input. - Field `delay_seconds` added to the `aws_sqs` output. - Fields `discard_unknown` and `use_proto_names` added to the `protobuf` processors. ### Fixed - Bloblang error messages for bad function/method names or parameters should now be improved in mappings that use shorthand for `root = ...`. - All redis components now support usernames within the configured URL for authentication. - The `protobuf` processor now supports targetting nested types from proto files. - The `schema_registry_encode` and `schema_registry_decode` processors should no longer double escape URL unsafe characters within subjects when querying their latest versions. ## 4.23.0 - 2023-10-30 ### Added - The `amqp_0_9` output now supports dynamic interpolation functions within the `exchange` field. - Field `custom_topic_creation` added to the `kafka` output. - New Bloblang method `ts_sub`. - The Bloblang method `abs` now supports integers in and integers out. - Experimental `extract_tracing_map` field added to the `nats`, `nats_jetstream` and `nats_stream` inputs. - Experimental `inject_tracing_map` field added to the `nats`, `nats_jetstream` and `nats_stream` outputs. - New `_fail_fast` variants for the `broker` output `fan_out` and `fan_out_sequential` patterns. - Field `summary_quantiles_objectives` added to the `prometheus` metrics exporter. - The `metric` processor now supports floating point values for `counter_by` and `gauge` types. ### Fixed - Allow labels on caches and rate limit resources when writing configs in CUE. - Go API: `log/slog` loggers injected into a stream builder via `StreamBuilder.SetLogger` should now respect formatting strings. - All Azure components now support container SAS tokens for authentication. - The `kafka_franz` input now provides properly typed metadata values. - The `trino` driver for the various `sql_*` components no longer panics when trying to insert nulls. - The `http_client` input no longer sends a phantom request body on subsequent requests when an empty `payload` is specified. - The `schema_registry_encode` and `schema_registry_decode` processors should no longer fail to obtain schemas containing slashes (or other URL path unfriendly characters). - The `parse_log` processor no longer extracts structured fields that are incompatible with Bloblang mappings. - Fixed occurrences where Bloblang would fail to recognise `float32` values. ## 4.22.0 - 2023-10-03 ### Added - The `-e/--env-file` cli flag for importing environment variable files now supports glob patterns. - Environment variables imported via `-e/--env-file` cli flags now support triple quoted strings. - New experimental `counter` function added to Bloblang. It is recommended that this function, although experimental, should be used instead of the now deprecated `count` function. - The `schema_registry_encode` and `schema_registry_decode` processors now support JSONSchema. - Field `metadata` added to the `nats` and `nats_jetstream` outputs. - The `cached` processor field `ttl` now supports interpolation functions. - Many new properties fields have been added to the `amqp_0_9` output. - Field `command` added to the `redis_list` input and output. ### Fixed - Corrected a scheduling error where the `generate` input with a descriptor interval (`@hourly`, etc) had a chance of firing twice. - Fixed an issue where a `redis_streams` input that is rejected from read attempts enters a reconnect loop without backoff. - The `sqs` input now periodically refreshes the visibility timeout of messages that take a significant amount of time to process. - The `ts_add_iso8601` and `ts_sub_iso8601` bloblang methods now return the correct error for certain invalid durations. - The `discord` output no longer ignores structured message fields containing underscores. - Fixed an issue where the `kafka_franz` input was ignoring batching periods and stalling. ### Changed - The `random_int` Bloblang function now prevents instantiations where either the `max` or `min` arguments are dynamic. This is in order to avoid situations where the random number generator is re-initialised across subsequent mappings in a way that surprises map authors. ## 4.21.0 - 2023-09-08 ### Added - Fields `client_id` and `rack_id` added to the `kafka_franz` input and output. - New experimental `command` processor. - Parameter `no_cache` added to the `file` and `env` Bloblang functions. - New `file_rel` function added to Bloblang. - Field `endpoint_params` added to the `oauth2` section of HTTP client components. ### Fixed - Allow comments in single root and directly imported bloblang mappings. - The `azure_blob_storage` input no longer adds `blob_storage_content_type` and `blob_storage_content_encoding` metadata values as string pointer types, and instead adds these values as string types only when they are present. - The `http_server` input now returns a more appropriate 503 service unavailable status code during shutdown instead of the previous 404 status. - Fixed a potential panic when closing a `pusher` output that was never initialised. - The `sftp` output now reconnects upon being disconnected by the Azure idle timeout. - The `switch` output now produces error logs when messages do not pass at least one case with `strict_mode` enabled, previously these rejected messages were potentially re-processed in a loop without any logs depending on the config. An inaccuracy to the documentation has also been fixed in order to clarify behaviour when strict mode is not enabled. - The `log` processor `fields_mapping` field should no longer reject metadata queries using `@` syntax. - Fixed an issue where heavily utilised streams with nested resource based outputs could lock-up when performing heavy resource mutating traffic on the streams mode REST API. - The Bloblang `zip` method no longer produces values that yield an "Unknown data type". ## 4.20.0 - 2023-08-22 ### Added - The `amqp1` input now supports `anonymous` SASL authentication. - New JWT Bloblang methods `parse_jwt_es256`, `parse_jwt_es384`, `parse_jwt_es512`, `parse_jwt_rs256`, `parse_jwt_rs384`, `parse_jwt_rs512`, `sign_jwt_es256`, `sign_jwt_es384` and `sign_jwt_es512` added. - The `csv-safe` input codec now supports custom delimiters with the syntax `csv-safe:x`. - The `open_telemetry_collector` tracer now supports secure connections, enabled via the `secure` field. - Function `v0_msg_exists_meta` added to the `javascript` processor. ### Fixed - Fixed an issue where saturated output resources could panic under intense CRUD activity. - The config linter no longer raises issues with codec fields containing colons within their arguments. - The `elasticsearch` output should no longer fail to send basic authentication passwords, this fixes a regression introduced in v4.19.0. ## 4.19.0 - 2023-08-17 ### Added - Field `topics_pattern` added to the `pulsar` input. - Both the `schema_registry_encode` and `schema_registry_decode` processors now support protobuf schemas. - Both the `schema_registry_encode` and `schema_registry_decode` processors now support references for AVRO and PROTOBUF schemas. - New Bloblang method `zip`. - New Bloblang `int8`, `int16`, `uint8`, `uint16`, `float32` and `float64` methods. ### Fixed - Errors encountered by the `gcp_pubsub` output should now present more specific logs. - Upgraded `kafka` input and output underlying sarama client library to v1.40.0 at new module path github.com/IBM/sarama - The CUE schema for `switch` processor now correctly reflects that it takes a list of clauses. - Fixed the CUE schema for fields that take a 2d-array such as `workflow.order`. - The `snowflake_put` output has been added back to 32-bit ARM builds since the build incompatibilities have been resolved. - The `snowflake_put` output and the `sql_*` components no longer trigger a panic when running on a readonly file system with the `snowflake` driver. This driver still requires access to write temporary files somewhere, which can be configured via the Go [`TMPDIR`](https://pkg.go.dev/os#TempDir) environment variable. Details [here](https://github.com/snowflakedb/gosnowflake/issues/700). - The `http_server` input and output now follow the same multiplexer rules regardless of whether the general `http` server block is used or a custom endpoint. - Config linting should now respect fields sourced via a merge key (`<<`). - The `lint` subcommand should now lint config files pointed to via `-r`/`--resources` flags. ### Changed - The `snowflake_put` output is now beta. - Endpoints specified by `http_server` components using both the general `http` server block or their own custom server addresses should no longer be treated as path prefixes unless the path ends with a slash (`/`), in which case all extensions of the path will match. This corrects a behavioural change introduced in v4.14.0. ## 4.18.0 - 2023-07-02 ### Added - Field `logger.level_name` added for customising the name of log levels in the JSON format. - Methods `sign_jwt_rs256`, `sign_jwt_rs384` and `sign_jwt_rs512` added to Bloblang. ### Fixed - HTTP components no longer ignore `proxy_url` settings when OAuth2 is set. - The `PATCH` verb for the streams mode REST API no longer fails to patch over newer components implemented with the latest plugin APIs. - The `nats_jetstream` input no longer fails for configs that set `bind` to `true` and do not specify both a `stream` and `durable` together. - The `mongodb` processor and output no longer ignores the `upsert` field. ### Changed - The old `parquet` processor (now superseded by `parquet_encode` and `parquet_decode`) has been removed from 32-bit ARM builds due to build incompatibilities. - The `snowflake_put` output has been removed from 32-bit ARM builds due to build incompatibilities. - Plugin API: The `(*BatchError).WalkMessages` method has been deprecated in favour of `WalkMessagesIndexedBy`. ## 4.17.0 - 2023-06-13 ### Added - The `dynamic` input and output have a new endpoint `/input/{id}/uptime` and `/output/{id}/uptime` respectively for obtaining the uptime of a given input/output. - Field `wait_time_seconds` added to the `aws_sqs` input. - Field `timeout` added to the `gcp_cloud_storage` output. - All NATS components now set the name of each connection to the component label when specified. ### Fixed - Restore message ordering support to `gcp_pubsub` output. This issue was introduced in 4.16.0 as a result of [#1836](https://github.com/benthosdev/benthos/pull/1836). - Specifying structured metadata values (non-strings) in unit test definitions should no longer cause linting errors. ### Changed - The `nats` input default value of `prefetch_count` has been increased from `32` to a more appropriate `524288`. ## 4.16.0 - 2023-05-28 ### Added - Fields `auth.user_jwt` and `auth.user_nkey_seed` added to all NATS components. - bloblang: added `ulid(encoding, random_source)` function to generate Universally Unique Lexicographically Sortable Identifiers (ULIDs). - Field `skip_on` added to the `cached` processor. - Field `nak_delay` added to the `nats` input. - New `splunk_hec` output. - Plugin API: New `NewMetadataExcludeFilterField` function and accompanying `FieldMetadataExcludeFilter` method added. - The `pulsar` input and output are now included in the main distribution of Benthos again. - The `gcp_pubsub` input now adds the metadata field `gcp_pubsub_delivery_attempt` to messages when dead lettering is enabled. - The `aws_s3` input now adds `s3_version_id` metadata to versioned messages. - All compress/decompress components (codecs, bloblang methods, processors) now support `pgzip`. - Field `connection.max_retries` added to the `websocket` input. - New `sentry_capture` processor. ### Fixed - The `open_telemetry_collector` tracer option no longer blocks service start up when the endpoints cannot be reached, and instead manages connections in the background. - The `gcp_pubsub` output should see significant performance improvements due to a client library upgrade. - The stream builder APIs should now follow `logger.file` config fields. - The experimental `cue` format in the cli `list` subcommand no longer introduces infinite recursion for `#Processors`. - Config unit tests no longer execute linting rules for missing env var interpolations. ## 4.15.0 - 2023-05-05 ### Added - Flag `--skip-env-var-check` added to the `lint` subcommand, this disables the new linting behaviour where environment variable interpolations without defaults throw linting errors when the variable is not defined. - The `kafka_franz` input now supports explicit partitions in the field `topics`. - The `kafka_franz` input now supports batching. - New `metadata` Bloblang function for batch-aware structured metadata queries. - Go API: Running the Benthos CLI with a context set with a deadline now triggers graceful termination before the deadline is reached. - Go API: New `public/service/servicetest` package added for functions useful for testing custom Benthos builds. - New `lru` and `ttlru` in-memory caches. ### Fixed - Provide msgpack plugins through `public/components/msgpack`. - The `kafka_franz` input should no longer commit offsets one behind the next during partition yielding. - The streams mode HTTP API should no longer route requests to `/streams/` to the `/streams` handler. This issue was introduced in v4.14.0. ## 4.14.0 - 2023-04-25 ### Added - The `-e/--env-file` cli flag can now be specified multiple times. - New `studio pull` cli subcommand for running [Benthos Studio](https://studio.benthos.dev) session deployments. - Metadata field `kafka_tombstone_message` added to the `kafka` and `kafka_franz` inputs. - Method `SetEnvVarLookupFunc` added to the stream builder API. - The `discord` input and output now use the official chat client API and no longer rely on poll-based HTTP requests, this should result in more efficient and less erroneous behaviour. - New bloblang timestamp methods `ts_add_iso8601` and `ts_sub_iso8601`. - All SQL components now support the `trino` driver. - New input codec `csv-safe`. - Added `base64rawurl` scheme to both the `encode` and `decode` Bloblang methods. - New `find_by` and `find_all_by` Bloblang methods. - New `skipbom` input codec. - New `javascript` processor. ### Fixed - The `find_all` bloblang method no longer produces results that are of an `unknown` type. - The `find_all` and `find` Bloblang methods no longer fail when the value argument is a field reference. - Endpoints specified by HTTP server components using both the general `http` server block or their own custom server addresses should now be treated as path prefixes. This corrects a behavioural change that was introduced when both respective server options were updated to support path parameters. - Prevented a panic caused when using the `encrypt_aes` and `decrypt_aes` Bloblang methods with a mismatched key/iv lengths. - The `snowpipe` field of the `snowflake_put` output can now be omitted from the config without raising an error. - Batch-aware processors such as `mapping` and `mutation` should now report correct error metrics. - Running `benthos blobl server` should no longer panic when a mapping with variable read/writes is executed in parallel. - Speculative fix for the `cloudwatch` metrics exporter rejecting metrics due to `minimum field size of 1, PutMetricDataInput.MetricData[0].Dimensions[0].Value`. - The `snowflake_put` output now prevents silent failures under certain conditions. Details [here](https://github.com/snowflakedb/gosnowflake/issues/701). - Reduced the amount of pre-compilation of Bloblang based linting rules for documentation fields, this should dramatically improve the start up time of Benthos (~1s down to ~200ms). - Environment variable interpolations with an empty fallback (`${FOO:}`) are now valid. - Fixed an issue where the `mongodb` output wasn't using bulk send requests according to batching policies. - The `amqp_1` input now falls back to accessing `Message.Value` when the data is empty. ### Changed - When a config contains environment variable interpolations without a default value (i.e. `${FOO}`), if that environment variable is not defined a linting error will be emitted. Shutting down due to linting errors can be disabled with the `--chilled` cli flag, and variables can be specified with an empty default value (`${FOO:}`) in order to make the previous behaviour explicit and prevent the new linting error. - The `find` and `find_all` Bloblang methods no longer support query arguments as they were incompatible with supporting value arguments. For query based arguments use the new `find_by` and `find_all_by` methods. ## 4.13.0 - 2023-03-15 ### Added - Fix vulnerability [GO-2023-1571](https://pkg.go.dev/vuln/GO-2023-1571) - New `nats_kv` processor, input and output. - Field `partition` added to the `kafka_franz` output, allowing for manual partitioning. ### Fixed - The `broker` output with the pattern `fan_out_sequential` will no longer abandon in-flight requests that are error blocked until the full shutdown timeout has occurred. - Fixed a regression bug in the `sequence` input where the returned messages have type `unknown`. This issue was introduced in v4.10.0 (cefa288). - The `broker` input no longer reports itself as unavailable when a child input has intentionally closed. - Config unit tests that check for structured data should no longer fail in all cases. - The `http_server` input with a custom address now supports path variables. ## 4.12.1 - 2023-02-23 ### Fixed - Fixed a regression bug in the `nats` components where panics occur during a flood of messages. This issue was introduced in v4.12.0 (45f785a). ## 4.12.0 - 2023-02-20 ### Added - Format `csv:x` added to the `unarchive` processor. - Field `max_buffer` added to the `aws_s3` input. - Field `open_message_type` added to the `websocket` input. - The experimental `--watcher` cli flag now takes into account file deletions and new files that match wildcard patterns. - Field `dump_request_log_level` added to HTTP components. - New `couchbase` cache implementation. - New `compress` and `decompress` Bloblang methods. - Field `endpoint` added to the `gcp_pubsub` input and output. - Fields `file_name`, `file_extension` and `request_id` added to the `snowflake_put` output. - Add interpolation support to the `path` field of the `snowflake_put` output. - Add ZSTD compression support to the `compression` field of the `snowflake_put` output. - New Bloblang method `concat`. - New `redis` ratelimit. - The `socket_server` input now supports `tls` as a network type. - New bloblang function `timestamp_unix_milli`. - New bloblang method `ts_unix_milli`. - JWT based HTTP authentication now supports `EdDSA`. - New `flow_control` fields added to the `gcp_pubsub` output. - Added bloblang methods `sign_jwt_hs256`, `sign_jwt_hs384` and `sign_jwt_hs512` - New bloblang methods `parse_jwt_hs256`, `parse_jwt_hs384`, `parse_jwt_hs512`. - The `open_telemetry_collector` tracer now automatically sets the `service.name` and `service.version` tags if they are not configured by the user. - New bloblang string methods `trim_prefix` and `trim_suffix`. ### Fixed - Fixed an issue where messages caught in a retry loop from inputs that do not support nacks (`generate`, `kafka`, `file`, etc) could be retried in their post-mutation form from the `switch` output rather than the original copy of the message. - The `sqlite` buffer should no longer print `Failed to ack buffer message` logs during graceful termination. - The default value of the `conn_max_idle` field has been changed from 0 to 2 for all `sql_*` components in accordance to the [`database/sql` docs](https://pkg.go.dev/database/sql#DB.SetMaxIdleConns). - The `parse_csv` bloblang method with `parse_header_row` set to `false` no longer produces rows that are of an `unknown` type. - Fixed a bug where the `oracle` driver for the `sql_*` components was returning timestamps which were getting marshalled into an empty JSON object instead of a string. - The `aws_sqs` input no longer backs off on subsequent empty requests when long polling is enabled. - It's now possible to mock resources within the main test target file in config unit tests. - Unit test linting no longer incorrectly expects the `json_contains` predicate to contain a string value only. - Config component initialisation errors should no longer show nested path annotations. - Prevented panics from the `jq` processor when querying invalid types. - The `jaeger` tracer no longer emits the `service.version` tag automatically if the user sets the `service.name` tag explicitly. - The `int64()`, `int32()`, `uint64()` and `uint32()` bloblang methods can now infer the number base as documented [here](https://pkg.go.dev/strconv#ParseInt). - The `mapping` and `mutation` processors should provide metrics and tracing events again. - Fixed a data race in the `redis_streams` input. - Upgraded the Redis components to `github.com/redis/go-redis/v9`. ## 4.11.0 - 2022-12-21 ### Added - Field `default_encoding` added to the `parquet_encode` processor. - Field `client_session_keep_alive` added to the `snowflake_put` output. - Bloblang now supports metadata access via `@foo` syntax, which also supports arbitrary values. - TLS client certs now support both PKCS#1 and PKCS#8 encrypted keys. - New `redis_script` processor. - New `wasm` processor. - Fields marked as secrets will no longer be printed with `benthos echo` or debug HTTP endpoints. - Add `no_indent` parameter to the `format_json` bloblang method. - New `format_xml` bloblang method. - New `batched` higher level input type. - The `gcp_pubsub` input now supports optionally creating subscriptions. - New `sqlite` buffer. - Bloblang now has `int64`, `int32`, `uint64` and `uint32` methods for casting explicit integer types. - Field `application_properties_map` added to the `amqp1` output. - Param `parse_header_row`, `delimiter` and `lazy_quotes` added to the `parse_csv` bloblang method. - Field `delete_on_finish` added to the `csv` input. - Metadata fields `header`, `path`, `mod_time_unix` and `mod_time` added to the `csv` input. - New `couchbase` processor. - Field `max_attempts` added to the `nsq` input. - Messages consumed by the `nsq` input are now enriched with metadata. - New Bloblang method `parse_url`. ### Fixed - Fixed a regression bug in the `mongodb` processor where message errors were not set any more. This issue was introduced in v4.7.0 (64eb72). - The `avro-ocf:marshaler=json` input codec now omits unexpected logical type fields. - Fixed a bug in the `sql_insert` output (see commit c6a71e9) where transaction-based drivers (`clickhouse` and `oracle`) would fail to roll back an in-progress transaction if any of the messages caused an error. - The `resource` input should no longer block the first layer of graceful termination. ### Changed - The `catch` method now defines the context of argument mappings to be the string of the caught error. In previous cases the context was undocumented, vague and would often bind to the outer context. It's still possible to reference this outer context by capturing the error (e.g. `.catch(_ -> this)`). - Field interpolations that fail due to mapping errors will no longer produce placeholder values and will instead provide proper errors that result in nacks or retries similar to other issues. ## 4.10.0 - 2022-10-26 ### Added - The `nats_jetstream` input now adds a range of useful metadata information to messages. - Field `transaction_type` added to the `azure_table_storage` output, which deprecates the previous `insert_type` field and supports interpolation functions. - Field `logged_batch` added to the `cassandra` output. - All `sql` components now support Snowflake. - New `azure_table_storage` input. - New `sql_raw` input. - New `tracing_id` bloblang function. - New `with` bloblang method. - Field `multi_header` added to the `kafka` and `kafka_franz` inputs. - New `cassandra` input. - New `base64_encode` and `base64_decode` functions for the awk processor. - Param `use_number` added to the `parse_json` bloblang method. - Fields `init_statement` and `init_files` added to all sql components. - New `find` and `find_all` bloblang array methods. ### Fixed - The `gcp_cloud_storage` output no longer ignores errors when closing a written file, this was masking issues when the target bucket was invalid. - Upgraded the `kafka_franz` input and output to use github.com/twmb/franz-go@v1.9.0 since some [bug fixes](https://github.com/twmb/franz-go/blob/master/CHANGELOG.md#v190) were made recently. - Fixed an issue where a `read_until` child input with processors affiliated would block graceful termination. - The `--labels` linting option no longer flags resource components. ## 4.9.1 - 2022-10-06 ### Added - Go API: A new `BatchError` type added for distinguishing errors of a given batch. ### Fixed - Rolled back `kafka` input and output underlying sarama client library to fix a regression introduced in 4.9.0 😅 where `invalid configuration (Consumer.Group.Rebalance.GroupStrategies and Consumer.Group.Rebalance.Strategy cannot be set at the same time)` errors would prevent consumption under certain configurations. We've decided to roll back rather than upgrade as a breaking API change was introduced that could cause issues for Go API importers (more info here: https://github.com/Shopify/sarama/issues/2358). ## 4.9.0 - 2022-10-03 ### Added - New `parquet` input for reading a batch of Parquet files from disk. - Field `max_in_flight` added to the `redis_list` input. ### Fixed - Upgraded `kafka` input and output underlying sarama client library to fix a regression introduced in 4.7.0 where `The requested offset is outside the range of offsets maintained by the server for the given topic/partition` errors would prevent consumption of partitions. - The `cassandra` output now inserts logged batches of data rather than the less efficient (and unnecessary) unlogged form. ## 4.8.0 - 2022-09-30 ### Added - All `sql` components now support Oracle DB. ### Fixed - All SQL components now accept an empty or unspecified `args_mapping` as an alias for no arguments. - Field `unsafe_dynamic_query` added to the `sql_raw` output. - Fixed a regression in 4.7.0 where HTTP client components were sending duplicate request headers. ## 4.7.0 - 2022-09-27 ### Added - Field `avro_raw_json` added to the `schema_registry_decode` processor. - Field `priority` added to the `gcp_bigquery_select` input. - The `hash` bloblang method now supports `crc32`. - New `tracing_span` bloblang function. - All `sql` components now support SQLite. - New `beanstalkd` input and output. - Field `json_marshal_mode` added to the `mongodb` input. - The `schema_registry_encode` and `schema_registry_decode` processors now support Basic, OAuth and JWT authentication. ### Fixed - The streams mode `/ready` endpoint no longer returns status `503` for streams that gracefully finished. - The performance of the bloblang `.explode` method now scales linearly with the target size. - The `influxdb` and `logger` metrics outputs should no longer mix up tag names. - Fix a potential race condition in the `read_until` connect check on terminated input. - The `parse_parquet` bloblang method and `parquet_decode` processor now automatically parse `BYTE_ARRAY` values as strings when the logical type is UTF8. - The `gcp_cloud_storage` output now correctly cleans up temporary files on error conditions when the collision mode is set to append. ## 4.6.0 - 2022-08-31 ### Added - New `squash` bloblang method. - New top-level config field `shutdown_delay` for delaying graceful termination. - New `snowflake_id` bloblang function. - Field `wait_time_seconds` added to the `aws_sqs` input. - New `json_path` bloblang method. - New `file_json_contains` predicate for unit tests. - The `parquet_encode` processor now supports the `UTF8` logical type for columns. ### Fixed - The `schema_registry_encode` processor now correctly assumes Avro JSON encoded documents by default. - The `redis` processor `retry_period` no longer shows linting errors for duration strings. - The `/inputs` and `/outputs` endpoints for dynamic inputs and outputs now correctly render configs, both structured within the JSON response and the raw config string. - Go API: The stream builder no longer ignores `http` configuration. Instead, the value of `http.enabled` is set to `false` by default. ## 4.5.1 - 2022-08-10 ### Fixed - Reverted `kafka_franz` dependency back to `1.3.1` due to a regression in TLS/SASL commit retention. - Fixed an unintentional linting error when using interpolation functions in the `elasticsearch` outputs `action` field. ## 4.5.0 - 2022-08-07 ### Added - Field `batch_size` added to the `generate` input. - The `amqp_0_9` output now supports setting the `timeout` of publish. - New experimental input codec `avro-ocf:marshaler=x`. - New `mapping` and `mutation` processors. - New `parse_form_url_encoded` bloblang method. - The `amqp_0_9` input now supports setting the `auto-delete` bit during queue declaration. - New `open_telemetry_collector` tracer. - The `kafka_franz` input and output now supports no-op SASL options with the mechanism `none`. - Field `content_type` added to the `gcp_cloud_storage` cache. ### Fixed - The `mongodb` processor and output default `write_concern.w_timeout` empty value no longer causes configuration issues. - Field `message_name` added to the logger config. - The `amqp_1` input and output should no longer spam logs with timeout errors during graceful termination. - Fixed a potential crash when the `contains` bloblang method was used to compare complex types. - Fixed an issue where the `kafka_franz` input or output wouldn't use TLS connections without custom certificate configuration. - Fixed structural cycle in the CUE representation of the `retry` output. - Tracing headers from HTTP requests to the `http_server` input are now correctly extracted. ### Changed - The `broker` input no longer applies processors before batching as this was unintentional behaviour and counter to documentation. Users that rely on this behaviour are advised to place their pre-batching processors at the level of the child inputs of the broker. - The `broker` output no longer applies processors after batching as this was unintentional behaviour and counter to documentation. Users that rely on this behaviour are advised to place their post-batching processors at the level of the child outputs of the broker. ## 4.4.1 - 2022-07-19 ### Fixed - Fixed an issue where an `http_server` input or output would fail to register prometheus metrics when combined with other inputs/outputs. - Fixed an issue where the `jaeger` tracer was incapable of sending traces to agents outside of the default port. ## 4.4.0 - 2022-07-18 ### Added - The service-wide `http` config now supports basic authentication. - The `elasticsearch` output now supports upsert operations. - New `fake` bloblang function. - New `parquet_encode` and `parquet_decode` processors. - New `parse_parquet` bloblang method. - CLI flag `--prefix-stream-endpoints` added for disabling streams mode API prefixing. - Field `timestamp_name` added to the logger config. ## 4.3.0 - 2022-06-23 ### Added - Timestamp Bloblang methods are now able to emit and process `time.Time` values. - New `ts_tz` method for switching the timezone of timestamp values. - The `elasticsearch` output field `type` now supports interpolation functions. - The `redis` processor has been reworked to be more generally useful, the old `operator` and `key` fields are now deprecated in favour of new `command` and `args_mapping` fields. - Go API: Added component bundle `./public/components/aws` for all AWS components, including a `RunLambda` function. - New `cached` processor. - Go API: New APIs for registering both metrics exporters and open telemetry tracer plugins. - Go API: The stream builder API now supports configuring a tracer, and tracer configuration is now isolated to the stream being executed. - Go API: Plugin components can now access input and output resources. - The `redis_streams` output field `stream` field now supports interpolation functions. - The `kafka_franz` input and outputs now support `AWS_MSK_IAM` as a SASL mechanism. - New `pusher` output. - Field `input_batches` added to config unit tests for injecting a series of message batches. ### Fixed - Corrected an issue where Prometheus metrics from batching at the buffer level would be skipped when combined with input/output level batching. - Go API: Fixed an issue where running the CLI API without importing a component package would result in template init crashing. - The `http` processor and `http_client` input and output no longer have default headers as part of their configuration. A `Content-Type` header will be added to requests with a default value of `application/octet-stream` when a message body is being sent and the configuration has not added one explicitly. - Logging in `logfmt` mode with `add_timestamp` enabled now works. ## 4.2.0 - 2022-06-03 ### Added - Field `credentials.from_ec2_role` added to all AWS based components. - The `mongodb` input now supports aggregation filters by setting the new `operation` field. - New `gcp_cloudtrace` tracer. - New `slug` bloblang string method. - The `elasticsearch` output now supports the `create` action. - Field `tls.root_cas_file` added to the `pulsar` input and output. - The `fallback` output now adds a metadata field `fallback_error` to messages when shifted. - New bloblang methods `ts_round`, `ts_parse`, `ts_format`, `ts_strptime`, `ts_strftime`, `ts_unix` and `ts_unix_nano`. Most are aliases of (now deprecated) time methods with `timestamp_` prefixes. - Ability to write logs to a file (with optional rotation) instead of stdout. ### Fixed - The default docker image no longer throws configuration errors when running streams mode without an explicit general config. - The field `metrics.mapping` now allows environment functions such as `hostname` and `env`. - Fixed a lock-up in the `amqp_0_9` output caused when messages sent with the `immediate` or `mandatory` flags were rejected. - Fixed a race condition upon creating dynamic streams that self-terminate, this was causing panics in cases where the stream finishes immediately. ## 4.1.0 - 2022-05-11 ### Added - The `nats_jetstream` input now adds headers to messages as metadata. - Field `headers` added to the `nats_jetstream` output. - Field `lazy_quotes` added to the CSV input. ### Fixed - Fixed an issue where resource and stream configs imported via wildcard pattern could not be live-reloaded with the watcher (`-w`) flag. - Bloblang comparisons between numerical values (including `match` expression patterns) no longer require coercion into explicit types. - Reintroduced basic metrics from the `twitter` and `discord` template based inputs. - Prevented a metrics label mismatch when running in streams mode with resources and `prometheus` metrics. - Label mismatches with the `prometheus` metric type now log errors and skip the metric without stopping the service. - Fixed a case where empty files consumed by the `aws_s3` input would trigger early graceful termination. ## 4.0.0 - 2022-04-20 This is a major version release, for more information and guidance on how to migrate please refer to [https://benthos.dev/docs/guides/migration/v4](https://www.benthos.dev/docs/guides/migration/v4). ### Added - In Bloblang it is now possible to reference the `root` of the document being created within a mapping query. - The `nats_jetstream` input now supports pull consumers. - Field `max_number_of_messages` added to the `aws_sqs` input. - Field `file_output_path` added to the `prometheus` metrics type. - Unit test definitions can now specify a label as a `target_processors` value. - New connection settings for all sql components. - New experimental `snowflake_put` output. - New experimental `gcp_cloud_storage` cache. - Field `regexp_topics` added to the `kafka_franz` input. - The `hdfs` output `directory` field now supports interpolation functions. - The cli `list` subcommand now supports a `cue` format. - Field `jwt.headers` added to all HTTP client components. - Output condition `file_json_equals` added to config unit test definitions. ### Fixed - The `sftp` output no longer opens files in both read and write mode. - The `aws_sqs` input with `reset_visibility` set to `false` will no longer reset timeouts on pending messages during gracefully shutdown. - The `schema_registry_decode` processor now handles AVRO logical types correctly. Details in [#1198](https://github.com/benthosdev/benthos/pull/1198) and [#1161](https://github.com/benthosdev/benthos/issues/1161) and also in https://github.com/linkedin/goavro/issues/242. ### Changed - All components, features and configuration fields that were marked as deprecated have been removed. - The `pulsar` input and output are no longer included in the default Benthos builds. - The field `pipeline.threads` field now defaults to `-1`, which automatically matches the host machine CPU count. - Old style interpolation functions (`${!json:foo,1}`) are removed in favour of the newer Bloblang syntax (`${! json("foo") }`). - The Bloblang functions `meta`, `root_meta`, `error` and `env` now return `null` when the target value does not exist. - The `clickhouse` SQL driver Data Source Name format parameters have been changed due to a client library update. This also means placeholders in `sql_raw` components should use dollar syntax. - Docker images no longer come with a default config that contains generated environment variables, use `-s` flag arguments instead. - All cache components have had their retry/backoff fields modified for consistency. - All cache components that support a general default TTL now have a field `default_ttl` with a duration string, replacing the previous field. - The `http` processor and `http_client` output now execute message batch requests as individual requests by default. This behaviour can be disabled by explicitly setting `batch_as_multipart` to `true`. - Outputs that traditionally wrote empty newlines at the end of batches with >1 message when using the `lines` codec (`socket`, `stdout`, `file`, `sftp`) no longer do this by default. - The `switch` output field `retry_until_success` now defaults to `false`. - All AWS components now have a default `region` field that is empty, allowing environment variables or profile values to be used by default. - Serverless distributions of Benthos (AWS lambda, etc) have had the default output config changed to reject messages when the processing fails, this should make it easier to handle errors from invocation. - The standard metrics emitted by Benthos have been largely simplified and improved, for more information [check out the metrics page](https://www.benthos.dev/docs/components/metrics/about). - The default metrics type is now `prometheus`. - The `http_server` metrics type has been renamed to `json_api`. - The `stdout` metrics type has been renamed to `logger`. - The `logger` configuration section has been simplified, with `logfmt` being the new default format. - The `logger` field `add_timestamp` is now `false` by default. - Field `parts` has been removed from all processors. - Field `max_in_flight` has been removed from a range of output brokers as it no longer required. - The `dedupe` processor now acts upon individual messages by default, and the `hash` field has been removed. - The `log` processor now executes for each individual message of a batch. - The `sleep` processor now executes for each individual message of a batch. - The `benthos test` subcommand no longer walks when targetting a directory, instead use triple-dot syntax (`./dir/...`) or wildcard patterns. - Go API: Module name has changed to `github.com/benthosdev/benthos/v4`. - Go API: All packages within the `lib` directory have been removed in favour of the newer [APIs within `public`](https://pkg.go.dev/github.com/benthosdev/benthos/v4/public). - Go API: Distributed tracing is now via the Open Telemetry client library. ## 3.65.0 - 2022-03-07 ### Added - New `sql_raw` processor and output. ### Fixed - Corrected a case where nested `parallel` processors that result in emptied batches (all messages filtered) would propagate an unack rather than an acknowledgement. ### Changed - The `sql` processor and output are no longer marked as deprecated and will therefore not be removed in V4. This change was made in order to provide more time to migrate to the new `sql_raw` processor and output. ## 3.64.0 - 2022-02-23 ### Added - Field `nack_reject_patterns` added to the `amqp_0_9` input. - New experimental `mongodb` input. - Field `cast` added to the `xml` processor and `parse_xml` bloblang method. - New experimental `gcp_bigquery_select` processor. - New `assign` bloblang method. - The `protobuf` processor now supports `Any` fields in protobuf definitions. - The `azure_queue_storage` input field `queue_name` now supports interpolation functions. ### Fixed - Fixed an issue where manually clearing errors within a `catch` processor would result in subsequent processors in the block being skipped. - The `cassandra` output should now automatically match `float` columns. - Fixed an issue where the `elasticsearch` output would collapse batched messages of matching ID rather than send as individual items. - Running streams mode with `--no-api` no longer removes the `/ready` endpoint. ### Changed - The `throttle` processor has now been marked as deprecated. ## 3.63.0 - 2022-02-08 ### Added - Field `cors` added to the `http_server` input and output, for supporting CORS requests when custom servers are used. - Field `server_side_encryption` added to the `aws_s3` output. - Field `use_histogram_timing` and `histogram_buckets` added to the `prometheus` metrics exporter. - New duration string and back off field types added to plugin config builders. - Experimental field `multipart` added to the `http_client` output. - Codec `regex` added to inputs. - Field `timeout` added to the `cassandra` output. - New experimental `gcp_bigquery_select` input. - Field `ack_wait` added to the `nats_jetstream` input. ### Changed - The old map-style resource config fields (`resources.processors.`, etc) are now marked as deprecated. Use the newer list based fields (`processor_resources`, etc) instead. ### Fixed - The `generate` input now supports zeroed duration strings (`0s`, etc) for unbounded document creation. - The `aws_dynamodb_partiql` processor no longer ignores the `endpoint` field. - Corrected duplicate detection for custom cache implementations. - Fixed panic caused by invalid bounds in the `range` function. - Resource config files imported now allow (and ignore) a `tests` field. - Fixed an issue where the `aws_kinesis` input would fail to back off during unyielding read attempts. - Fixed a linting error with `zmq4` input/output `urls` fields that was incorrectly expecting a string. ## 3.62.0 - 2022-01-21 ### Added - Field `sync` added to the `gcp_pubsub` input. - New input, processor, and output config field types added to the plugin APIs. - Added new experimental `parquet` processor. - New Bloblang method `format_json`. - Field `collection` in `mongodb` processor and output now supports interpolation functions. - Field `output_raw` added to the `jq` processor. - The lambda distribution now supports a `BENTHOS_CONFIG_PATH` environment variable for specifying a custom config path. - Field `metadata` added to `http` and `http_client` components. - Field `ordering_key` added to the `gcp_pubsub` output. - A suite of new experimental `geoip_` methods have been added. - Added flag `--deprecated` to the `benthos lint` subcommand for detecting deprecated fields. ### Changed - The `sql` processor and output have been marked deprecated in favour of the newer `sql_insert`, `sql_select` alternatives. ### Fixed - The input codec `chunked` is no longer capped by the packet size of the incoming streams. - The `schema_registry_decode` and `schema_registry_encode` processors now honour trailing slashes in the `url` field. - Processors configured within `pipeline.processors` now share processors across threads rather than clone them. - Go API: Errors returned from input/output plugin `Close` methods no longer cause shutdown to block. - The `pulsar` output should now follow authentication configuration. - Fixed an issue where the `aws_sqs` output might occasionally retry a failed message send with an invalid empty message body. ## 3.61.0 - 2021-12-28 ### Added - Field `json_marshal_mode` added to the MongoDB processor. - Fields `extract_headers.include_prefixes` and `extract_headers.include_patterns` added to the `http_client` input and output and to the `http` processor. - Fields `sync_response.metadata_headers.include_prefixes` and `sync_response.metadata_headers.include_patterns` added to the `http_server` input. - The `http_client` input and output and the `http` processor field `copy_response_headers` has been deprecated in favour of the `extract_headers` functionality. - Added new cli flag `--no-api` for the `streams` subcommand to disable the REST API. - New experimental `kafka_franz` input and output. - Added new Bloblang function `ksuid`. - All `codec` input fields now support custom csv delimiters. ### Fixed - Streams mode paths now resolve glob patterns in all cases. - Prevented the `nats` input from error logging when acknowledgments can't be fulfilled due to the lack of message replies. - Fixed an issue where GCP inputs and outputs could terminate requests early due to a cancelled client context. - Prevented more parsing errors in Bloblang mappings with windows style line endings. ## 3.60.1 - 2021-12-03 ### Fixed - Fixed an issue where the `mongodb` output would incorrectly report upsert not allowed on valid operators. ## 3.60.0 - 2021-12-01 ### Added - The `pulsar` input and output now support `oauth2` and `token` authentication mechanisms. - The `pulsar` input now enriches messages with more metadata. - Fields `message_group_id`, `message_deduplication_id`, and `metadata` added to the `aws_sns` output. - Field `upsert` added to the `mongodb` processor and output. ### Fixed - The `schema_registry_encode` and `schema_registry_decode` processors now honour path prefixes included in the `url` field. - The `mqtt` input and output `keepalive` field is now interpreted as seconds, previously it was being erroneously interpreted as nanoseconds. - The header `Content-Type` in the field `http_server.sync_response.headers` is now detected in a case insensitive way when populating multipart message encoding types. - The `nats_jetstream` input and outputs should now honour `auth.*` config fields. ## 3.59.0 - 2021-11-22 ### Added - New Bloblang method `parse_duration_iso8601` for parsing ISO-8601 duration strings into an integer. - The `nats` input now supports metadata from headers when supported. - Field `headers` added to the `nats` output. - Go API: Optional field definitions added for config specs. - New (experimental) `sql_select` input. - New (experimental) `sql_select` and `sql_insert` processors, which will supersede the existing `sql` processor. - New (experimental) `sql_insert` output, which will supersede the existing `sql` output. - Field `retained_interpolated` added to the `mqtt` output. - Bloblang now allows optional carriage returns before line feeds at line endings. - New CLI flag `-w`/`-watcher` added for automatically detecting and applying configuration file changes. - Field `avro_raw_json` added to the `schema_registry_encode` processor. - New (experimental) `msgpack` processor. - New `parse_msgpack` and `format_msgpack` Bloblang methods. ### Fixed - Fixed an issue where the `azure_table_storage` output would attempt to send >100 size batches (and fail). - Fixed an issue in the `subprocess` input where saturated stdout streams could become corrupted. ## 3.58.0 - 2021-11-02 ### Added - `amqp_0_9` components now support TLS EXTERNAL auth. - Field `urls` added to the `amqp_0_9` input and output. - New experimental `schema_registry_encode` processor. - Field `write_timeout` added to the `mqtt` output, and field `connect_timeout` added to both the input and output. - The `websocket` input and output now support custom `tls` configuration. - New output broker type `fallback` added as a drop-in replacement for the now deprecated `try` broker. ### Fixed - Removed a performance bottleneck when consuming a large quantity of small files with the `file` input. ## 3.57.0 - 2021-10-14 ### Added - Go API: New config field types `StringMap`, `IntList`, and `IntMap`. - The `http_client` input, output and processor now include the response body in request error logs for more context. - Field `dynamic_client_id_suffix` added to the `mqtt` input and output. ### Fixed - Corrected an issue where the `sftp` input could consume duplicate documents before shutting down when ran in batch mode. ## 3.56.0 - 2021-09-22 ### Added - Fields `cache_control`, `content_disposition`, `content_language` and `website_redirect_location` added to the `aws_s3` output. - Field `cors.enabled` and `cors.allowed_origins` added to the server wide `http` config. - For Kafka components the config now supports the `rack_id` field which may contain a rack identifier for the Kafka client. - Allow mapping imports in Bloblang environments to be disabled. - Go API: Isolated Bloblang environments are now honored by all components. - Go API: The stream builder now evaluates environment variable interpolations. - Field `unsafe_dynamic_query` added to the `sql` processor. - The `kafka` output now supports `zstd` compression. ### Fixed - The `test` subcommand now expands resource glob patterns (`benthos -r "./foo/*.yaml" test ./...`). - The Bloblang equality operator now returns `false` when comparing non-null values with `null` rather than a mismatched types error. ## 3.55.0 - 2021-09-08 ### Added - New experimental `gcp_bigquery` output. - Go API: It's now possible to parse a config spec directly with `ParseYAML`. - Bloblang methods and functions now support named parameters. - Field `args_mapping` added to the `cassandra` output. - For NATS, NATS Streaming and Jetstream components the config now supports specifying either `nkey_file` or `user_credentials_file` to configure authentication. ## 3.54.0 - 2021-09-01 ### Added - The `mqtt` input and output now support sending a last will, configuring a keep alive timeout, and setting retained out output messages. - Go API: New stream builder `AddBatchProducerFunc` and `AddBatchConsumerFunc` methods. - Field `gzip_compression` added to the `elasticsearch` output. - The `redis_streams` input now supports creating the stream with the `MKSTREAM` command (enabled by default). - The `kafka` output now supports manual partition allocation using interpolation functions in the field `partition`. ### Fixed - The bloblang method `contains` now correctly compares numerical values in arrays and objects. ## 3.53.0 - 2021-08-19 ### Added - Go API: Added ability to create and register `BatchBuffer` plugins. - New `system_window` buffer for processing message windows (sliding or tumbling) following the system clock. - Field `root_cas` added to all TLS configuration blocks. - The `sftp` input and output now support key based authentication. - New Bloblang function `nanoid`. - The `gcp_cloud_storage` output now supports custom collision behaviour with the field `collision_mode`. - Field `priority` added to the `amqp_0_9` output. - Operator `keys` added to the `redis` processor. - The `http_client` input when configured in stream mode now allows message body interpolation functions within the URL and header parameters. ### Fixed - Fixed a panic that would occur when executing a pipeline where processor or input resources reference rate limits. ## 3.52.0 - 2021-08-02 ### Added - The `elasticsearch` output now supports delete, update and index operations. - Go API: Added ability to create and register `BatchInput` plugins. ### Fixed - Prevented the `http_server` input from blocking graceful pipeline termination indefinitely. - Removed annoying nil error log from HTTP client components when parsing responses. ## 3.51.0 - 2021-07-26 ### Added - The `redis_streams`, `redis_pubsub` and `redis_list` outputs now all support batching for higher throughput. - The `amqp_1` input and output now support passing and receiving metadata as annotations. - Config unit test definitions can now use files for both the input and expected output. - Field `track_properties` added to the `azure_queue_storage` input for enriching messages with properties such as the message backlog. - Go API: The new plugin APIs, available at `./public/service`, are considered stable. - The streams mode API now uses the setting `http.read_timeout` for timing out stream CRUD endpoints. ### Fixed - The Bloblang function `random_int` now only resolves dynamic arguments once during the lifetime of the mapping. Documentation has been updated in order to clarify the behaviour with dynamic arguments. - Fixed an issue where plugins registered would return `failed to obtain docs for X type Y` linting errors. - HTTP client components are now more permissive regarding invalid Content-Type headers. ## 3.50.0 - 2021-07-19 ### Added - New CLI flag `--set` (`-s`) for overriding arbitrary fields in a config. E.g. `-s input.type=http_server` would override the config setting the input type to `http_server`. - Unit test definitions now support mocking components. ## 3.49.0 - 2021-07-12 ### Added - The `nats` input now supports acks. - The `memory` and `file` cache types now expose metrics akin to other caches. ### Fixed - The `switch` output when `retry_until_success` is set to `false` will now provide granular nacks to pre-batched messages. - The URL printed in error messages when HTTP client components fail should now show interpolated values as they were interpreted. - Go Plugins API V2: Batched processors should now show in tracing, and no longer complain about spans being closed more than once. ## 3.48.0 - 2021-06-25 ### Added - Algorithm `lz4` added to the `compress` and `decompress` processors. - New experimental `aws_dynamodb_partiql` processor. - Go Plugins API: new run opt `OptUseContext` for an extra shutdown mechanism. ### Fixed - Fixed an issue here the `http_client` would prematurely drop connections when configured with `stream.enabled` set to `true`. - Prevented closed output brokers from leaving child outputs running when they've failed to establish a connection. - Fixed metrics prefixes in streams mode for nested components. ## 3.47.0 - 2021-06-16 ### Added - CLI flag `max-token-length` added to the `blobl` subcommand. - Go Plugins API: Plugin components can now be configured seamlessly like native components, meaning the namespace `plugin` is no longer required and configuration fields can be placed within the namespace of the plugin itself. Note that the old style (within `plugin`) is still supported. - The `http_client` input fields `url` and `headers` now support interpolation functions that access metadata and contents of the last received message. - Rate limit resources now emit `checked`, `limited` and `error` metrics. - A new experimental plugins API is available for early adopters, and can be found at `./public/x/service`. - A new experimental template system is available for early adopters, examples can be found in `./template`. - New beta Bloblang method `bloblang` for executing dynamic mappings. - All `http` components now support a beta `jwt` authentication mechanism. - New experimental `schema_registry_decode` processor. - New Bloblang method `parse_duration` for parsing duration strings into an integer. - New experimental `twitter_search` input. - New field `args_mapping` added to the `sql` processor and output for mapping explicitly typed arguments. - Added format `csv` to the `unarchive` processor. - The `redis` processor now supports `incrby` operations. - New experimental `discord` input and output. - The `http_server` input now adds a metadata field `http_server_verb`. - New Bloblang methods `parse_yaml` and `format_yaml`. - CLI flag `env-file` added to Benthos for parsing dotenv files. - New `mssql` SQL driver for the `sql` processor and output. - New POST endpoint `/resources/{type}/{id}` added to Benthos streams mode for dynamically mutating resource configs. ### Changed - Go Plugins API: The Bloblang `ArgSpec` now returns a public error type `ArgError`. - Components that support glob paths (`file`, `csv`, etc) now also support super globs (double asterisk). - The `aws_kinesis` input is now stable. - The `gcp_cloud_storage` input and output are now beta. - The `kinesis` input is now deprecated. - Go Plugins API: the minimum version of Go required is now 1.16. ### Fixed - Fixed a rare panic caused when executing a `workflow` resource processor that references `branch` resources across parallel threads. - The `mqtt` input with multiple topics now works with brokers that would previously error on multiple subscriptions. - Fixed initialisation of components configured as resources that reference other resources, where under certain circumstances the components would fail to obtain a true reference to the target resource. This fix makes it so that resources are accessed only when used, which will also make it possible to introduce dynamic resources in future. - The streams mode endpoint `/streams/{id}/stats` should now work again provided the default manager is used. ## 3.46.1 - 2021-05-19 ### Fixed - The `branch` processor now writes error logs when the request or result map fails. - The `branch` processor (and `workflow` by proxy) now allow errors to be mapped into the branch using `error()` in the `request_map`. - Added a linting rule that warns against having a `reject` output under a `switch` broker without `retry_until_success` disabled. - Prevented a panic or variable corruption that could occur when a Bloblang mapping is executed by parallel threads. ## 3.46.0 - 2021-05-06 ### Added - The `create` subcommand now supports a `--small`/`-s` flag that reduces the output down to only core components and common fields. - Go Plugins API: Added method `Overlay` to the public Bloblang package. - The `http_server` input now adds path parameters (`/{foo}/{bar}`) to the metadata of ingested messages. - The `stdout` output now has a `codec` field. - New Bloblang methods `format_timestamp_strftime` and `parse_timestamp_strptime`. - New experimental `nats_jetstream` input and output. ### Fixed - Go Plugins API: Bloblang method and function plugins now automatically resolve dynamic arguments. ## 3.45.1 - 2021-04-27 ### Fixed - Fixed a regression where the `http_client` input with an empty `payload` would crash with a `url` containing interpolation functions. - Broker output types (`broker`, `try`, `switch`) now automatically match the highest `max_in_flight` of their children. The field `max_in_flight` can still be manually set in order to enforce a minimum value for when inference isn't possible, such as with dynamic output resources. ## 3.45.0 - 2021-04-23 ### Added - Experimental `azure_renew_lock` field added to the `amqp_1` input. - New beta `root_meta` function. - Field `dequeue_visibility_timeout` added to the `azure_queue_storage` input. - Field `max_in_flight` added to the `azure_queue_storage` output. - New beta Bloblang methods `format_timestamp_unix` and `format_timestamp_unix_nano`. - New Bloblang methods `reverse` and `index_of`. - Experimental `extract_tracing_map` field added to the `kafka` input. - Experimental `inject_tracing_map` field added to the `kafka` output. - Field `oauth2.scopes` added to HTTP components. - The `mqtt` input and output now support TLS. - Field `enable_renegotiation` added to `tls` configurations. - Bloblang `if` expressions now support an arbitrary number of `else if` blocks. ### Fixed - The `checkpoint_limit` field for the `kafka` input now works according to explicit messages in flight rather than the actual offset. This means it now works as expected with compacted topics. - The `aws_kinesis` input should now automatically recover when the shard iterator has expired. - Corrected an issue where messages prefixed with valid JSON documents or values were being decoded in truncated form when the remainder was invalid. ### Changed - The following beta components have been promoted to stable: + `ristretto` cache + `csv` and `generate` inputs + `reject` output + `branch`, `jq` and `workflow` processors ## 3.44.1 - 2021-04-15 ### Fixed - Fixed an issue where the `kafka` input with partition balancing wasn't committing offsets. ## 3.44.0 - 2021-04-09 ### Added - The `http_server` input now provides a metadata field `http_server_request_path`. - New methods `sort_by` and `key_values` added to Bloblang. ### Fixed - Glob patterns for various components no longer resolve to bad paths in the absence of matches. - Fixed an issue where acknowledgements from the `azure_queue_storage` input would timeout prematurely, resulting in duplicated message delivery. - Unit test definitions no longer have implicit test cases when omitted. ## 3.43.1 - 2021-04-05 ### Fixed - Vastly improved Bloblang mapping errors. - The `azure_blob_storage` input will now gracefully terminate if the client credentials become invalid. - Prevented the experimental `gcp_cloud_storage` input from closing early during large file consumption. ## 3.43.0 - 2021-03-31 ### New - New (experimental) Apache Pulsar input and output. - Field `codec` added to the `socket` output. - New Bloblang method `map_each_key`. - General config linting improvements. - Bloblang mappings and interpolated fields within configs are now compile checked during linting. - New output level `metadata.exclude_prefixes` config field for restricting metadata values sent to the following outputs: `kafka`, `aws_s3`, `amqp_0_9`, `redis_streams`, `aws_sqs`, `gcp_pubsub`. - All NATS components now have `tls` support. - Bloblang now supports context capture in query lambdas. - New subcommand `benthos blobl server` that hosts a Bloblang editor web application. - New (experimental) `mongodb` output, cache and processor. - New (experimental) `gcp_cloud_storage` input and output. - Field `batch_as_multipart` added to the `http_client` output. - Inputs, outputs, processors, caches and rate limits now have a component level config field `label`, which sets the metrics and logging prefix. - Resources can now be declared in the new `_resources` fields at the root of config files, the old `resources.s.