Benthos Fails to Fully Process Environment Values Containing #

[pv123444]

Slack Reference:

Channel: Redpanda Community
Message Link: Slack Message

Issue:

We found that Benthos fails to fully process environment variable values containing the # character. When fetching such values, it stops reading at the # and ignores the remaining part of the value.

Example:
Given an environment variable set as:
MY_ENV_VAR=abc#def#ghi

Benthos only reads abc, and everything after the first # (def#ghi) is ignored.

Expected Behavior:
Benthos should read and process the entire value, abc#def#ghi.

Actual Behavior:
Benthos only reads abc, causing issues for workflows that need the full value.

[mihaitodor]

Hey @pv123444, can you please confirm that MY_ENV_VAR="abc#def#ghi" works for you? As mentioned on Slack, we can potentially look into adjusting the env file parser, but we need to know which standard to follow.

[pv123444]

It works for values with #, but when using strings with special characters like \n or \t, they are treated as escape characters and don't work as expected.

…

On Tue, Dec 24, 2024 at 6:31 PM Mihai Todor ***@***.***> wrote: Hey @pv123444 <https://github.com/pv123444>, can you please confirm that MY_ENV_VAR="abc#def#ghi" works for you? As mentioned on Slack, we can potentially look into adjusting the env file parser, but we need to know which standard to follow. — Reply to this email directly, view it on GitHub <#147 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BN6AYE6TWSLYM62W74I72CL2HFLMPAVCNFSM6AAAAABUEGLYVOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRRGEYDMNRVGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- *Disclaimer:* The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain confidential or privileged information. If you are not the intended recipient, please notify the sender at Crest Data Systems immediately and destroy all copies of this message and any attachments.

[mihaitodor]

I'm not sure I follow... Can you please provide an example? Also, as mentioned, if you believe we need to change something, please provide a reference to a detailed spec that we should follow.

I added the following in test.env:

FOOBAR="abc\tdef\nghi"

And then running this YAML:

input:
  generate:
    count: 1
    mapping: |
      root = "${FOOBAR}"

I get:

$ benthos run -e foo.env foo.yaml
INFO Running main config from specified file       @service=redpanda-connect benthos_version="" path=foo.yaml
INFO Listening for HTTP requests at: http://0.0.0.0:4195  @service=redpanda-connect
INFO Launching a Redpanda Connect instance, use CTRL+C to close  @service=redpanda-connect
INFO Input type generate is now active             @service=redpanda-connect label="" path=root.input
INFO Output type stdout is now active              @service=redpanda-connect label="" path=root.output
abc	def
ghi
INFO Pipeline has terminated. Shutting down the service  @service=redpanda-connect

Also, note that we do support raw strings as well via """...""". See an example here.

[pv123444]

If we support raw strings, that’s great! However, I’d like to know if there’s a way to store an environment variable without using string or raw string format, but still fetch the exact value as is.

For example, when I set the environment variable FOOBAR=abc\tdef\nghi, I’d like to retrieve( env("FOOBAR") ) it in its literal form with the escape sequences intact, as abc\tdef\nghi, without interpreting them as tab or newline characters.

Is there a specific method we can use to achieve this? If there’s a detailed specification or guide that outlines how to handle such cases, I’d greatly appreciate a reference to it.

[mihaitodor]

Setting FOOBAR=abc\tdef\nghi inside an env file works just fine. You can also do FOOBAR='abc\tdef\nghi' benthos run test.yaml and that should also work (note the ' instead of " to prevent the terminal from interpreting escape characters). If you really need to use env("FOOBAR") instead of ${FOOBAR} in your bloblang code, you might have to do env("FOOBAR").replace_all_many(["\\t", "\t", "\\n", "\n"]) (see the replace_all_many() method docs).

For #, the issue is defining what should the parser interpret as a comment. If the current implementation isn't suitable, then we'll need a reference spec to consult and determine if there's a way to adhere to it without breaking existing code.

[pv123444]

Hello @mihaitodor
We provide credentials through the UI and store them in AWS Secrets Manager. These credentials are then exported as raw strings using a command like: export variable_name="\"\"\" aws_secret_manager_variable_path \"\"\"" and saved in the environment variables. Benthos is expected to read the value from these environment variables, but it does not interpret the value correctly. For example, when we set a value like ##%'#Benthos##", we checked its behavior using logged output with env("variable_name"), which shows additional raw string characters, such as \"\\\"\\\"\\\"##%'#Benthos##\\\"'\\\"\\\"\\\"\". This indicates that Benthos is not properly parsing the value.

[mihaitodor]

@pv123444 the logging can be misleading since it needs to do all sorts of string escaping. This looks fine to me:

input:
  generate:
    count: 1
    mapping: root = env("FOO")

output:
  stdout: {}

$ export FOO="##%'#Databee##\""
$ rpk connect run test.yaml
...
##%'#Databee##"
INFO Pipeline has terminated. Shutting down the service  @service=redpanda-connect

With an env file, you'll have to use a raw string:

FOO="""##%'#Databee##" """

$ rpk connect run -e test.env test.yaml
....
##%'#Databee##" 
INFO Pipeline has terminated. Shutting down the service  @service=redpanda-connect

Not ideal if you have a " right at the end since you'll get an extra space after the password, but that feels like a niche edge case... We could look into handling this particular situation better if need be.

Have you considered base64-encoding these passwords first? It might save you a lot of headaches...

[pv123444]

Thanks @mihaitodor for your support ,
We will try with base64-encoding.
You can close this Issue.

[abheda-crest]

@mihaitodor I found that Benthos is unable to interpolate the Bloblang expression for the oauth2 and basic_auth fields. To confirm this, I modified the environment variable reference e.g. from ${PASSWORD} to ${! env("PASSWORD")}.

http:
  url: "http://example.com"
  verb: GET
  oauth2:
    enabled: true
    client_key: ${! env("CLIENT_KEY")}
    client_secret: ${! env("CLIENT_SECRET")}
    token_url: ${TOKEN_URL}
    endpoint_params:
      grant_type:
        - password
      username:
        - ${! env("USERNAME")}
      password:
        - ${! env("PASSWORD")}

This issue likely occurs because the oauth2 and basic_auth fields do not explicitly support interpolation functions in the same way that the url and headers fields do. For more details, you can refer to the HTTP processor component page, where the description of the url and headers fields specifically states: This field supports interpolation functions.

Additionally, if we decide to use the base64 encoding approach, we would still need the Bloblang expression to decode the base64 credentials.

How should we handle base64-encoded environment variables for oauth2 and basic_auth?

[mihaitodor]

@abheda-crest Using env() is not supported in arbitrary fields unfortunately. You can only use bloblang expressions in interpolated and mapping fields.

I think I have a workaround for ##%'#Databee##" without requiring raw strings:

.env:

FOO="##%'#Databee##\""

This config:

input:
  generate:
    count: 1
    mapping: root = env("FOO")
  processors:
    - log:
        message: |
          |${FOO}|
    - log:
        message: '|${! content() }|'

output:
  drop: {}

prints

INFO |##%'#Databee##"|
INFO |##%'#Databee##"|

Note: I added the | characters just to show there are no extra spaces inserted in there.

Basically, you'd have to parse your secrets and:

• convert \ to \\
• convert " to \"
• add surrounding "

Would that work for you? If not, happy to look into adjusting the parser, but we'd need to know which standard to follow rather than a list of examples which don't currently work as expected.

[abheda-crest]

Thanks, @mihaitodor, We've attempted the workaround you mentioned, but it doesn't work when trying to use environment variables i.e. ${FOO} within the mapping. Instead, we need to use env to fetch the environment variable directly. Additionally, it's important to note that these environment variables will be used to store credentials. We're looking for a more general solution.

To summarize: We'll store the credentials as environment variables, which will then be used for API calls. The authentication method will vary based on the configuration. This can include standard methods like basic_auth and oauth2, but may also require passing credentials in the request body or headers. Therefore, we need a more flexible and general approach.

Some potential solutions I can think of are:

1. Adding support for interpolation (or Bloblang expressions) inside fields for basic_auth, oauth2, etc., allowing us to use Bloblang's env method (and possibly other methods like decode).
2. Enabling the ability to retrieve environment variables as raw strings without the need of any parsing, e.g., ${FOO | raw_string}, to directly access the environment variable's value.
3. Providing functionality to apply encoding or decoding operations to environment variables, e.g., ${FOO | decode_base64}, to decode the environment variable from base64 format.

Please share your feedback on these potential solutions, or suggest any additional ones you may have.

[mihaitodor]

but it doesn't work when trying to use environment variables i.e. ${FOO} within the mapping

I think you might be hitting some yaml parser issues if you simply do foo: ${FOO}. We first read the config data as a string, swap the env vars and then parse it as yaml (see here). Please try the yaml raw string approach that I showed above (you'll still need to escape double quotes and backslashes):

foo: |
  ${FOO}

or maybe this if the above adds a newline:

foo: >-
  ${FOO}

1. Adding support for interpolation (or Bloblang expressions) inside fields for basic_auth, oauth2, etc., allowing us to use Bloblang's env method (and possibly other methods like decode).

That's a heavy lift unfortunately. We could consider adding interpolation support for certain fields, like maybe some of the auth stuff. I think there was another issue about that a while back. If it can be scoped down to a few fields, happy to track that as part of a separate issue.

2. Enabling the ability to retrieve environment variables as raw strings without the need of any parsing, e.g., ${FOO | raw_string}, to directly access the environment variable's value.
3. Providing functionality to apply encoding or decoding operations to environment variables, e.g., ${FOO | decode_base64}, to decode the environment variable from base64 format.

These are good suggestions, but they won't be a quick enhancement. I'll leave the issue open and see when we'll get a chance to figure out what can be done here. At the very least, we need more docs to cover some of these edge cases.

[abheda-crest]

Thanks, @mihaitodor. We also believe that adopting the third approach, ${FOO | decode_base64}, would be a great choice. We’re looking forward to this enhancement!