Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

http_client inside read_until input does not stop on error even with retries=0 #182

Open
JSchniertshauerUS opened this issue Feb 21, 2025 · 0 comments

Comments

@JSchniertshauerUS
Copy link

JSchniertshauerUS commented Feb 21, 2025

To reproduce, use the following snippets for client and server

client

input:
  read_until:
    input:
      http_client:
        url: "http://localhost:8088/retry-test"
        verb: GET
        timeout: 10s
        retries: 0
        stream:
          enabled: false
          reconnect: false
          scanner:
            to_the_end: {}
    check: true
    restart_input: false

pipeline:
  processors:
    - noop: {}

output:
  stdout: {}
server

http:
  enabled: true
  address: ":localhost:8089"

logger:
  level: INFO
  format: json
  add_timestamp: true

input:
  http_server:
    address: "localhost:8088"
    path: /retry-test
    allowed_verbs:
      - GET
    timeout: 60s
    rate_limit: ""

pipeline:
  processors:
    - log:
        level: INFO
        message: "Received message"
    - sleep:
        duration: 15s
    - mapping: |
        root = {"content": "test"}
output:
  sync_response:
    status: "200"
    headers:
      Content-Type: application/json

This is supposed to simulate a download from an HTTP server that runs into a timeout. The file is supposed to be download once and then processed by a pipeline. My expectation is that this stops after the first attempt, either by retries=0 on the http_client or the true condition on the read_until. However, this just keep sending requests to the server:

Server output

{"@service":"redpanda-connect","benthos_version":"v4.47.1","level":"info","msg":"Running main config from specified file","path":"/benthos/mock_server.yaml","time":"2025-02-21T12:57:07Z"}
{"@service":"redpanda-connect","level":"info","msg":"Listening for HTTP requests at: http://localhost:8089","time":"2025-02-21T12:57:07Z"}
{"@service":"redpanda-connect","label":"","level":"info","msg":"Receiving HTTP messages at: http://localhost:8088/retry-test","path":"root.input","time":"2025-02-21T12:57:07Z"}
{"@service":"redpanda-connect","level":"info","msg":"Launching a Redpanda Connect instance, use CTRL+C to close","time":"2025-02-21T12:57:07Z"}
{"@service":"redpanda-connect","label":"","level":"info","msg":"Output type sync_response is now active","path":"root.output","time":"2025-02-21T12:57:07Z"}
{"@service":"redpanda-connect","custom_source":true,"label":"","level":"info","msg":"Received message","path":"root.pipeline.processors.0","time":"2025-02-21T12:57:27Z"}
{"@service":"redpanda-connect","custom_source":true,"label":"","level":"info","msg":"Received message","path":"root.pipeline.processors.0","time":"2025-02-21T12:57:38Z"}
{"@service":"redpanda-connect","custom_source":true,"label":"","level":"info","msg":"Received message","path":"root.pipeline.processors.0","time":"2025-02-21T12:57:48Z"}

As far as I understand the documentation, it may be valid that the read_until does not cause a stop even with the condition always evaluating to true, as it checks the condition after producing a message. However, the http_client should not retry as this is being set to 0 retries
The only thing that limits the permanent retry is setting the idle_timeout on the read_until. However, in my opinion this should not apply at all as the input is not idle. I suspect this is only checking whether new messages are produced?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant