network: add support for tcp keepalive probes #3096

abiliojr · 2021-02-21T09:31:49Z

Signed-off-by: Abilio Marques [email protected]

The term TCP keepalive is been widely used in networking to describe a mechanism totally different than just "not closing the connection after a request". This PR adds support for real TCP keepalive probing. A matching PR with suggestions on how to make these changes clearer in the documentation is also available.

TCP keepalives have 2 potentially useful applications:

To prevent disconnection due to network inactivity: any TCP connection is subject to be silently dropped by intermediate equipment in the network (e.g., routers) if it's quiet for too long.
To detect dead connections, either because of a crash on the other side, or because the network channel has gone down.

All happens at the transport layer, without any intervention, and is transparent to upper (e.g., HTTP) layers.

For a more detailed description, please see: https://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html

TCP-keepalive values could be configured system wide, but this is definitely not ideal, as it still needs to be enabled by the application. The other option is to do it per socket. For that, fluentbit needs to apply the values to the socket. This patch adds support for that. By default, it's disabled.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

Documentation required for this feature

Documentation provided here: fluent/fluent-bit-docs#471

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

abiliojr · 2021-02-21T09:32:48Z

Configuration example:

[INPUT]
    Name mqtt
    Port 1885
    Tag demo

[OUTPUT]
    Name http
    Match *
    host localhost
    port 3000
    URI /messages
    net.keepalive_idle_timeout 180
    net.tcp_keepalive_time 30
    net.tcp_keepalive_interval 15
    net.tcp_keepalive_probes 3

abiliojr · 2021-02-21T09:45:41Z

Debug log:

Fluent Bit v1.8.0
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/02/21 10:44:20] [ info] Configuration:
[2021/02/21 10:44:20] [ info]  flush time     | 5.000000 seconds
[2021/02/21 10:44:20] [ info]  grace          | 5 seconds
[2021/02/21 10:44:20] [ info]  daemon         | 0
[2021/02/21 10:44:20] [ info] ___________
[2021/02/21 10:44:20] [ info]  inputs:
[2021/02/21 10:44:20] [ info]      mqtt
[2021/02/21 10:44:20] [ info] ___________
[2021/02/21 10:44:20] [ info]  filters:
[2021/02/21 10:44:20] [ info] ___________
[2021/02/21 10:44:20] [ info]  outputs:
[2021/02/21 10:44:20] [ info]      http.0
[2021/02/21 10:44:20] [ info] ___________
[2021/02/21 10:44:20] [ info]  collectors:
[2021/02/21 10:44:20] [ info] [engine] started (pid=7968)
[2021/02/21 10:44:20] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2021/02/21 10:44:20] [debug] [storage] [cio stream] new stream registered: mqtt.0
[2021/02/21 10:44:20] [ info] [storage] version=1.1.0, initializing...
[2021/02/21 10:44:20] [ info] [storage] in-memory
[2021/02/21 10:44:20] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/02/21 10:44:20] [ info] [input:mqtt:mqtt.0] listening on 0.0.0.0:1885
[2021/02/21 10:44:20] [debug] [http:http.0] created event channels: read=19 write=20
[2021/02/21 10:44:20] [debug] [router] match rule mqtt.0:http.0
[2021/02/21 10:44:20] [ info] [sp] stream processor started
[2021/02/21 10:44:28] [debug] [input:mqtt:mqtt.0] [fd=25] new TCP connection
[2021/02/21 10:44:30] [debug] [task] created task=0x7f07a8009210 id=0 OK
[2021/02/21 10:44:30] [debug] [http_client] not using http_proxy for header
[2021/02/21 10:44:30] [debug] [http_client] header=POST /messages HTTP/1.1
Host: localhost:3000
Content-Length: 34
Content-Type: application/msgpack
User-Agent: Fluent-Bit


[2021/02/21 10:44:30] [ info] [output:http:http.0] localhost:3000, HTTP status=200
{"status": "received"}
[2021/02/21 10:44:30] [debug] [upstream] KA connection #25 to localhost:3000 is now available
[2021/02/21 10:44:30] [debug] [out coro] cb_destroy coro_id=0
[2021/02/21 10:44:30] [debug] [task] destroy task=0x7f07a8009210 (task_id=0)
^C[2021/02/21 10:44:51] [engine] caught signal (SIGINT)
[2021/02/21 10:44:51] [ warn] [engine] service will stop in 5 seconds
[2021/02/21 10:44:56] [ info] [engine] service stopped

abiliojr · 2021-02-21T09:47:06Z

Valgrind report:

==8219== 
==8219== HEAP SUMMARY:
==8219==     in use at exit: 0 bytes in 0 blocks
==8219==   total heap usage: 513 allocs, 513 frees, 1,032,942 bytes allocated
==8219== 
==8219== All heap blocks were freed -- no leaks are possible
==8219== 
==8219== For counts of detected and suppressed errors, rerun with: -v
==8219== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

abiliojr · 2021-02-21T09:50:39Z

Sample extracted using wireshark (configuration was 15 seconds during this test):

No.     Time           Source                Destination           Protocol Length Info                                                            Source Port Destination Port
     12 15.203402344   127.0.0.1             127.0.0.1             TCP      66     [TCP Keep-Alive] 35680 → 3000 [ACK] Seq=162 Ack=180 Win=65536 Len=0 TSval=2983848706 TSecr=2983833505 35680       3000

Frame 12: 66 bytes on wire (528 bits), 66 bytes captured (528 bits) on interface lo, id 0
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1
Transmission Control Protocol, Src Port: 35680, Dst Port: 3000, Seq: 162, Ack: 180, Len: 0

edsiper · 2021-02-22T21:19:33Z

src/flb_io.c

@@ -102,6 +102,13 @@ int flb_io_net_connect(struct flb_upstream_conn *u_conn,
                  u_conn->fd, u->tcp_host, u->tcp_port);
    }

+    /* set TCP keepalive if configured */
+    ret = flb_net_socket_tcp_keepalive(fd, &u->net);


this line is changing the default system values, I think this call must be optional based on if the user set some parameters. I mean: all the internal tcp_something fields should have a non-set value like -1 and IF in the invoked call that value is >= 0, then perform the syscall to change the behavior.

changed. Added a boolean: net.tcp_keepalive, that allows enabling keepalives while keeping the system-wide default values

src/flb_network.c

edsiper · 2021-02-22T21:21:30Z

src/flb_upstream.c

@@ -65,6 +65,25 @@ struct flb_config_map upstream_net[] = {
     "before it is retired."
    },

+    {
+     FLB_CONFIG_MAP_INT, "net.tcp_keepalive_time", "0",


I think default values should be avoided since every system can have an implicit default that might not match this 0. I am thinking mostly on current users that my face a different internal setup due to this change.

Replaced with -1, and as indicated in other comment, the system-wide default will be then used if the configuration does not specify a desired value

Signed-off-by: Abilio Marques <[email protected]>

github-actions · 2021-03-26T02:07:22Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

fdr · 2022-03-07T23:16:17Z

I have some need for this particular feature, presuming it can work with out_cloudwatch_logs, to work around #4606. It looks like a lot of the work has been done though there has not been correspondence in some time. If it's a bit more effort to get this committed, please let me know.

Sidebar: though perhaps more suitable for 1.9 given the change level, for reasons like bug #4606, I think it's a bad idea to not have keepalive set when doing synchronous I/O unless some unusual things are done to time out blocking system calls. i.e., there should be a default.

PettitWesley · 2022-03-16T06:39:02Z

@abiliojr @edsiper Can we resurrect this PR?

I am hoping it can fix this issue: aws/aws-for-fluent-bit#293

Do you have recommended values for the settings?

github-actions · 2022-06-15T02:19:42Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

leonardo-albertovich

With those minimal changes this should be ready to be merged.

leonardo-albertovich · 2024-08-19T10:56:59Z

src/flb_network.c

+    int intvl = net->tcp_keepalive_interval;
+    int probes = net->tcp_keepalive_probes;
+
+    printf("keepalive = %d, %d %d\n", time, intvl, probes);


Please remove this line.

leonardo-albertovich · 2024-08-19T11:01:55Z

src/flb_network.c

+/*
+ * Enable TCP keepalive
+ */
+int flb_net_socket_tcp_keepalive(flb_sockfd_t fd, struct flb_net_setup *net)


Please replace the bitwise assignments of ret with regular assignments and add ret == 0 && as a prefix to the conditional blocks so a failed setsockopt call is not overwritten and thus missed by the conditional in line 221.

Please change the conditional in line 221 so it explicitly compares against zero (ie. if (ret != 0) {).

This is important because apparently according to msdn, windows doesn't accept values larger than 255 for TCP_KEEPCNT whereas other operating systems accept much larger values,.

leonardo-albertovich · 2024-08-19T11:06:50Z

src/flb_io.c

@@ -102,6 +102,15 @@ int flb_io_net_connect(struct flb_upstream_conn *u_conn,
                  u_conn->fd, u->tcp_host, u->tcp_port);
    }

+    /* set TCP keepalive and it's options */
+    if (u->net.tcp_keepalive != FLB_FALSE) {


Please either use if (u->net.tcp_keepalive == FLB_TRUE) { or the preferred syntax according to the coding style if (u->net.tcp_keepalive) {.

leonardo-albertovich · 2024-08-19T11:09:42Z

include/fluent-bit/flb_network.h

@@ -43,6 +43,18 @@ struct flb_net_setup {

    /* maximum of times a keepalive connection can be used */
    int keepalive_max_recycle;
+
+    /* enable/disable tcp keepalive */
+    char tcp_keepalive;


This property needs to be an int

leonardo-albertovich · 2024-08-19T11:11:27Z

@abiliojr if you are not able to make those changes please let me know and I'll make them for you.

Thanks a lot!

abiliojr · 2024-08-19T16:55:34Z

@leonardo-albertovich , I haven't used fluentbit for at least a couple of years. Please go ahead and make any needed changes.

edsiper · 2024-08-19T19:10:34Z

Closing this as changes are part of #9249 .

abiliojr requested review from edsiper, fujimotos and koleini as code owners February 21, 2021 09:31

abiliojr mentioned this pull request Feb 21, 2021

network: add documentation for TCP keepalive probes fluent/fluent-bit-docs#471

Merged

edsiper requested changes Feb 22, 2021

View reviewed changes

network: add support for tcp keepalive probes

58be375

Signed-off-by: Abilio Marques <[email protected]>

abiliojr force-pushed the tcp_keepalive branch from f9ac8e4 to 58be375 Compare February 23, 2021 06:46

github-actions bot added the Stale label Mar 26, 2021

github-actions bot removed the Stale label Mar 8, 2022

PettitWesley mentioned this pull request Mar 16, 2022

cloudwatch_logs blocks on read forever #4606

Closed

github-actions bot added the Stale label Jun 15, 2022

edsiper requested a review from leonardo-albertovich as a code owner August 14, 2024 17:58

github-actions bot removed the Stale label Aug 15, 2024

leonardo-albertovich requested changes Aug 19, 2024

View reviewed changes

leonardo-albertovich mentioned this pull request Aug 19, 2024

network: tcp keepalive addition #9249

Merged

edsiper added this to the Fluent Bit v3.2.0 milestone Aug 19, 2024

edsiper closed this Aug 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

network: add support for tcp keepalive probes #3096

network: add support for tcp keepalive probes #3096

abiliojr commented Feb 21, 2021 •

edited

Loading

abiliojr commented Feb 21, 2021

abiliojr commented Feb 21, 2021

abiliojr commented Feb 21, 2021

abiliojr commented Feb 21, 2021

edsiper Feb 22, 2021

abiliojr Feb 23, 2021

edsiper Feb 22, 2021

abiliojr Feb 23, 2021

github-actions bot commented Mar 26, 2021

fdr commented Mar 7, 2022 •

edited

Loading

PettitWesley commented Mar 16, 2022

github-actions bot commented Jun 15, 2022

leonardo-albertovich left a comment

leonardo-albertovich Aug 19, 2024

leonardo-albertovich Aug 19, 2024

leonardo-albertovich Aug 19, 2024

leonardo-albertovich Aug 19, 2024

leonardo-albertovich Aug 19, 2024

leonardo-albertovich commented Aug 19, 2024

abiliojr commented Aug 19, 2024

edsiper commented Aug 19, 2024

network: add support for tcp keepalive probes #3096

network: add support for tcp keepalive probes #3096

Conversation

abiliojr commented Feb 21, 2021 • edited Loading

Documentation provided here: fluent/fluent-bit-docs#471

abiliojr commented Feb 21, 2021

abiliojr commented Feb 21, 2021

abiliojr commented Feb 21, 2021

abiliojr commented Feb 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Mar 26, 2021

fdr commented Mar 7, 2022 • edited Loading

PettitWesley commented Mar 16, 2022

github-actions bot commented Jun 15, 2022

leonardo-albertovich left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leonardo-albertovich commented Aug 19, 2024

abiliojr commented Aug 19, 2024

edsiper commented Aug 19, 2024

abiliojr commented Feb 21, 2021 •

edited

Loading

fdr commented Mar 7, 2022 •

edited

Loading