Automatically detect blocked anonymized DNS relay servers according to transport layer protocol

[InsaneKnight]

I once used the following config for anonymized DNS relay config:

[anonymized_dns]
  routes = [ { server_name='*', via=['*'] } ]

But some Autonomous system may block unfamiliar protocol over TCP (e.g. anonymized DNSCrypt protocol over port 443). As I observed, they do so by swallowing the outgoing SYN packets, leaving a lot of TCP connections stuck in SYN state. When observing my dnscrypt-proxy 2.0.45 via its log, I could see [WARNING] Too many incoming connections, as such:

[2022-06-30 02:55:37] [NOTICE] Anonymizing queries for [dnscrypt.ca-2] via [anon-cs-ore]
[2022-06-30 02:55:38] [NOTICE] Anonymizing queries for [pwoss.org-dnscrypt] via [anon-cs-de]
[2022-06-30 02:55:43] [NOTICE] [pwoss.org-dnscrypt] TIMEOUT
[2022-06-30 02:55:43] [NOTICE] Anonymizing queries for [resolver4.dns.openinternet.io] via [anon-cs-finland]
[2022-06-30 02:55:50] [NOTICE] Anonymizing queries for [wevpn-useast] via [anon-serbica]
[2022-06-30 02:55:51] [NOTICE] Anonymizing queries for [pryv8boi] via [anon-inconnu]
[2022-06-30 02:49:40] [WARNING] Too many incoming connections (max=250)
[2022-06-30 02:49:45] [WARNING] Too many incoming connections (max=250)
[2022-06-30 02:55:37] [NOTICE] Anonymizing queries for [dnscrypt.ca-2] via [anon-cs-ore]
[2022-06-30 02:55:38] [NOTICE] Anonymizing queries for [pwoss.org-dnscrypt] via [anon-cs-de]
[2022-06-30 02:55:43] [NOTICE] [pwoss.org-dnscrypt] TIMEOUT
[2022-06-30 02:55:43] [NOTICE] Anonymizing queries for [resolver4.dns.openinternet.io] via [anon-cs-finland]
[2022-06-30 02:55:50] [NOTICE] Anonymizing queries for [wevpn-useast] via [anon-serbica]
[2022-06-30 02:55:51] [NOTICE] Anonymizing queries for [pryv8boi] via [anon-inconnu]
[2022-06-30 02:49:08] [WARNING] Too many incoming connections (max=250)
[2022-06-30 02:49:13] [WARNING] Too many incoming connections (max=250)

but in reality, it is OUTGOING TCP connections stuck in SYN state that crowd out the connection pool, and a great amount of them are observable via lsof(8) -np.

I do not know exactly how dnscrypt-proxy handle network traffics for (anonymized) DNSCrypt protocol, but TCP connections to a server being blocked does not means UDP accesses too being blocked, since UDP is not connection-oriented. A (relay) server may be accessible via UDP, but TCP connections to it are blocked by the intermediary AS. In "Anonymizing queries for" state dnscrypt-proxy may successfully access the server via UDP, and mark the server usable, but when it later tries to query the same server via TCP (may be for queries larger than MTU), connections will get blocked, but dnscrypt-proxy 2.0.45 will keep trying dozens of time, leaving a lot of connections stuck in SYN state crowding out the connection pool.

Now I have to write some scripts to pick relay servers without TCP connections to them blocked, and only use these servers in the config file, but TCP-blocked servers should be programmatically detectable.

I suggest that dnscrypt-proxy should be able to detect and mark whether a DNSCrypt (relay) server is accessible via TCP and UDP independently, and refuse to perform queries via the blocked transport layer protocol until the next checking for configured servers.

[lifenjoiner]

Some of my thoughts:

1. Try the latest stable release.
2. I guess it will not be that many "stuck" TCP connections exhaust the max_clients, there should be many queries.
3. Did lsof(8) -np show many TCP connections related to dnscrypt-proxy. If yes, how many are connecting to, and how many are connecting out from dnscrypt-proxy?
   Do you specify force_tcp = true or use ODoH? It makes connecting-out in TCP. Or do your apps (downstream DNS) choose only TCP? It makes connecting-to in TCP.
4. Try to figure out how many queries are made to dnscrypt-proxy? If there are many, are they normal?
5. max=250 is high enough for a normal local DNS. If all your queries are normal, it is not enough for your scenario, increase max_clients.
6. The relays are set up on purpose to do the proxy.
   SYN is just a request of TCP handshakes, which are maintained by the TCP driver automatically. If a server got a request, it will response. SYN will not be forwarded/proxied, there is no payload.
   There is possible that, the SYN is dropped by ISP, for example.
   Most of the time, SYN indicates a Dial , which would time out easily. Did you see the connections are really stuck, or connection flood?
7. By default, dnscrypt-proxy tries UDP query first.
8. dnscrypt-proxy has validation, server ranking and load balancing algorithm try to make better service.

Maybe the issue is unique in your scenario, others neither have it nor can reproduce it. It will need you provide more exact information to get helped.

[InsaneKnight]

Did lsof(8) -np show many TCP connections related to dnscrypt-proxy?

Yes, the '-p' option should be followed the PID of the process to be observed, and will only show connections for that process. When observing the running dnscrypt-proxy on my computer just after the log with [WARNING] Too many incoming connections (max=250), I can see so many OUTGOING connections stuck in SYN state, usually to the same server, with seldom incoming TCP connection, like this (with [anonymized_dns] set to routes = [ { server_name='*', via=['*'] } ] and restart dnscrypt-proxy):

$ sudo lsof -np 3740006
COMMAND       PID            USER   FD      TYPE             DEVICE SIZE/OFF     NODE NAME
dnscrypt- 3740006 _dnscrypt-proxy  cwd       DIR              253,1     4096   671793 /etc/dnscrypt-proxy
dnscrypt- 3740006 _dnscrypt-proxy  rtd       DIR              253,1     4096        2 /
dnscrypt- 3740006 _dnscrypt-proxy  txt       REG              253,1  8776056   919410 /usr/sbin/dnscrypt-proxy
dnscrypt- 3740006 _dnscrypt-proxy  mem       REG              253,1    27040   788611 /lib/x86_64-linux-gnu/libnss_dns-2.33.so
dnscrypt- 3740006 _dnscrypt-proxy  mem       REG              253,1    93080   788674 /lib/x86_64-linux-gnu/libresolv-2.33.so
dnscrypt- 3740006 _dnscrypt-proxy  mem       REG              253,1    47664   788612 /lib/x86_64-linux-gnu/libnss_files-2.33.so
dnscrypt- 3740006 _dnscrypt-proxy  mem       REG              253,1  1900992   788540 /lib/x86_64-linux-gnu/libc-2.33.so
dnscrypt- 3740006 _dnscrypt-proxy  mem       REG              253,1   143768   788671 /lib/x86_64-linux-gnu/libpthread-2.33.so
dnscrypt- 3740006 _dnscrypt-proxy  mem       REG              253,1    18504   788367 /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
dnscrypt- 3740006 _dnscrypt-proxy  mem       REG              253,1   202552   787077 /lib/x86_64-linux-gnu/ld-2.33.so
dnscrypt- 3740006 _dnscrypt-proxy    0r      CHR                1,3      0t0        4 /dev/null
dnscrypt- 3740006 _dnscrypt-proxy    1u     unix 0x00000000e7b94454      0t0 27926877 type=STREAM (CONNECTED)
dnscrypt- 3740006 _dnscrypt-proxy    2u     unix 0x00000000e7b94454      0t0 27926877 type=STREAM (CONNECTED)
dnscrypt- 3740006 _dnscrypt-proxy    3u     IPv4           27929011      0t0      TCP 192.168.0.2:44028->37.120.235.187:https (SYN_SENT)
dnscrypt- 3740006 _dnscrypt-proxy    4w      REG              253,8  6468942  1574063 /var/log/dnscrypt-proxy/query.log
dnscrypt- 3740006 _dnscrypt-proxy    5u  a_inode               0,14        0     8267 [eventpoll:3,6,8,9,10,11,13,14,15,16,17,18,19,20,21,25,26,27,29,30,31,32,33,34,35,36,38,39,40,41,42,43...]
dnscrypt- 3740006 _dnscrypt-proxy    6r     FIFO               0,13      0t0 27925290 pipe
dnscrypt- 3740006 _dnscrypt-proxy    7w     FIFO               0,13      0t0 27925290 pipe
dnscrypt- 3740006 _dnscrypt-proxy    8u     IPv4           27926886      0t0      TCP 127.0.0.1:3000 (LISTEN)
dnscrypt- 3740006 _dnscrypt-proxy    9u     IPv4              20195      0t0      TCP 127.0.2.1:domain (LISTEN)
dnscrypt- 3740006 _dnscrypt-proxy   10u     IPv4              20197      0t0      UDP 127.0.2.1:domain 
dnscrypt- 3740006 _dnscrypt-proxy   11u     IPv4           27927913      0t0      TCP 192.168.0.2:50144->37.120.235.187:https (SYN_SENT)
dnscrypt- 3740006 _dnscrypt-proxy   12w      REG              253,8  9780201  1575739 /var/log/dnscrypt-proxy/nx.log
dnscrypt- 3740006 _dnscrypt-proxy   13u     IPv4           27927308      0t0      TCP 192.168.0.2:50140->37.120.235.187:https (SYN_SENT)
dnscrypt- 3740006 _dnscrypt-proxy   14u     IPv4           27927859      0t0      TCP 192.168.0.2:50110->37.120.235.187:https (SYN_SENT)
dnscrypt- 3740006 _dnscrypt-proxy   15u     IPv4           27927285      0t0      TCP 192.168.0.2:50112->37.120.235.187:https (SYN_SENT)
dnscrypt- 3740006 _dnscrypt-proxy   16u     IPv4           27927868      0t0      TCP 192.168.0.2:50124->37.120.235.187:https (SYN_SENT)
dnscrypt- 3740006 _dnscrypt-proxy   17u     IPv4           27929747      0t0      TCP 192.168.0.2:50156->37.120.235.187:https (SYN_SENT)
dnscrypt- 3740006 _dnscrypt-proxy   18u     IPv4           27927930      0t0      TCP 192.168.0.2:50160->37.120.235.187:https (SYN_SENT)
dnscrypt- 3740006 _dnscrypt-proxy   19u     IPv4           27927336      0t0      TCP 192.168.0.2:50176->37.120.235.187:https (SYN_SENT)
dnscrypt- 3740006 _dnscrypt-proxy   20u     IPv4           27928945      0t0      TCP 192.168.0.2:50178->37.120.235.187:https (SYN_SENT)
dnscrypt- 3740006 _dnscrypt-proxy   21u     IPv4           27927152      0t0      TCP 192.168.0.2:37448->193.70.85.11:https (ESTABLISHED)
dnscrypt- 3740006 _dnscrypt-proxy   22u     IPv4           27927358      0t0      TCP 192.168.0.2:50186->37.120.235.187:https (SYN_SENT)
dnscrypt- 3740006 _dnscrypt-proxy   23u     IPv4           27929755      0t0      TCP 192.168.0.2:50200->37.120.235.187:https (SYN_SENT)
dnscrypt- 3740006 _dnscrypt-proxy   24u     IPv4           27928035      0t0      TCP 192.168.0.2:50202->37.120.235.187:https (SYN_SENT)
dnscrypt- 3740006 _dnscrypt-proxy   25u     IPv4           27927387      0t0      TCP 192.168.0.2:44042->37.120.235.187:https (SYN_SENT)
dnscrypt- 3740006 _dnscrypt-proxy   27u     IPv4           27929013      0t0      UDP 192.168.0.2:40382->37.120.235.187:https 
dnscrypt- 3740006 _dnscrypt-proxy   28u     IPv4           27929791      0t0      UDP 127.0.0.1:41254->127.0.2.1:domain 
dnscrypt- 3740006 _dnscrypt-proxy   29u     IPv4           27929792      0t0      UDP 192.168.0.2:42948->37.120.235.187:https 
dnscrypt- 3740006 _dnscrypt-proxy   30u     IPv4           27929802      0t0      UDP 192.168.0.2:46375->37.120.235.187:https 
dnscrypt- 3740006 _dnscrypt-proxy   31u     IPv4           27929022      0t0      TCP 192.168.0.2:44046->37.120.235.187:https (SYN_SENT)
dnscrypt- 3740006 _dnscrypt-proxy   32u     IPv4           27929803      0t0      TCP 192.168.0.2:44052->37.120.235.187:https (SYN_SENT)
dnscrypt- 3740006 _dnscrypt-proxy   33u     IPv4           27927394      0t0      TCP 192.168.0.2:44054->37.120.235.187:https (SYN_SENT)
dnscrypt- 3740006 _dnscrypt-proxy   34u     IPv4           27928052      0t0      UDP 192.168.0.2:41233->37.120.235.187:https 
dnscrypt- 3740006 _dnscrypt-proxy   35u     IPv4           27929810      0t0      UDP 192.168.0.2:32801->37.120.235.187:https 
dnscrypt- 3740006 _dnscrypt-proxy   36u     IPv4           27927410      0t0      TCP 192.168.0.2:44068->37.120.235.187:https (SYN_SENT)
dnscrypt- 3740006 _dnscrypt-proxy   37u     IPv4           27929756      0t0      UDP 192.168.0.2:60659->37.120.235.187:https 
dnscrypt- 3740006 _dnscrypt-proxy   38u     IPv4           27928034      0t0      UDP 192.168.0.2:52585->37.120.235.187:https 
dnscrypt- 3740006 _dnscrypt-proxy   39u     IPv4           27929764      0t0      UDP 192.168.0.2:54206->37.120.235.187:https 
dnscrypt- 3740006 _dnscrypt-proxy   40u     IPv4           27929765      0t0      UDP 192.168.0.2:47175->37.120.235.187:https 
dnscrypt- 3740006 _dnscrypt-proxy   41u     IPv4           27928064      0t0      UDP 192.168.0.2:59707->37.120.235.187:https 
dnscrypt- 3740006 _dnscrypt-proxy   42u     IPv4           27929818      0t0      UDP 192.168.0.2:39882->37.120.235.187:https 
dnscrypt- 3740006 _dnscrypt-proxy   43u     IPv4           27929819      0t0      UDP 192.168.0.2:51818->37.120.235.187:https

where 37.120.235.187:https is anon-cs-ireland, and nc -z 37.120.235.187 443 will return 1, indicating TCP connections to it gets blocked.

Do you specify force_tcp = true or use ODoH?

No.

There is possible that, the SYN is dropped by ISP, for example.

This is highly probable, but when most TCP connections to a specific relay/server will timeout, should dnscrypt-proxy avoid using TCP to access this relay/server?

Did you see the connections are really stuck, or connection flood?

They are stuck, but it looks very similar that dnscrypt-proxy is trying to SYN-flood the server, before these connection (trials) get timed out 5 seconds later.

[lifenjoiner]

With anonymized DNSCrypt (routes = [ { server_name='*', via=['*'] } ]) and force_tcp = false:

1. The relay is chosen when a server is estimated by
   

   dnscrypt-proxy/dnscrypt-proxy/serversInfo.go

   Lines 474 to 478 in 38e87f9

   	if !wildcard || len(relayStamps) == 1 {
   	relayCandidateStamp = &relayStamps[rand.Intn(len(relayStamps))]
   	} else {
   	relayCandidateStamp = findFarthestRoute(proxy, name, relayStamps)
   	}

   
   the relay-server are ranked and then used in pair;
2. The relay stands in the middle, and stands for the server's quality if it is less usable;
3. If the pair is available,dnscrypt-proxy always tries in UDP first;
4. and then tries in TCP, if UDP get truncated response or is timeout;
5. once the TCP query to upstream fails, the corresponding pair's average delay increases;
6. then, in the next server pick-up for a new incoming client query, the previous pair gets less possibility to win, unless others are worse.

The logic has no problem.

SYN_SENT stuck

Any SYN_SENT should wait corresponding time until timeout. Waiting (5s) is necessary. And it do help you find out the issue with those relays.

There are 19 TCP and 12 UDP connecting-outs to those relays in your paste. That is not too many. There shouldn't be full client workload all the time.
For one incoming query, there can be more than one connecting-outs:

1. only of them returns, the client will be answered, and client count decreases;
2. if all are timeout, the pair will be downgraded, like previously described.

And for the relays you listed, UDP should be very slow to timeout.

Here are 2 inferences:

1. Those pairs were available when they were estimated, but now they are unavailable or low quality. Seems your network is not very stable.
2. Those low quality pairs are picked up again before the 1st use timeout.

So, for the [WARNING] Too many incoming connections (max=250):

1. Occasional happening is OK. It depends on the workload of your computer. Turn up max_clients, if necessary.
2. Take a look at the [NOTICE] Sorted latencies:. If many connecting-outs are waiting to timeout,
   2.1. those latencies could be high, the main problem is your network;
   2.2. if the latencies are normal, your network is unstable, try turn down cert_refresh_delay.
3. Specify good quality relays.

I guess the key problem is the network is not stable enough. Detecting for each query is not good strategy, it still may wait for timeout.