X-Forwarded-For not being rewritten

[mattkbach]

Hello,

First, thank you for making this plugin .. much appreciated!

The issue i'm facing is that even though the logs state that the X-Forwarded-For header is being rewritten, it does not appear to actually be working. I'm running Traefik v2.7.3

Middleswares config:

    middlewares-real-ip-foo:
      plugin:
        realip:
          Proxy:
            - proxyHeadername: X-From-Cdn
              proxyHeadervalue: cf-my
              realIP: Cf-Connecting-Ip
              overwriteXFF: true

Logs from Traefik (###.###.###.### is the correct external IP):

time="2022-06-29T22:53:20-07:00" level=debug msg="🐸 Current Proxy：cf-my" plugin=plugin-realip module=github.com/Paxxs/traefik-get-real-ip
time="2022-06-29T22:53:20-07:00" level=debug msg="👀 IPs: '1' detail:'[###.###.###.###]'" module=github.com/Paxxs/traefik-get-real-ip plugin=plugin-realip
time="2022-06-29T22:53:20-07:00" level=debug msg="exluded:false， currentIP:###.###.###.###, index:0" plugin=plugin-realip module=github.com/Paxxs/traefik-get-real-ip
time="2022-06-29T22:53:20-07:00" level=debug msg="🐸 Modify XFF to: ###.###.###.###" module=github.com/Paxxs/traefik-get-real-ip plugin=plugin-realip
time="2022-06-29T22:53:20-07:00" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"whoami.##########.ca\",\"Path\":\"/auth/token\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip\"],\"Accept-Language\":[\"en-US,en;q=0.9\"],\"Cdn-Loop\":[\"cloudflare\"],\"Cf-Connecting-Ip\":[\"###.###.###.###\"],\"Cf-Ipcountry\":[\"CA\"],\"Cf-Ray\":[\"7234a41689a5844f-YVR\"],\"Cf-Visitor\":[\"{\\\"scheme\\\":\\\"https\\\"}\"],\"Content-Length\":[\"503\"],\"Content-Type\":[\"multipart/form-data; boundary=----WebKitFormBoundaryv1Vv36paBqVGWFSP\"],\"Cookie\":[\"_forward_auth=#################################|1661121816|###@##########.ca\"],\"Dnt\":[\"1\"],\"Origin\":[\"https://whoami.##########.ca\"],\"Referer\":[\"https://whoami.##########.ca"],\"Sec-Ch-Ua\":[\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"102\\\", \\\"Google Chrome\\\";v=\\\"102\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Windows\\\"\"],\"Sec-Fetch-Dest\":[\"empty\"],\"Sec-Fetch-Mode\":[\"cors\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36\"],\"X-Forwarded-For\":[\"###.###.###.###\"],\"X-Forwarded-Host\":[\"whoami.##########.ca\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"###########\"],\"X-From-Cdn\":[\"cf-my\"],\"X-Real-Ip\":[\"###.###.###.###\"]},\"ContentLength\":503,\"TransferEncoding\":null,\"Host\":\"whoami.##########.ca\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.10.10.1:10388\",\"RequestURI\":\"/auth/token\",\"TLS\":null}"
time="2022-06-29T22:53:20-07:00" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"whoami.##########.ca\",\"Path\":\"/auth/token\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip\"],\"Accept-Language\":[\"en-US,en;q=0.9\"],\"Cdn-Loop\":[\"cloudflare\"],\"Cf-Connecting-Ip\":[\"###.###.###.###\"],\"Cf-Ipcountry\":[\"CA\"],\"Cf-Ray\":[\"7234a41689a5844f-YVR\"],\"Cf-Visitor\":[\"{\\\"scheme\\\":\\\"https\\\"}\"],\"Content-Length\":[\"503\"],\"Content-Type\":[\"multipart/form-data; boundary=----WebKitFormBoundaryv1Vv36paBqVGWFSP\"],\"Cookie\":[\"_forward_auth=#################################|1661121816|###@##########.ca\"],\"Dnt\":[\"1\"],\"Origin\":[\"https://whoami.##########.ca\"],\"Referer\":[\"https://whoami.##########.ca"],\"Sec-Ch-Ua\":[\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"102\\\", \\\"Google Chrome\\\";v=\\\"102\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Windows\\\"\"],\"Sec-Fetch-Dest\":[\"empty\"],\"Sec-Fetch-Mode\":[\"cors\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36\"],\"X-Forwarded-For\":[\"###.###.###.###\"],\"X-Forwarded-Host\":[\"whoami.##########.ca\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"##########\"],\"X-From-Cdn\":[\"cf-my\"],\"X-Real-Ip\":[\"###.###.###.###\"]},\"ContentLength\":503,\"TransferEncoding\":null,\"Host\":\"whoami.##########.ca\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.10.10.1:10388\",\"RequestURI\":\"/auth/token\",\"TLS\":null}" ForwardURL="http://172.17.0.1:8123"
time="2022-06-29T22:53:20-07:00" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"whoami.##########.ca\",\"Path\":\"/auth/token\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip\"],\"Accept-Language\":[\"en-US,en;q=0.9\"],\"Cdn-Loop\":[\"cloudflare\"],\"Cf-Connecting-Ip\":[\"###.###.###.###\"],\"Cf-Ipcountry\":[\"CA\"],\"Cf-Ray\":[\"7234a41689a5844f-YVR\"],\"Cf-Visitor\":[\"{\\\"scheme\\\":\\\"https\\\"}\"],\"Content-Length\":[\"503\"],\"Content-Type\":[\"multipart/form-data; boundary=----WebKitFormBoundaryv1Vv36paBqVGWFSP\"],\"Cookie\":[\"_forward_auth=#################################|1661121816|###@##########.ca\"],\"Dnt\":[\"1\"],\"Origin\":[\"https://whoami.##########.ca\"],\"Referer\":[\"https://whoami.##########.ca"],\"Sec-Ch-Ua\":[\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"102\\\", \\\"Google Chrome\\\";v=\\\"102\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Windows\\\"\"],\"Sec-Fetch-Dest\":[\"empty\"],\"Sec-Fetch-Mode\":[\"cors\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36\"],\"X-Forwarded-For\":[\"###.###.###.###\"],\"X-Forwarded-Host\":[\"whoami.##########.ca\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"##########\"],\"X-From-Cdn\":[\"cf-my\"],\"X-Real-Ip\":[\"###.###.###.###\"]},\"ContentLength\":503,\"TransferEncoding\":null,\"Host\":\"whoami.##########.ca\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.10.10.1:10388\",\"RequestURI\":\"/auth/token\",\"TLS\":null}"

From Whoami (with your plugin enabled):

Hostname: ###########
IP: 127.0.0.1
IP: 192.168.90.22
GET / HTTP/1.1
Host: whoami.###########.ca
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip
Accept-Language: en-US,en;q=0.9
Cache-Control: max-age=0
Cdn-Loop: cloudflare
Cf-Connecting-Ip: ###.###.###.### (Correct IP)
Cf-Ipcountry: CA
Cf-Ray: ###########
Cf-Visitor: {"scheme":"https"}
Cookie: _forward_auth=#############|1661121816|#################
Dnt: 1
Sec-Ch-Ua: " Not A;Brand";v="99", "Chromium";v="102", "Google Chrome";v="102"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
X-Forwarded-For: ###.###.###.###(Correct IP), 10.10.10.1
X-Forwarded-Host: whoami.##########.ca
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: ###########
X-From-Cdn: cf-my
X-Real-Ip: ###.###.###.###(Correct IP)

From Whoami (with your plugin disabled):

Hostname: ##########
IP: 127.0.0.1
IP: 192.168.90.22
GET / HTTP/1.1
Host: whoami.###########.ca
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip
Accept-Language: en-US,en;q=0.9
Cdn-Loop: cloudflare
Cf-Connecting-Ip: ###.###.###.###(Correct IP)
Cf-Ipcountry: CA
Cf-Ray: #############
Cf-Visitor: {"scheme":"https"}
Cookie: _forward_auth=######################|1661121816|####################
Dnt: 1
Sec-Ch-Ua: " Not A;Brand";v="99", "Chromium";v="102", "Google Chrome";v="102"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
X-Forwarded-For: ###.###.###.###(Correct IP), 10.10.10.1
X-Forwarded-Host: whoami.################.ca
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: ##################
X-From-Cdn: cf-my
X-Real-Ip: 10.10.10.1

So with your plugin enabled, the X-Real-IP is being set to the correct IP. The problem is with X-Forwarded-For. In the logs it says its being set to the correct IP, but the 10.10.10.1 address is still in the X-Forwarded-For. Testing with your plugin enabled/disabled, the X-Forwarded-For is the same so it appears as if the X-Forwarded-For is not being rewritten. 10.10.10.1 is the gateway/router IP.

I think I have it setup correctly but some help would be much appreciated!

Thank you :)

[alexdelprete]

First of all, thanks to @Paxxs for this plugin, it's working fine and I love the fact that you can rewrite the realIP based on the content of any header: thanks to this feature, and the fact I'm using cloudflared (tunnel), I have an existing Cf-Warp-Tag-Id that never changes, it probably identifies my tunnel instance, so I didn't need to configure any Transformation Rules on CF, I simply used that header and its constant value in the plugin, and everything's working beautifully. :)

The only problem I noticed is that X-Forwarded-For is not being rewritten, even though in the debug log it seems the plugin says it has been rewritten, exactly like @sleepymatto clearly explained.

I hope this small issue can be fixed, so finally we'll have the perfect realIP plugin.

Keep up the good work. :)

[xanderificnl]

The issue I faced was that no plugin seemed to work, so I took a step back and realized the plugins don't create an inherit trust between upstream and Traefik. What I had to do is make Traefik trust the 'X-Forwarded-For' header, see: 'traefik.yml'.

I'm hoping this is applicable to your situations and this'll help. You may need to update a few things.

Dynamic config

• Rewrite Cf-Connecting-Ip to X-Forwarded-For.
• I'm not filtering any upstream headers yet.

http:
  middlewares:
    real-ip-cf:
      plugin:
        real-ip:
          Proxy:
            - proxyHeadername: "*"
              realIP: Cf-Connecting-Ip
              OverwriteXFF: true

traefik.yml

• Trust the (rewritten) X-Forwarded-For header from 10.0.0.0/8.

entryPoints:
  web:
    address: ":80"
    forwardedHeaders:
      trustedIPs:
        - "10.0.0.0/8"
experimental:
  plugins:
    real-ip:
      moduleName: github.com/Paxxs/traefik-get-real-ip
      version: "v1.0.2"

Docker

• I use annotations w/ docker swarm
• The important bit is traefik.http.routers.xyz.middlewares: real-ip-cf@file – you can also enter this in your dynamic config.

---
version: "3.3"
services:
  xyz:
    deploy:
      labels:
        traefik.enable: 'true'
        traefik.http.routers.xyz.middlewares: real-ip-cf@file
        traefik.http.services.xyz.loadbalancer.server.port: 8888
        traefik.docker.network: ingress_default

Result:

• Traefik access log

2a02:58:xxxxxxxxx - - [26/Sep/2022:19:06:48 +0000] "GET

[alexdelprete]

What I had to do is make Traefik trust the 'X-Forwarded-For' header, see: 'traefik.yml'.

I'm hoping this is applicable to your situations and this'll help. You may need to update a few things.

I already had this configuration, but thanks anyway for the heads up. As I wrote before, the debug logs of the plugin show that the XFF header has been rewritten, but that is not actually the case.

Did you check that XFF in your case is being rewritten?

[Paxxs]

Hi @mattkbach,

I appreciate your detailed testing and feedback. Thank you for using my plugin and for bringing this issue to my attention.

@alexdelprete also reported a similar issue. Regarding the X-Forwarded-For header, I believe that Traefik's middleware may not be able to completely overwrite X-Forwarded-For header, but only modify the last few IP values. Therefore, even though the logs indicate that the X-Forwarded-For header is being overwritten, it may not actually be working as expected.

The current function of the OverwriteXFF feature in this plugin is to clean the content of the X-Forwarded-For header.

For example, with the following configuration:

if you want the plugin to modify the X-Forwarded-For header, you need to configure Traefik.

you can refer to the method provided by @xanderificnl. (thanks @xanderificnl XD

# traefik static configuration
entryPoints:
  http:
    address: ":8000"
    forwardedHeaders:
      insecure: true

# dynamic plugin configuration
Proxy:
    - proxyHeadername: X-From-Cdn
      proxyHeadervalue: 1-cdn
      realIP: Client-Ip
      OverwriteXFF: true

Using the curl command with added headers to directly request whoami (###.###.###.### is the correct external IP)：

❯ curl -H "X-From-Cdn: 1-cdn" \
      -H "X-Forwarded-For: this all fake ip, 111.11.11.1, 2.2.2.2" \
      -H "Client-Ip: ###.###.###.###" \
      http://[::1]:8000/whoami
Hostname: 60accadf3b58
IP: 127.0.0.1
IP: 172.18.0.3
RemoteAddr: 172.18.0.2:41770
GET /whoami HTTP/1.1
Host: [::1]:8000
User-Agent: curl/8.1.2
Accept: */*
Accept-Encoding: gzip
Client-Ip: ###.###.###.###
X-Forwarded-For: this all fake ip, 111.11.11.1, 2.2.2.2, 172.18.0.1
X-Forwarded-Host: [::1]:8000
X-Forwarded-Port: 8000
X-Forwarded-Proto: http
X-Forwarded-Server: 9d29b1ddae77
X-From-Cdn: 1-cdn
X-Real-Ip: 172.18.0.1

But with the plugin enabled in the request:

❯ curl -H "X-From-Cdn: 1-cdn" \
      -H "X-Forwarded-For: this all fake ip, 111.11.11.1, 2.2.2.2" \
      -H "Client-Ip: ###.###.###.###" \
      http://[::1]:8000/whoami
Hostname: 60accadf3b58
IP: 127.0.0.1
IP: 172.18.0.3
RemoteAddr: 172.18.0.2:47034
GET /whoami HTTP/1.1
Host: [::1]:8000
User-Agent: curl/8.1.2
Accept: */*
Accept-Encoding: gzip
Client-Ip: ###.###.###.###
X-Forwarded-For: ###.###.###.###, 172.18.0.1
X-Forwarded-Host: [::1]:8000
X-Forwarded-Port: 8000
X-Forwarded-Proto: http
X-Forwarded-Server: 9d29b1ddae77
X-From-Cdn: 1-cdn
X-Real-Ip: ###.###.###.###

At the moment, I don't have a way to fully overwrite the content of the X-Forwarded-For header. If you find any way to achieve this, please let me know :)

Thanks again for using my plugin and for your feedback! 💗 I would be happy to help you with this issue.

Best regards.

[firestrife23]

This is how I managed to make it work.

entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
          permanent: true
    forwardedHeaders:
      trustedIPs: &trustedIps
        # Local IP
        - 127.0.0.1/32
        - 192.168.1.0/24
        - 172.31.0.0/16
        # Cloudflare CDN and WARP
        - 173.245.48.0/20
        - 103.21.244.0/22
        - 103.22.200.0/22
        - 103.31.4.0/22
        - 141.101.64.0/18
        - 108.162.192.0/18
        - 190.93.240.0/20
        - 188.114.96.0/20
        - 197.234.240.0/22
        - 198.41.128.0/17
        - 162.158.0.0/15
        - 104.16.0.0/13
        - 104.24.0.0/14
        - 172.64.0.0/13
        - 131.0.72.0/22
        - 2400:cb00::/32
        - 2606:4700::/32
        - 2803:f800::/32
        - 2405:b500::/32
        - 2405:8100::/32
        - 2a06:98c0::/29
        - 2c0f:f248::/32
  https:
    address: ":443"
    http3:
      advertisedPort: 443
    forwardedHeaders:
      trustedIPs: *trustedIps