-
-
Notifications
You must be signed in to change notification settings - Fork 235
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CSRF verification failed if behind a reverse proxy (traefik) #403
Comments
Thankyou very much!! Great work!! |
Had the very same issue and traced it to the same cause. |
You are my hero! Had the same problem. Looked around for a fix and finally thought I might not be the only idiot to upgrade right after release - let's look at the GitHub issues. Thank you 👍 |
As noted in #24601, the right fix is to set We can try to highlight this new requirement better in the upgrading documentation, but I want to verify that doing that fixes it for you. Setting |
X-Forwarded-Proto is correctly set and has been working since i've setup my Zulip instance (version 5). It's the update to version 7 that required using |
Ah -- the other half that changed in 7.0 is that we only trust the X-Forwarded-Proto if it is set by a trusted proxy. Do you have You can check if that 'correctly set by looking at the nginx access logs, and seeing if it's the IP of the traefik gateway, or the end-user. |
See this comment on chat.zulip.org where someone else was configuring traefik requests to come from a static IP so Zulip can know to trust them. |
I personally don't use traefik, i have my own solution based on nginx-proxy, maybe @chgayot can shed some light on that part. In any case, if there's a need to define |
Following the link and suggestion work, but I wasn't able to get it working on domain names rather than IPs, and I had to change the Traefik configuration which makes it a workaround rather than a production solution. It puts the entry barrier of docker-zulip really out of reach for beginners like me :( For someone who knows more than me, there must be an init line or something similar we can run to get traefik's IP on container launch and set it up. To sum up what I had to do: In the TRAEFIK docker-compose:
AND
Finally, "just" set, in ZULIP docker-compose
No need for SETTING_CSRF_TRUSTED_ORIGINS. But I'm all ear for the right solution! |
(continuing the discussion from #404, which is now closed) The issue is not that When using reverse proxy, the request goes trough two proxies:
In other words, there are three connections: User agent <- 1 -> outer proxy <- 2 -> inner nginx <- 3 -> zulip. When we configure the outer proxy, the However, the inner nginx sets it's own That means, if the outer proxy talks to inner nginx via http and not https (which is sensible if both are on the same machine), the inner nginx will tell to the zulip app that As a workaround, configure the connection 2 to use https and remove The P.S.: The |
@chgayot wrote:
I'm not clear what you mean by "domain names rather than IPs" -- can you clarify? But changing the Traefik configuration seems necessary here, not a "workaround" -- nginx needs to know where the proxy is for it to be able to unroll
I'm not familiar with Traefik, but my belief is that if you don't set the IP explicitly, it may actually move around as Docker reallocates it if the container restarts. So Zulip getting the traefik IP address at boot time isn't necessarily sufficient, since it might later become stale. Aside about "172.18.0.0/24"
In case it's helpful, One option, which might remove the need to set up a specific subnet and IP for traefik, is to tell Zulip that the loadbalancer IPs are @tomkv wrote:
Agree with all of the above. Specifically, the request from the outer proxy to the inner nginx, at
Yup, that's absolutely what a naïve implementation would do (and what Zulip used to do!) -- but we explicitly set the This is specifically a change in 7.0, and probably one of the causes of the problems folks are seeing. Have you tried setting I'll work up a documentation and/or changelog change to highlight the need to have the loadbalancer IPs set in 7.0, since this issue is very much showing that we need to improve that. :)
That's absolutely an error in the documentation, and I'll push a fix. Apologies for any confusion that caused!
Thanks -- I've updated to point to the canonical documentation. |
Previously, `X-Forwarded-Proto` did not need to be set, and failure to set `loadbalancer.ips` would merely result in bad IP-address rate-limiting and incorrect access logs; after 0935d38, however, failure to do either of those, if Zulip is deployed with `http_only`, will lead to infinite redirect loops after login. These are accompanied by a misleading error, from Tornado, of: Forbidden (Origin checking failed - https://zulip.example.com does not match any trusted origins.): /json/events This is most common with Docker deployments, where deployments use another docker container, such as nginx or Traefik, to do SSL termination. See zulip/docker-zulip#403. Update the documentation to reinforce that `loadbalancer.ips` also controls trust of `X-Forwarded-Proto`, and that failure to set it will cause the application to not function correctly.
@alexmv wrote:
After some investigation, it turnet out, that exactly this has been the issue:
(and put After this, it works. |
Mmm, I can see this being unclear.
Yup --
I'm glad you got things working. Can you take a look at #405 and zulip/zulip#26011 and see if the changes in those would have helped make this more evident? |
Combine nginx and Django middlware to stop putting misleading warnings about `CSRF_TRUSTED_ORIGINS` when the issue is untrusted proxies. This attempts to, in the error logs, diagnose and suggest next steps to fix common proxy misconfigurations. See also zulip#24599 and zulip/docker-zulip#403.
Combine nginx and Django middlware to stop putting misleading warnings about `CSRF_TRUSTED_ORIGINS` when the issue is untrusted proxies. This attempts to, in the error logs, diagnose and suggest next steps to fix common proxy misconfigurations. See also zulip#24599 and zulip/docker-zulip#403.
Combine nginx and Django middlware to stop putting misleading warnings about `CSRF_TRUSTED_ORIGINS` when the issue is untrusted proxies. This attempts to, in the error logs, diagnose and suggest next steps to fix common proxy misconfigurations. See also zulip#24599 and zulip/docker-zulip#403.
Combine nginx and Django middlware to stop putting misleading warnings about `CSRF_TRUSTED_ORIGINS` when the issue is untrusted proxies. This attempts to, in the error logs, diagnose and suggest next steps to fix common proxy misconfigurations. See also zulip#24599 and zulip/docker-zulip#403.
Previously, `X-Forwarded-Proto` did not need to be set, and failure to set `loadbalancer.ips` would merely result in bad IP-address rate-limiting and incorrect access logs; after 0935d38, however, failure to do either of those, if Zulip is deployed with `http_only`, will lead to infinite redirect loops after login. These are accompanied by a misleading error, from Tornado, of: Forbidden (Origin checking failed - https://zulip.example.com does not match any trusted origins.): /json/events This is most common with Docker deployments, where deployments use another docker container, such as nginx or Traefik, to do SSL termination. See zulip/docker-zulip#403. Update the documentation to reinforce that `loadbalancer.ips` also controls trust of `X-Forwarded-Proto`, and that failure to set it will cause the application to not function correctly.
Combine nginx and Django middlware to stop putting misleading warnings about `CSRF_TRUSTED_ORIGINS` when the issue is untrusted proxies. This attempts to, in the error logs, diagnose and suggest next steps to fix common proxy misconfigurations. See also zulip#24599 and zulip/docker-zulip#403.
Combine nginx and Django middlware to stop putting misleading warnings about `CSRF_TRUSTED_ORIGINS` when the issue is untrusted proxies. This attempts to, in the error logs, diagnose and suggest next steps to fix common proxy misconfigurations. See also zulip#24599 and zulip/docker-zulip#403.
Combine nginx and Django middlware to stop putting misleading warnings about `CSRF_TRUSTED_ORIGINS` when the issue is untrusted proxies. This attempts to, in the error logs, diagnose and suggest next steps to fix common proxy misconfigurations. See also zulip#24599 and zulip/docker-zulip#403.
Combine nginx and Django middlware to stop putting misleading warnings about `CSRF_TRUSTED_ORIGINS` when the issue is untrusted proxies. This attempts to, in the error logs, diagnose and suggest next steps to fix common proxy misconfigurations. See also zulip#24599 and zulip/docker-zulip#403.
Combine nginx and Django middlware to stop putting misleading warnings about `CSRF_TRUSTED_ORIGINS` when the issue is untrusted proxies. This attempts to, in the error logs, diagnose and suggest next steps to fix common proxy misconfigurations. See also #24599 and zulip/docker-zulip#403.
Previously, `X-Forwarded-Proto` did not need to be set, and failure to set `loadbalancer.ips` would merely result in bad IP-address rate-limiting and incorrect access logs; after 0935d38, however, failure to do either of those, if Zulip is deployed with `http_only`, will lead to infinite redirect loops after login. These are accompanied by a misleading error, from Tornado, of: Forbidden (Origin checking failed - https://zulip.example.com does not match any trusted origins.): /json/events This is most common with Docker deployments, where deployments use another docker container, such as nginx or Traefik, to do SSL termination. See zulip/docker-zulip#403. Update the documentation to reinforce that `loadbalancer.ips` also controls trust of `X-Forwarded-Proto`, and that failure to set it will cause the application to not function correctly. (cherry picked from commit d46279c)
Combine nginx and Django middlware to stop putting misleading warnings about `CSRF_TRUSTED_ORIGINS` when the issue is untrusted proxies. This attempts to, in the error logs, diagnose and suggest next steps to fix common proxy misconfigurations. See also zulip#24599 and zulip/docker-zulip#403. (cherry picked from commit 8a77cca)
Just upgraded to Zulip 7.0 and I got "CSRF verification failed. Request aborted." from Django when logging in (or infinite reloads when already logged in)
As it is behind a reverse proxy, DISABLE_HTTPS: is set to True.
As per issue zulip/zulip#24599 "CSRF_TRUSTED_ORIGINS no longer filled by EXTERNAL_HOST", an easy fix to this is to add:
SETTING_CSRF_TRUSTED_ORIGINS: "['https://chat.domain']"
in the docker-compose/zulip/environment
I don't know if it is the best way to fix the issue, and/or it's worth to document it (or document fully how to set it up behind a reverse proxy) or add it in the docker-compose, but for anyone facing the issue, that's a fix!
The text was updated successfully, but these errors were encountered: