step-ca breaks after some time #598
Replies: 1 comment 9 replies
-
|
Hi @MCWertGaming hard to say if this is a step-ca issue or some interaction between step-ca and caddy. It certainly seems like a step-ca issue in how we are handling requests for domains that don't exist. In terms of "production" environment with lots of requests, generally speaking a single step-ca instance can handle a lot of ACME traffic and it doesn't require a lot of CPU to issue certificates. I'd suggest checking out our Production Considerations documentation. |
Beta Was this translation helpful? Give feedback.
-
|
The container Volume had 8 gigabytes in storage (system + docker + step-ca are only 1G so it's more than enough.) I have logged a bit now and my caddy is flodding my ca with certificate requests (8 per second) and I can watch how the storage of my lxc increases in size. Caddy is creating that much requests because the ca can't validate it due to a wrong dns name. I think that it would be a good thing to simply limit the requests for a domain, like only 5 open requests total and dropping old ones after maybe 5 minutes or something like that. Here is a screenshot: Here is a whole line: (I have removed IPs, tokens and domains) I'll Create an Issue on the caddy github repository as well as it shouldn't try to order 8 times a second as long as it fails... Tell me if you need further informations and thank you really much for helping me out with this issue! |
Beta Was this translation helpful? Give feedback.
-
|
Small addition: My step-ca machine started with a total ram usage of 1.05 gigabytes, then it increased to 1.10 gigabytes in roughly about 1.5h. Now (another ~1.5h later) it's back at 1.07 gigabytes. Interesting, I'll keep an eye on that the next days! |
Beta Was this translation helpful? Give feedback.
-
|
See also caddyserver/caddy#4186 for details of the Caddy side of things (potential issue with Caddy's ACME client). |
Beta Was this translation helpful? Give feedback.
-
|
Just opened #601 to look further into this on the step-ca side. Thanks Damon. |
Beta Was this translation helpful? Give feedback.
-
|
Here is a demonstration of the caddy side caddyserver/caddy#4186 (comment). Maybe that helps in any way. |
Beta Was this translation helpful? Give feedback.
-
Hello,
I have step-ca running in a docker container inside of an lxc and faced the problem that one of my reverse proxies (caddy) tried to order certificates for an internal domain which is not set up in my DNS server. The result was that step-ca created 100% cpu usage in my lxc and was not responding anymore (the health check was still passing, but certificate orders and renews were not possible anymore.) As I restarted the lxc, step-ca was functioning normally again. Some days later the same problem (with an different caddy that started ordering for one or two days). This time I restarted the step-ca lxc again but the docker container was refusing to start because it returned that there is a problem with the badger database:
value log truncate required to run db. this might result in data lossThe database was over 200mb in size and as I have found nothing via a google search, I have simply deleted the database to let step-ca re-create it. Apparently step-ca started and I could order certificates for my hypervisor, but as I started my caddy reverse proxy, step-ca complained that it had read/write errors in the badger database file. I restarted it and not even starts since then as it immediately shows an error that it can't write into the badger database...
My question is why is step-ca / badger broken just because of many certificate requests? And how do I prevent those errors? Or are there other recommendations about hosting step-ca in a "production environment"? (in quotes because it's just for my private and local infrastructure)
Beta Was this translation helpful? Give feedback.
All reactions