Flask UI broken , probably after load peaks #100
Comments
|
When you say the UI, do you mean when you visit the dagobah page in a web browser, it doesnt load? Or that the page loads, but the page doesn't do anything? |
|
Yes , the first one. But I got no 404 or smth. else. When it occurs next time I'll make screenshot from the page and the process. |
|
It my be useful if you can open the developer tools in whatever browser you have (I know chrome/firefox/safari have similar options) and look at the network tab. That way, when the page fails to load, you can see what network call is failing |
|
Yes I'll do and try to force getting this event, because sometime it's ok for weeks. One idea is , it has smth. to do with the status job reload (open browser) during a high load on the server? |
|
Yesterday I had again this issue. I used the network tab in chrome and problem is that flask don't able to response , so no request information. But it's not like the "webserver" is offline. |
|
The proper solution here is probably to serve the app through a legit webserver (probably gunicorn or something) rather than Flask's built-in dev server. The Flask request thread must be dying for some reason and never getting restarted. |
|
Good point. Perhaps with supervisord incl. it is possible getting more log information... |
|
I am having the same issue and i am going to try running it with gunicorn and see. Thanks! |
|
Just make sure you only run 1 process if you run it behind something like gunicorn (which supports multiple app processes). Otherwise you'll also spin up multiple scheduler threads, and you don't want that. |
|
I had the same issue and I run it behind gunicorn . But it did't work. It's ok for days , but today ,when I added a job ,dagobah_jobs didn't get a an update for next_run. It did't happen everytime , when i add a job . |
|
@zhenlongbai Are you able to retrieve the logs from that point? We've added a bunch of logging since this issue was originally reported. Additionally, since you're running into so many issues, it would probably be helpful to set your logging level to |
|
Ok , I have used Dogbah on my work,and it run very well for days .The logs had 89350 lines and I had change some code to make it works well for my job. for example ,utc time and email . Thanks for you help! |
|
today I had again this issue , when I add a job . When I click "start job from begin" ,it work once and don't get a an update for next_run automatic。 my start script : nohup gunicorn -b 0.0.0.0:9876 -w 1 dagobah_app:app & my log : |
|
I can also find the command ： |
|
This is my DEBUG log. I think ' DEBUG:paramiko.transport:EOF in transport thread ' is the key info. When the thread isn't EOF , dagobah_jobs don't get a an update. |
|
i will try to use the supervisord to see if it will broken again . update 2016-12-30 my solution is use docker , and use cron to restart it every hour , then currently it works well ,but should find the deep reason why the ui broken. |
Hi,
sometimes (actually more often) I can't reach the UI anymore.
My suspicion is a peak in load on the server which broke flask UI. In the log I found only
the last 200's.
INFO:werkzeug:... - - [13/Jun/2014 08:37:17] "GET /api/job?job_name=DMProcessing HTTP/1.1" 200 -
INFO:werkzeug:... - - [13/Jun/2014 08:37:19] "GET /api/job?job_name=DMProcessing HTTP/1.1" 200 -
INFO:werkzeug:... - - [13/Jun/2014 08:37:20] "GET /api/job?job_name=DMProcessing HTTP/1.1" 200 -
INFO:werkzeug:... - - [13/Jun/2014 08:37:22] "GET /api/job?job_name=DMProcessing HTTP/1.1" 200 -
INFO:werkzeug:... - - [13/Jun/2014 08:37:23] "GET /api/job?job_name=DMProcessing HTTP/1.1" 200 -
I use mongodb backend and dagobah collections are in a separate db.
Many thanks for a hint
Christian
The text was updated successfully, but these errors were encountered: