Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A private role with no users breaks /api/roles + /api/roles times out #19654

Open
cat-bro opened this issue Feb 19, 2025 · 5 comments
Open

A private role with no users breaks /api/roles + /api/roles times out #19654

cat-bro opened this issue Feb 19, 2025 · 5 comments
Assignees
@jdavcs
Labels
area/database Galaxy's database or data access layer area/performance kind/bug

Comments

@cat-bro
Copy link
Contributor

cat-bro commented Feb 19, 2025 •
edited
Loading

On release_24.2 following our update, https://usegalaxy.org.au/api/roles leads to 'Internal Server Error'.

Stack trace ends in Did not find exactly one user for private role <galaxy.model.Role(1) at 0x7f36a0e1fa30> - I'll add this as a comment because it's long.

At first we were looking for a role with more than one user_role_association, then Nuwan worked out that it could be a role with no user_role_assocation - we found this and updated its type.

We think this might have solved the problem but it's not clear because the api now times out when loading. Setting ?limit= or ?page= makes no difference
@cat-bro cat-bro changed the title A private role with no users breaks /api/roles A private role with no users breaks /api/roles + /api/roles times out Feb 19, 2025
@cat-bro
Copy link
Contributor Author

cat-bro commented Feb 19, 2025 

Feb 19 03:40:55 galaxy galaxyctl[2622924]: [2025-02-19 03:40:55 +0000] [2622924] [ERROR] Exception in ASGI application
Feb 19 03:40:55 galaxy galaxyctl[2622924]: Traceback (most recent call last):
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     result = await app(  # type: ignore[func-returns-value]
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return await self.app(scope, receive, send)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     await super().__call__(scope, receive, send)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 409, in _sentry_patched_asgi_app
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return await middleware(scope, receive, send)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py", line 161, in _run_asgi3
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return await self._run_app(scope, receive, send, asgi_version=3)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py", line 262, in _run_app
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     raise exc from None
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py", line 257, in _run_app
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return await self.app(
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/applications.py", line 113, in __call__
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     await self.middleware_stack(scope, receive, send)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 200, in _create_span_call
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return await old_call(app, scope, new_receive, new_send, **kwargs)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in __call__
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     raise exc
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in __call__
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     await self.app(scope, receive, _send)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 200, in _create_span_call
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return await old_call(app, scope, new_receive, new_send, **kwargs)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette_context/middleware/raw_middleware.py", line 92, in __call__
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     await self.app(scope, receive, send_wrapper)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 200, in _create_span_call
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return await old_call(app, scope, new_receive, new_send, **kwargs)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 185, in __call__
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     with collapse_excgroups():
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     self.gen.throw(typ, value, traceback)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     raise exc
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 187, in __call__
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     response = await self.dispatch_func(request, call_next)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/galaxy-app/lib/galaxy/webapps/galaxy/fast_app.py", line 108, in add_x_frame_options
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     response = await call_next(request)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 163, in call_next
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     raise app_exc
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 149, in coro
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     await self.app(scope, receive_or_disconnect, send_no_error)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 298, in _sentry_exceptionmiddleware_call
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     await old_call(self, scope, receive, send)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 200, in _create_span_call
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return await old_call(app, scope, new_receive, new_send, **kwargs)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     raise exc
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     await app(scope, receive, sender)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/routing.py", line 715, in __call__
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     await self.middleware_stack(scope, receive, send)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/routing.py", line 735, in app
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     await route.handle(scope, receive, send)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     await self.app(scope, receive, send)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     raise exc
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     await app(scope, receive, sender)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/routing.py", line 73, in app
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     response = await f(request)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/sentry_sdk/integrations/fastapi.py", line 143, in _sentry_app
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return await old_app(*args, **kwargs)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/fastapi/routing.py", line 301, in app
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     raw_response = await run_endpoint_function(
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return await run_in_threadpool(dependant.call, **values)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return await anyio.to_thread.run_sync(func, *args)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return await get_async_backend().run_sync_in_worker_thread(
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2364, in run_sync_in_worker_thread
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return await future
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 864, in run
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     result = context.run(func, *args)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/sentry_sdk/integrations/fastapi.py", line 99, in _sentry_call
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return old_call(*args, **kwargs)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/galaxy-app/lib/galaxy/webapps/galaxy/api/roles.py", line 37, in index
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return self.service.get_index(trans=trans)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/galaxy-app/lib/galaxy/webapps/galaxy/services/roles.py", line 36, in get_index
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return RoleListResponse(root=[role_to_model(r) for r in roles])
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/galaxy-app/lib/galaxy/webapps/galaxy/services/roles.py", line 36, in <listcomp>
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return RoleListResponse(root=[role_to_model(r) for r in roles])
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/galaxy-app/lib/galaxy/webapps/galaxy/services/roles.py", line 18, in role_to_model
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     item = role.to_dict(view="element")
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/galaxy-app/lib/galaxy/util/dictifiable.py", line 92, in to_dict
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return self._dictify_view_keys(view=view, value_mapper=value_mapper)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/galaxy-app/lib/galaxy/util/dictifiable.py", line 70, in _dictify_view_keys
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     item = self.__getattribute__(key)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/venv/lib/python3.10/site-packages/sqlalchemy/ext/hybrid.py", line 1120, in __get__
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     return self.fget(instance)
Feb 19 03:40:55 galaxy galaxyctl[2622924]:   File "/mnt/galaxy/galaxy-app/lib/galaxy/model/__init__.py", line 3850, in name
Feb 19 03:40:55 galaxy galaxyctl[2622924]:     assert len(user_assocs) == 1, f"Did not find exactly one user for private role {self}"
Feb 19 03:40:55 galaxy galaxyctl[2622924]: AssertionError: Did not find exactly one user for private role <galaxy.model.Role(1) at 0x7f36a0e1fa30>

@jdavcs jdavcs self-assigned this Feb 19, 2025
@jdavcs
Copy link
Member

jdavcs commented Feb 19, 2025

At first we were looking for a role with more than one user_role_association, then Nuwan worked out that it could be a role with no user_role_assocation - we found this and updated its type.

@cat-bro: what did you update? The type of the role that did not have an association? (I can't reproduce the api timeout yet)

@cat-bro
Copy link
Contributor Author

cat-bro commented Feb 20, 2025

Thanks @jdavcs. For that one private role with no user_role_associations, I changed deleted to 't', then type to 'sharing'.

Galaxy Australia has about 40K users. The timeout with /api/roles does not occur on Galaxy Main for an admin user?

How can I check how long the query is taking when that page loads?

@jdavcs
Copy link
Member

jdavcs commented Feb 20, 2025

Reproduced the timeout. I know what's causing it and I know how to fix it. It's a 24.2 performance bug.

@jdavcs
Copy link
Member

jdavcs commented Feb 20, 2025

It's a 2n+1 query problem. It stems from the change we made in 24.2 decoupling user emails and private role names. The unintended consequence (those are always unintended!) is that for 40K users you'll get around 80K select queries hitting the database (the exact number depends on what other roles you have and which user records are cached by sqlalchemy... but it's terrible whatever those are). I'm working on a fix.

@jdavcs jdavcs added kind/bug area/performance area/database Galaxy's database or data access layer labels Feb 20, 2025
@jdavcs jdavcs mentioned this issue Feb 21, 2025
Draft
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jdavcs jdavcs

Labels
area/database Galaxy's database or data access layer area/performance kind/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jdavcs @cat-bro