Move user data export to background tasks

[amCap1712]

User data exports take a long time and get disrupted because the request times out during the process. To fix the issues, do the exports to an archive in the background and allow the users to download the archive later. Archives are auto-deleted after a fixed period.

TODO:

1. Create docker volume to store exports and update consul config in production.
2. Discuss with @zas where to store the export and possibility of using gateways for a speed up.
3. Discuss with @MonkeyDo to add a screen for users to view their completed export and download it, also update the existing export frontend.
4. Add tests.

[pep8speaks]

Hello @amCap1712! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file listenbrainz/background/export.py:

Line 198:19: E222 multiple spaces after operator
Line 207:32: W291 trailing whitespace
Line 257:1: E302 expected 2 blank lines, found 1

• In the file listenbrainz/webserver/views/export.py:

Line 37:131: E501 line too long (148 > 130 characters)

Comment last updated at 2024-09-17 09:43:40 UTC

[MonkeyDo]

Let's chat about the UI for ongoing tasks.
Happy to see this PR!

[MonkeyDo]

Initial suggestions for the UI:

I reckon we want to show any ongoing export (along with progress, download links eventually) on the same page it was initially requested, at the top of the page.

Maybe something like this?

@Aerozol might have some valuable input and suggestions?

[Aerozol]

I don't see any issue with that UI. Additional thoughts:

• If users can leave and come back later we could include the date + time of when the export was requested (or the date/time of the last listen in the export could be more useful)
• If requesting an export is about to overwrite a current one we might want to indicate that (or just a small note like "will reset any current exports").

[amCap1712]

@MonkeyDo I have endpoints to retrieve existing and in progress exports, and also the current progress of a given export. I think we should also add a table of available exports to the page.