Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store Uploaded Datasets In Local Or Remote File Storage Instead of Database Column #95

Open
klondikemarlen opened this issue Apr 9, 2024 · 0 comments
Labels
⚠️ out of scope Out of current scope, issue was created to keep track of features that will likely be needed.

Comments

@klondikemarlen
Copy link
Member

klondikemarlen commented Apr 9, 2024

Relates to:

Context

Is your feature request related to a problem? Please describe.
Current dataset file upload storage is in the database, and this is limiting from a size perspective.
It also makes it impossible to handle files that are larger than the available memory limit the entire file must be loaded into memory to save/load from the database.

Describe the solution you'd like
Store dataset files URI in the dataset files table, and store the actual file in local storage or in the cloud.
This will make it possible to read a dataset without loading the entire thing into memory, and generally scale much better.

@klondikemarlen klondikemarlen added the ⚠️ out of scope Out of current scope, issue was created to keep track of features that will likely be needed. label Apr 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
⚠️ out of scope Out of current scope, issue was created to keep track of features that will likely be needed.
Projects
None yet
Development

No branches or pull requests

1 participant