Store Uploaded Datasets In Local Or Remote File Storage Instead of Database Column #95
Labels
⚠️ out of scope
Out of current scope, issue was created to keep track of features that will likely be needed.
Relates to:
Context
Is your feature request related to a problem? Please describe.
Current dataset file upload storage is in the database, and this is limiting from a size perspective.
It also makes it impossible to handle files that are larger than the available memory limit the entire file must be loaded into memory to save/load from the database.
Describe the solution you'd like
Store dataset files URI in the dataset files table, and store the actual file in local storage or in the cloud.
This will make it possible to read a dataset without loading the entire thing into memory, and generally scale much better.
The text was updated successfully, but these errors were encountered: