Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

50 clowder20 submit file to extractor #51

Closed
wants to merge 70 commits into from

Conversation

tcnichol
Copy link
Contributor

@tcnichol tcnichol commented Aug 15, 2022

These changes will allow a file to be submitted to an extractor and the metadata will post. I have not yet handled cases where new files are uploaded or tags. Easiest way to test is to use the wordcount extractor.

With this branch, add .env file to the pyclowder directory and put in

clowder_version=2.0

Right now I am sending in the Bearer Token from clowder2.0, and then using the Bearer Token in place of the extractor-key or secretKey. I am not sure that this will be a good strategy long term. If an extractor takes a long time to complete, the token may expire, but this seemed like a good enough approach for now.

The branch this works with for clowder2.0 is

https://github.com/clowder-framework/clowder2/tree/register-extractor-submit-file

at different points in the code, the version will be checked
different methods will use different endpoints and will use the bearer token instead of the key
problem - the method in the extractor is not getting the secret key
possible fix - use token for secret key
need to add extractor info
@tcnichol tcnichol requested review from lmarini and max-zilla August 15, 2022 21:37
@tcnichol tcnichol linked an issue Aug 15, 2022 that may be closed by this pull request
@tcnichol tcnichol marked this pull request as draft August 17, 2022 20:22
more to be added later for datasets once that completed in clowderv2
…o make sure future metadata that matches is not reprocessed)
some routes not implemented in v2 clowder, left for later
@tcnichol
Copy link
Contributor Author

tcnichol commented Oct 6, 2022

This clowder2 pull request is also reliant on this one:

clowder-framework/clowder2#128

@lmarini lmarini requested a review from robkooper October 24, 2022 21:24
@max-zilla
Copy link
Contributor

Ran this with Clowder v1 develop and wordcount worked!

@max-zilla max-zilla self-requested a review January 5, 2023 15:16
Copy link
Contributor

@max-zilla max-zilla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested with clowder v1 and v2 both.

@tcnichol
Copy link
Contributor Author

For testing, if you are testing with clowder v2, here is the entry for extractor 'wordcount' you can add to listeners. if you run clowder2 and have wordcount running at the same time, it will submit and post metadata back, which should now be visible on main.

{ "_id": { "$oid": "63b5cd4aeb1180d52266214e" }, "author": "Rob Kooper <[email protected]>", "name": "ncsa.wordcount", "version": "2.0", "description": "WordCount extractor. Counts the number of characters, words and lines in the text file that was uploaded.", "creator": null, "created": { "$date": { "$numberLong": "1672858954451" } }, "modified": { "$date": { "$numberLong": "1672858954451" } }, "properties": { "author": "Rob Kooper <[email protected]>", "process": { "file": [ "text/*", "application/json" ] }, "maturity": "Development", "name": "ncsa.wordcount", "contributors": [], "contexts": [ { "lines": "http://clowder.ncsa.illinois.edu/metadata/ncsa.wordcount#lines", "words": "http://clowder.ncsa.illinois.edu/metadata/ncsa.wordcount#words", "characters": "http://clowder.ncsa.illinois.edu/metadata/ncsa.wordcount#characters" } ], "repository": [ { "id": { "$oid": "63b5cd4aeb1180d52266214d" }, "repository_type": "git", "repository_url": "" } ], "external_services": [], "libraries": [], "bibtex": [], "default_labels": [], "categories": [], "parameters": { "schema": { "X_MIN_START": { "type": "integer", "title": "X_MIN_START" }, "X_MIN_END": { "type": "integer", "title": "X_MIN_END" }, "Y_MIN_START": { "type": "integer", "title": "Y_MIN_START" }, "Y_MIN_END": { "type": "integer", "title": "Y_MIN_END" }, "ZONE": { "type": "string", "title": "ZONE" } } }, "version": "2.0" } }

@tcnichol tcnichol closed this Mar 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

clowder2.0 - submit file to extractor update pyclowder to use v2 api
6 participants