Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow Content-Type to be specified and saved as S3 metadata #207

Merged
merged 16 commits into from
Mar 19, 2021

Conversation

dchiquito
Copy link
Contributor

@dchiquito dchiquito commented Feb 22, 2021

No description provided.

# Test default Content-Disposition when none is specified
assert actual_content_disposition == content_disposition
else:
assert actual_content_disposition == 'attachment; filename="test.txt"'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test case is getting very complex, which could make it expensive to maintain. Is there a sane way to break it up a bit?

( @brianhelba may have something to say about this )

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copying the test for each use case would just double the complexity, so I've been avoiding it.

The only solution I can think of would be a test helper that does most of the upload process on it's own, which I consider something of an antipattern. I'm open to suggestions though, this is a very bloated test.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The body itself is big and branchy, but also just the cardinality of the cartesian product of all these parametrize decorators is getting big. That's fine if we really want to test that whole space, but if not, it'd be good to identify the most important cases and maybe collapse the decorators some.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, I think we should rework this whole test. I'm ok with merging the PR with this, as long as we refactor things at some point.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not 100% sure but it looks like maybe @dchiquito already went through and removed some of the complexity here. It looks less scary to me now, at least in terms of this changeset.

resp = self._client.create_multipart_upload(
Bucket=self._bucket_name,
Key=object_key,
**boto3_kwargs, # type: ignore
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the MyPy error here? If it can't be fixed, could you use a more specific ignore?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have an idea for how to fix this. I'll try locally.

object_key: str,
content_type: Union[str, None] = None,
) -> str:
boto3_kwargs = {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we know the file name at this point, can we set ContentDisposition too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could, but AWS already provides a sensible default value when it is not set. I don't think at this point we would be able to make a better guess than that.

object_key: str,
content_type: Union[str, None] = None,
) -> str:
metadata = {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we know the file name at this point, can we set Content-Disposition too?

For some reason the default Content-Type for objects uploaded during CI
testing is `binary/octet-stream`, instead of the standard
`application/octet-stream`.

It doesn't make sense for the upload test to be verifying the default
Content-Type returned when no Content-Type is specified, as that is an
implementation detail of the object store.
Copy link
Contributor

@brianhelba brianhelba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question, otherwise this LGTM!

@brianhelba brianhelba changed the title Content headers Allow Content-Type to be specified and sent to S3 Mar 19, 2021
@brianhelba brianhelba changed the title Allow Content-Type to be specified and sent to S3 Allow Content-Type to be specified and saved as S3 metadata Mar 19, 2021
@brianhelba brianhelba merged commit c72d6ed into master Mar 19, 2021
@brianhelba brianhelba deleted the content-headers branch March 19, 2021 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants