Add support for trailing checksums to PutObject when object lock is enabled

[jpveooys]

Hi,

As known, botocore 1.36.0 contains this change:

• feature:s3: S3 client behavior is updated to always calculate a CRC32 checksum by default for operations that support it (such as PutObject or UploadPart), or require it (such as DeleteObjects). Checksum behavior can be configured using when_supported and when_required options - in code using the request_checksum_calculation parameter for botocore.config.Config, in the shared AWS config file using request_checksum_calculation, or as an env variable using AWS_REQUEST_CHECKSUM_CALCULATION. Note: Botocore will no longer automatically compute and populate the Content-MD5 header.

At least by default, botocore is no longer setting the Content-MD5 header. For the PutObject request I'm looking at, it is setting these (among other headers):

X-Amz-Trailer: x-amz-checksum-crc32
x-amz-sdk-checksum-algorithm: CRC32

A checksum is required for PutObject when object lock is enabled. However, currently moto only allows Content-MD5 for these:

moto/moto/s3/responses.py

Lines 1678 to 1686 in 0283895

	if lock_mode or lock_until or legal_hold == "ON":
	if not self.headers.get("Content-Md5"):
	raise InvalidContentMD5
	if not lock_enabled:
	raise LockNotEnabled
	elif lock_enabled and bucket.has_default_lock:
	if not self.headers.get("Content-Md5"):
	raise InvalidContentMD5

#8495 made the relevant tests explicitly set Content-MD5, however that doesn't reflect the out-of-the-box behaviour of botocore>=1.36.0.

The S3 documentation for Object Lock says Content-MD5 or x-amz-sdk-checksum-algorithm is required:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-managing.html#object-lock-put-object

Some more info here too:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html

I appreciate the desire to be cautious around these changes. However, there should be a way to make moto compatible with the new default behaviour of botocore in this scenario.

Edit:

Some additional links I came across:

• https://aws.amazon.com/blogs/aws/introducing-default-data-integrity-protections-for-new-objects-in-amazon-s3/
• https://docs.aws.amazon.com/sdkref/latest/guide/feature-dataintegrity.html

[bpandola]

I appreciate the desire to be cautious around these changes. However, there should be a way to make moto compatible with the new default behaviour of botocore in this scenario.

Agreed. As noted in #8495, I was pretty sure this botocore change was going to necessitate further moto changes, I just didn't know what I didn't know. Thank you for the detailed issue report. A PR to fix this would certainly be welcome, if this is something you have the ability to tackle.