Getting SignatureDoesNotMatch error when using Cloudflare R2

[weibeu]

Upon upgrading to version aiobotocore==2.19.0, I have started getting the following strange error. I am using Cloudflare S3 compatible API. The really strange fact is that it doesn't happens everytime. It occurs say 7 out of 10 times.

(aiobotocore/client.py in _make_api_call at line 412)
ClientError - An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.

What Fixes?
Downgrading and locking the dependecies to following works normally:
aiobotocore==2.11.2
botocore==1.34.34

Checklist

• I have reproduced in environment where pip check passes without errors
• I have provided pip freeze results
• I have provided sample code or detailed way to reproduce
• I have tried the same code in botocore to ensure this is an aiobotocore specific issue
• I have tried similar code in aiohttp to ensure this is is an aiobotocore specific issue
• I have checked the latest and older versions of aiobotocore/aiohttp/python to see if this is a regression / injection

pip freeze results
aio-pika==9.4.0
aiobotocore==2.19.0
aiodns==3.2.0
aiohappyeyeballs==2.4.4
aiohttp==3.11.12
aioitertools==0.11.0
aiormq==6.8.0
aiosignal==1.3.1
aiosqlite==0.20.0
alembic==1.14.1
alembic-postgresql-enum==1.6.1
annotated-types==0.6.0
anyio==4.8.0
apprise==1.7.2
asgi-lifespan==2.1.0
asgiref==3.8.1
async-timeout==4.0.3
asyncpg==0.30.0
attrs==23.2.0
bcrypt==4.2.1
bidict==0.23.1
botocore==1.36.3
cachetools==5.5.1
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
cloudpickle==3.0.0
colorama==0.4.6
contourpy==1.2.0
coolname==2.2.0
cron-validator==1.0.8
croniter==2.0.1
cryptography==42.0.4
cycler==0.12.1
dateparser==1.2.1
Deprecated==1.2.15
dnspython==2.6.1
docker==6.1.3
ecdsa==0.18.0
ecs-logging==2.2.0
elastic-transport==8.17.0
elasticsearch==8.17.1
elasticsearch-dsl==8.17.1
email-validator==2.1.0.post1
exceptiongroup==1.2.2
executing==2.2.0
fast-depends==2.4.2
fastapi==0.115.8
fastapi-cli==0.0.7
faststream==0.5.34
fonttools==4.49.0
frozenlist==1.4.1
fsspec==2024.2.0
google-auth==2.28.1
googleapis-common-protos==1.66.0
graphviz==0.20.1
greenback==1.2.1
greenlet==3.0.3
griffe==1.5.6
h11==0.14.0
h2==4.1.0
hpack==4.0.0
httpcore==1.0.7
httptools==0.6.4
httpx==0.28.1
humanize==4.11.0
Hypercorn==0.17.3
hyperframe==6.0.1
idna==3.6
importlib_metadata==8.5.0
inflection==0.5.1
iniconfig==2.0.0
itsdangerous==2.2.0
jc==1.25.3
Jinja2==3.1.5
jinja2-humanize-extension==0.4.0
jmespath==1.0.1
jsonpatch==1.33
jsonpointer==2.4
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
kiwisolver==1.4.5
kubernetes==29.0.0
logfire==3.5.3
Mako==1.3.2
Markdown==3.5.2
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.8.3
mdurl==0.1.2
multidict==6.0.5
mypy==1.15.0
mypy-extensions==1.0.0
networkx==3.4.2
numpy==1.26.4
oauthlib==3.2.2
opentelemetry-api==1.29.0
opentelemetry-exporter-otlp-proto-common==1.29.0
opentelemetry-exporter-otlp-proto-http==1.29.0
opentelemetry-instrumentation==0.50b0
opentelemetry-instrumentation-aio-pika==0.50b0
opentelemetry-instrumentation-aiohttp-client==0.50b0
opentelemetry-instrumentation-asgi==0.50b0
opentelemetry-instrumentation-asyncpg==0.50b0
opentelemetry-instrumentation-dbapi==0.50b0
opentelemetry-instrumentation-elasticsearch==0.50b0
opentelemetry-instrumentation-fastapi==0.50b0
opentelemetry-instrumentation-httpx==0.50b0
opentelemetry-instrumentation-jinja2==0.50b0
opentelemetry-instrumentation-psycopg2==0.50b0
opentelemetry-instrumentation-redis==0.50b0
opentelemetry-instrumentation-requests==0.50b0
opentelemetry-instrumentation-sqlalchemy==0.50b0
opentelemetry-instrumentation-sqlite3==0.50b0
opentelemetry-instrumentation-starlette==0.50b0
opentelemetry-instrumentation-system-metrics==0.50b0
opentelemetry-instrumentation-urllib==0.50b0
opentelemetry-instrumentation-urllib3==0.50b0
opentelemetry-proto==1.29.0
opentelemetry-sdk==1.29.0
opentelemetry-semantic-conventions==0.50b0
opentelemetry-util-http==0.50b0
orjson==3.9.14
outcome==1.3.0.post0
packaging==23.2
pamqp==3.3.0
pandas==2.2.0
pathspec==0.12.1
pendulum==3.0.0
pillow==10.2.0
pluggy==1.5.0
prefect==3.1.15
priority==2.0.0
prometheus_client==0.21.1
propcache==0.2.1
protobuf==5.29.2
psutil==6.1.1
psycopg2-binary==2.9.9
pyasn1==0.5.1
pyasn1-modules==0.3.0
pycares==4.4.0
pycparser==2.21
pydantic==2.10.6
pydantic-extra-types==2.10.2
pydantic-settings==2.7.1
pydantic_core==2.27.2
Pygments==2.17.2
pygraphviz==1.12
pyparsing==3.1.1
pytest==8.3.4
pytest-asyncio==0.25.3
python-baseconv==1.2.2
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
python-engineio==4.11.2
python-jose==3.3.0
python-multipart==0.0.20
python-slugify==8.0.4
python-socketio==5.11.2
python-socks==2.7.1
python-stdnum==1.20
pytz==2024.2
pytzdata==2020.1
PyYAML==6.0.1
readchar==4.0.5
redis==5.2.1
referencing==0.33.0
regex==2023.12.25
requests==2.32.3
requests-oauthlib==1.3.1
rfc3339-validator==0.1.4
rich==13.9.4
rich-toolkit==0.13.2
rpds-py==0.18.0
rsa==4.9
ruamel.yaml==0.18.6
ruamel.yaml.clib==0.2.8
ruff==0.8.1
scipy==1.12.0
sentry-sdk==2.20.0
shellingham==1.5.4
simple-websocket==1.0.0
six==1.16.0
sniffio==1.3.0
SQLAlchemy==2.0.38
starlette==0.45.3
structlog==25.1.0
svix-ksuid==0.6.2
temporalio==1.9.0
text-unidecode==1.3
time-machine==2.16.0
toml==0.10.2
typer==0.15.1
types-protobuf==5.29.1.20241207
typing_extensions==4.12.2
tzdata==2024.1
tzlocal==5.2
ujson==5.9.0
urllib3==2.0.7
uvicorn==0.34.0
uvloop==0.19.0
watchfiles==0.21.0
websocket-client==1.7.0
websockets==12.0
wrapt==1.16.0
wsproto==1.2.0
xmltodict==0.14.0
yarl==1.18.3
zipp==3.21.0

Environment:

• Python Version: 3.11
• OS name and version: Ubuntu 22.04 LTS

[jakob-keller]

Could this be another duplicate of boto/boto3#4398?

[taeven]

I can confirm this is still an existing issue. I had a really weird issue where I was able to upload the file occasionally, but only when I did any read operations before the write. But it failed around 90% of the time when just doing the write operations.
To avoid the issues with signature expiry, I even used semaphores with the same size as pool size, but it still didn't work.
At last, I had to update my codebase to use boto3 "^1.36.11" (which works phenomenally).

[taeven]

Maybe I'll give it a try another time with the dependencies mentioned by OP:

aiobotocore==2.11.2
botocore==1.34.34

[thehesiod]

can you verify if this works keeping botocore the same and using the botocore instead of the aiobotocore client? Note that if you're using multiple tasks with the same client in aiobotocore you'll need to replace it with multiple threads with botocore. That will help narrow down if it's a aiobotocore or botocore issue

[aek]

I have got this error Today February 19 after being working before and seems that the problem appears with boto3 and botocore versions 1.36.0 and later. In my case I was using 1.36.1 and after updating to 1.36.24 seems that my issue was fixed

Here is the announcement of the change that hit me with this issue
boto/boto3#4392

[jakob-keller]

In my case I was using 1.36.1 and after updating to 1.36.24 seems that my issue was fixed

We just shipped aiobotocore 2.20.0 which supports botocore up until 1.36.23. I would assume that this should address your issue, given that botocore 1.36.24 does not include any related changes on top of what was already included before. Please let us know, if that's the case.

[aek]

than

We just shipped aiobotocore 2.20.0 which supports botocore up until 1.36.23

Thank you, indeed yesterday I noticed that there was some dependencies in aiobotocore with botocore 1.36.1 so I kind of was forcing it to have botocore 1.36.24 installed

Since I was using aioboto3 instead of directly using aiobotocore I need to adjust my scripts to use directly aiobotocore and change the way I handle the filelikeobj because my scenarios involves backup an restoration of huge zip files that are downloaded from s3, unziped on the fly and upload it's content to another s3 location
I will update my code share if it works but I think it will