ImportError: cannot import name 'BatchServiceClient' from 'azure.batch'

[ShermanCAA]

• azure-batch
• 14.2
• Linux
• 3.10

Describe the bug
Unable to import BatchServiceClient from azure.batch, despite being on a stable version of the package

newest version of azure-cli is depending on a pre-release version of azure-batch, which causes discrepancies when both packages are present

Full stack trace

ImportError while importing test module '/home/vsts/work/1/s/Pipelines/RestartPools/tests/test_restart_pools.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
Pipelines/RestartPools/tests/test_restart_pools.py:11: in <module>
    import restart_pools as p
Pipelines/RestartPools/activities/python/restart_pools.py:35: in <module>
    import common.batch.pool as p
common/common/batch/pool.py:6: in <module>
    from azure.batch import BatchServiceClient
E   ImportError: cannot import name 'BatchServiceClient' from 'azure.batch' (/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/site-packages/azure/batch/__init__.py)

To Reproduce
Steps to reproduce the behavior:

1. Install azure-batch v14.2
2. Install azure-cli v2.69.0
3. from azure.batch import BatchServiceClient

Expected behavior
BatchServiceClient should be imported and dependency resolved

Screenshots

Additional context

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @cRui861 @dpwatrous @jingjlii @JJJessieWang @wanghoppe @wiboris.

[danchengcaa]

Any updates on this? Thanks!

[kristapratico]

@ShermanCAA @danchengcaa I'm not able to reproduce this. I created a fresh virtual environment, installed azure-batch==14.2.0, and was able to import the client:

krpratic@KRPRATIC:~/azure-sdk-for-python$ pyenv virtualenv 3.10 azurebatch
krpratic@KRPRATIC:~/azure-sdk-for-python$ pyenv activate azurebatch
(azurebatch) krpratic@KRPRATIC:~/azure-sdk-for-python$ pip install azure-batch==14.2.0 --quiet
(azurebatch) krpratic@KRPRATIC:~/azure-sdk-for-python$ python
Python 3.10.0 (default, Feb 12 2025, 10:07:03) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from azure.batch import BatchServiceClient
>>> 

Can you describe your environment in more detail?

[github-actions]

Hi @ShermanCAA. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[danchengcaa]

@kristapratico Thank you! Here is our vm settings in azure dev-ops logs. Also we are installing python 3.10.16 on the vm. Let us know if you need anything else.

Found tool in cache: Python 3.10.16 x64
Prepending PATH environment variable with directory: /opt/hostedtoolcache/Python/3.10.16/x64
Prepending PATH environment variable with directory: /opt/hostedtoolcache/Python/3.10.16/x64/bin
Finishing: Install Python 3.10

2025-02-12T00:02:28.8963944Z ##[section]Starting: Initialize job
2025-02-12T00:02:28.8966571Z Agent name: 'Azure Pipelines 3'
2025-02-12T00:02:28.8967105Z Agent machine name: 'fv-az417-328'
2025-02-12T00:02:28.8967347Z Current agent version: '4.251.0'
2025-02-12T00:02:28.8994335Z ##[group]Operating System
2025-02-12T00:02:28.8994628Z Ubuntu
2025-02-12T00:02:28.8994806Z 22.04.5
2025-02-12T00:02:28.8994979Z LTS
2025-02-12T00:02:28.8995152Z ##[endgroup]
2025-02-12T00:02:28.8995363Z ##[group]Runner Image
2025-02-12T00:02:28.8995589Z Image: ubuntu-22.04
2025-02-12T00:02:28.8995813Z Version: 20250202.1.0
2025-02-12T00:02:28.8996120Z Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20250202.1/images/ubuntu/Ubuntu2204-Readme.md
2025-02-12T00:02:28.8996518Z Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20250202.1
2025-02-12T00:02:28.8996784Z ##[endgroup]
2025-02-12T00:02:28.8996999Z ##[group]Runner Image Provisioner
2025-02-12T00:02:28.8997346Z 2.0.422.1
2025-02-12T00:02:28.8997538Z ##[endgroup]
2025-02-12T00:02:28.9000622Z Current image version: '20250202.1.0'
2025-02-12T00:02:28.9990765Z Agent running as: 'vsts'
2025-02-12T00:02:29.0032115Z Prepare build directory.
2025-02-12T00:02:29.0268415Z Set build variables.
2025-02-12T00:02:29.0285531Z Download all required tasks.
2025-02-12T00:02:29.0364053Z Downloading task: UsePythonVersion (0.248.1)
2025-02-12T00:02:29.5489174Z Downloading task: CmdLine (2.250.1)
2025-02-12T00:02:29.8378663Z Checking job knob settings.
2025-02-12T00:02:29.8381956Z Knob: DockerActionRetries = true Source: $(VSTSAGENT_DOCKER_ACTION_RETRIES)
2025-02-12T00:02:29.8382545Z Knob: AgentToolsDirectory = /opt/hostedtoolcache Source: ${AGENT_TOOLSDIRECTORY}
2025-02-12T00:02:29.8384681Z Knob: UseGitLongPaths = true Source: $(USE_GIT_LONG_PATHS)
2025-02-12T00:02:29.8385774Z Knob: AgentPerflog = /home/vsts/perflog Source: ${VSTS_AGENT_PERFLOG}
2025-02-12T00:02:29.8387542Z Knob: EnableIssueSourceValidation = true Source: $(ENABLE_ISSUE_SOURCE_VALIDATION)
2025-02-12T00:02:29.8388594Z Knob: AgentEnablePipelineArtifactLargeChunkSize = true Source: $(AGENT_ENABLE_PIPELINEARTIFACT_LARGE_CHUNK_SIZE)
2025-02-12T00:02:29.8391465Z Knob: ContinueAfterCancelProcessTreeKillAttempt = true Source: $(VSTSAGENT_CONTINUE_AFTER_CANCEL_PROCESSTREEKILL_ATTEMPT)
2025-02-12T00:02:29.8392387Z Knob: ProcessHandlerSecureArguments = false Source: $(AZP_75787_ENABLE_NEW_LOGIC)
2025-02-12T00:02:29.8393056Z Knob: ProcessHandlerSecureArguments = false Source: $(AZP_75787_ENABLE_NEW_LOGIC_LOG)
2025-02-12T00:02:29.8393696Z Knob: ProcessHandlerTelemetry = true Source: $(AZP_75787_ENABLE_COLLECT)
2025-02-12T00:02:29.8394305Z Knob: UseNewNodeHandlerTelemetry = True Source: $(DistributedTask.Agent.USENEWNODEHANDLERTELEMETRY)
2025-02-12T00:02:29.8394955Z Knob: ProcessHandlerEnableNewLogic = true Source: $(AZP_75787_ENABLE_NEW_PH_LOGIC)
2025-02-12T00:02:29.8395872Z Knob: EnableResourceMonitorDebugOutput = true Source: $(AZP_ENABLE_RESOURCE_MONITOR_DEBUG_OUTPUT)
2025-02-12T00:02:29.8396645Z Knob: EnableResourceUtilizationWarnings = true Source: $(AZP_ENABLE_RESOURCE_UTILIZATION_WARNINGS)
2025-02-12T00:02:29.8397547Z Knob: IgnoreVSTSTaskLib = true Source: $(AZP_AGENT_IGNORE_VSTSTASKLIB)
2025-02-12T00:02:29.8398107Z Knob: FailJobWhenAgentDies = true Source: $(FAIL_JOB_WHEN_AGENT_DIES)
2025-02-12T00:02:29.8398897Z Knob: CheckForTaskDeprecation = true Source: $(AZP_AGENT_CHECK_FOR_TASK_DEPRECATION)
2025-02-12T00:02:29.8399772Z Knob: CheckIfTaskNodeRunnerIsDeprecated246 = True Source: $(DistributedTask.Agent.CheckIfTaskNodeRunnerIsDeprecated246)
2025-02-12T00:02:29.8400641Z Knob: UseNode20ToStartContainer = True Source: $(DistributedTask.Agent.UseNode20ToStartContainer)
2025-02-12T00:02:29.8401433Z Knob: LogTaskNameInUserAgent = true Source: $(AZP_AGENT_LOG_TASKNAME_IN_USERAGENT)
2025-02-12T00:02:29.8402052Z Knob: UseFetchFilterInCheckoutTask = true Source: $(AGENT_USE_FETCH_FILTER_IN_CHECKOUT_TASK)
2025-02-12T00:02:29.8403093Z Knob: Rosetta2Warning = true Source: $(ROSETTA2_WARNING)
2025-02-12T00:02:29.8404660Z Knob: AddForceCredentialsToGitCheckout = True Source: $(DistributedTask.Agent.AddForceCredentialsToGitCheckout)
2025-02-12T00:02:29.8405467Z Finished checking job knob settings.
2025-02-12T00:02:29.9013500Z Plugin: 'Test Result Parser plugin' is running in background.
2025-02-12T00:02:29.9013942Z Plugin: 'Test File Publisher plugin' is running in background.
2025-02-12T00:02:29.9014570Z Start tracking orphan processes.
2025-02-12T00:02:29.9275423Z ##[section]Finishing: Initialize job

[kristapratico]

@danchengcaa Given this doesn't reproduce outside the pipeline, I think it would be helpful to do some debug logging in your pipeline to verify that azure-batch is getting installed and it is the version you expect. Can you try to output the result of pip show azure-batch? I'd also be curious if it is getting installed in a virtual environment or globally.

[ShermanCAA]

@kristapratico I think here might be the problem: on Azure DevOps somehow it was building from a different wheel distribution

We are on Microsoft-hosted machines in DevOps so it's harder to debug, but this is what I found by inspecting the module attributes of azure.batch there:

==================================== ERRORS ====================================
_____ ERROR collecting Pipelines/RestartPools/tests/test_restart_pools.py ______
ImportError while importing test module '/home/vsts/work/1/s/Pipelines/RestartPools/tests/test_restart_pools.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
Pipelines/RestartPools/tests/test_restart_pools.py:15: in <module>
    import restart_pools as p
Pipelines/RestartPools/activities/python/restart_pools.py:35: in <module>
    import common.batch.pool as p
common/common/batch/pool.py:5: in <module>
    raise ImportError(f"Looking at the module attributes: {dir(azure_batch)}")
E   ImportError: Looking at the module attributes: ['BatchClient', 'VERSION', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_client', '_configuration', '_model_base', '_operations', '_patch', '_patch_all', '_patch_sdk', '_serialization', '_vendor', '_version', 'models']
=============================== warnings summary ===============================

so 14.2 (https://github.com/Azure/azure-sdk-for-python/tree/azure-batch_14.2.0/sdk/batch/azure-batch/azure/batch) has BatchServiceClient, but main ( https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/batch/azure-batch/azure/batch) doesn't. Logs show that we are pulling version 14.2 for azure-batch, but is it possible that somehow it got repointed to main in PyPi?

...
Collecting azure-batch (from common==1.0)
  Downloading azure_batch-14.2.0-py3-none-any.whl.metadata 
...

[danchengcaa]

@kristapratico I think we found out the root cause. We have a dependency on azure-cli which the latest stable version got installed on 02/10/2025 , https://pypi.org/project/azure-cli/2.69.0/
However, this stable version is using an underlying preview version of azure-batch https://pypi.org/project/azure-batch/15.0.0b1/
instead of azure-batch 14.2.0

Here is my steps to replicate locally after running pip install azure-batch==15.0.0b1

from azure.batch import BatchServiceClient
Traceback (most recent call last):
File "", line 1, in
ImportError: cannot import name 'BatchServiceClient' from 'azure.batch' (C:\Users\DCheng\AppData\Local\pypoetry\Cache\virtualenvs\datafactory-mhz9-pmm-py3.10\lib\site-packages\azure\batch_init_.py)

[kristapratico]

@ShermanCAA @danchengcaa nice investigation! So it seems there is a dependency conflict for azure-batch between the versions that azure-cli and Devops wants? I suggest opening an issue on azure-cli repo and referencing what you found in this issue.

[ShermanCAA]

@kristapratico so is there any plan for official new release of azure-batch, since azure-cli is dependent on the new changes now?

[kristapratico]

@kristapratico so is there any plan for official new release of azure-batch, since azure-cli is dependent on the new changes now?

Can you elaborate what you mean by "official new release" of azure-batch? Do you mean stable release? I would have to tag the owners of the library to answer that question: @cRui861 @dpwatrous @jingjlii @JJJessieWang @wanghoppe @wiboris

[ShermanCAA]

@kristapratico correct stable release for azure-batch, since 15.0.0b1 has been a pre-release version for almost 6 months now but somehow the latest stable release from other packages like azure-cli started depending on it, which is the root cause of this problem