Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MLClient.create_or_update fails with pip issue #39764

Closed
BillmanH opened this issue Feb 16, 2025 · 3 comments
Closed

MLClient.create_or_update fails with pip issue #39764

BillmanH opened this issue Feb 16, 2025 · 3 comments
Labels
Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. Machine Learning needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team.

Comments

@BillmanH
Copy link
Contributor

  • Package Name: AzureML
  • Package Version: Whatever version is on the AML Instance. The issue happens in the cloud.
  • Operating System: AzureML Service: mcr.microsoft.com/azureml/base:latest
  • Python Version : 3.12.9
    env = Environment(
        image="mcr.microsoft.com/azureml/base:latest",
        name=env_config["name"],
        version = "1",
        conda_file="env.yaml",
    )


# Register the environment in Azure ML
ml_client.environments.create_or_update(env)

Fails when trying to create a conda env. From the std_log it looks like a pip issue.

Error:

2025-02-16T00:26:20: AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
2025-02-16T00:26:20: �[0m�[91m

2025-02-16T00:26:20: CondaValueError: pip returned an error

2025-02-16T00:26:25: The command '/bin/sh -c ldconfig /usr/local/cuda/lib64/stubs && conda env create -p /azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09 -f azureml-environment-setup/mutated_conda_dependencies.yml && rm -rf "$HOME/.cache/pip" && conda clean -aqy && CONDA_ROOT_DIR=$(conda info --root) && rm -rf "$CONDA_ROOT_DIR/pkgs" && find "$CONDA_ROOT_DIR" -type d -name __pycache__ -exec rm -rf {} + && ldconfig' returned a non-zero code: 1

2025-02-16T00:26:25: �[0m

2025-02-16T00:26:25: CalledProcessError(1, ['docker', 'build', '-f', 'azureml-environment-setup/Dockerfile', '.', '-t', 'billmanhamlacr.azurecr.io/azureml/azureml_b8611fa83e4cf272933615613dc2fc6b', '-t', 'billmanhamlacr.azurecr.io/azureml/azureml_b8611fa83e4cf272933615613dc2fc6b:1'])

2025-02-16T00:26:25: Building docker image failed with exit code: 1

2025-02-16T00:26:25: Logging out of Docker registry: billmanhamlacr.azurecr.io
2025-02-16T00:26:25: Removing login credentials for https://index.docker.io/v1/


2025-02-16T00:26:25: Traceback (most recent call last):
  File "/mnt/azureml/cr/j/f24974886d0f496bbb734e82c6cf16cc/exe/wd/docker_utilities.py", line 152, in _docker_build_or_error
    docker_execute_function(docker_command, build_command, print_command_args=True)
  File "/mnt/azureml/cr/j/f24974886d0f496bbb734e82c6cf16cc/exe/wd/docker_utilities.py", line 23, in docker_execute_function
    return killable_subprocess.check_call(command_args, *popen_args,
  File "/mnt/azureml/cr/j/f24974886d0f496bbb734e82c6cf16cc/exe/wd/killable_subprocess.py", line 261, in check_call
    raise subprocess.CalledProcessError(process.returncode, cmd)
subprocess.CalledProcessError: Command '['docker', 'build', '-f', 'azureml-environment-setup/Dockerfile', '.', '-t', 'billmanhamlacr.azurecr.io/azureml/azureml_b8611fa83e4cf272933615613dc2fc6b', '-t', 'billmanhamlacr.azurecr.io/azureml/azureml_b8611fa83e4cf272933615613dc2fc6b:1']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "script.py", line 110, in <module>
    docker_utilities._docker_build_or_error(
  File "/mnt/azureml/cr/j/f24974886d0f496bbb734e82c6cf16cc/exe/wd/docker_utilities.py", line 156, in _docker_build_or_error
    _write_error_and_exit(error_msg, error_file_path=error_file_path)
  File "/mnt/azureml/cr/j/f24974886d0f496bbb734e82c6cf16cc/exe/wd/docker_utilities.py", line 217, in _write_error_and_exit
    sys.exit(1)
SystemExit: 1

Code above should repro. Building the same enviornment locally yields no problems. I'll include the env.yaml as well as the full stack trace. However, I noticed that the line: conda env create -p /azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09 -f azureml-environment-setup/mutated_conda_dependencies.yml

mutated_conda_dependencies.yml is adding or removing packages? Is it possible you are adding something that I'm not?

I saw this line in the stack trace:

2025-02-16T00:26:20:   File "/azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09/lib/python3.12/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module>
2025-02-16T00:26:20:     register_finder(pkgutil.ImpImporter, find_on_path)
2025-02-16T00:26:20:                     ^^^^^^^^^^^^^^^^^^^
2025-02-16T00:26:20: AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
2025-02-16T00:26:20: �[0m�[91m

2025-02-16T00:26:20: CondaValueError: pip returned an error

Sounds similar to this SO post: https://stackoverflow.com/questions/77364550/attributeerror-module-pkgutil-has-no-attribute-impimporter-did-you-mean

This makes it impossible to create an env using the sdk.

Expected behavior
If you AML is adding packages via mutated_conda_dependencies, then you should fix the packages dependencies so that you can build an environment without fail.

My env.yaml (pretty short)

name: billmanh-env
channels:
  - conda-forge
dependencies:
  - python=3.12.9
  - pip
  - pip:
      - mlflow
      - argparse
      - azure-ai-ml
      - azureml-mlflow
      - azureml-core
      - azure-identity
      - pandas
      - numpy
      - scikit-learn
      - matplotlib
      - joblib
      - pyyaml
      - uuid
@github-actions github-actions bot added Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. Machine Learning needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team. labels Feb 16, 2025
@github-actions GitHub Actions
Copy link

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/azure-ml-sdk @azureml-github.

@BillmanH
Copy link
Contributor Author

Full stack trace:

2025-02-16T00:22:57: Logging into Docker registry: billmanhamlacr.azurecr.io
2025-02-16T00:22:57: WARNING! Using --password via the CLI is insecure. Use --password-stdin.
2025-02-16T00:22:57: WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
2025-02-16T00:22:57: Configure a credential helper to remove this warning. See
2025-02-16T00:22:57: https://docs.docker.com/engine/reference/commandline/login/#credentials-store

2025-02-16T00:22:57: Login Succeeded


2025-02-16T00:22:57: Running: ['docker', 'build', '-f', 'azureml-environment-setup/Dockerfile', '.', '-t', 'billmanhamlacr.azurecr.io/azureml/azureml_b8611fa83e4cf272933615613dc2fc6b', '-t', 'billmanhamlacr.azurecr.io/azureml/azureml_b8611fa83e4cf272933615613dc2fc6b:1']
2025-02-16T00:22:57: DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
2025-02-16T00:22:57:             Install the buildx component to build images with BuildKit:
2025-02-16T00:22:57:             https://docs.docker.com/go/buildx/

2025-02-16T00:22:57: Sending build context to Docker daemon  74.24kB
2025-02-16T00:22:57: Step 1/17 : FROM mcr.microsoft.com/azureml/base:latest@sha256:dde960c143b301c9372611607f5d0e8f1e600ab1d40a8f1435090b02990cd51b
2025-02-16T00:22:57: mcr.microsoft.com/azureml/base:latest@sha256:dde960c143b301c9372611607f5d0e8f1e600ab1d40a8f1435090b02990cd51b: Pulling from azureml/base
2025-02-16T00:22:57: 3386e6af03b0: Pulling fs layer
2025-02-16T00:22:57: 49ac0bbe6c8e: Pulling fs layer
2025-02-16T00:22:57: d1983a67e104: Pulling fs layer
2025-02-16T00:22:57: 1a0f3a523f04: Pulling fs layer
2025-02-16T00:22:57: f0658fc0d198: Pulling fs layer
2025-02-16T00:22:57: 661aa7a3d7b1: Pulling fs layer
2025-02-16T00:22:57: 00f885d273ab: Pulling fs layer
2025-02-16T00:22:57: e2fbc5971987: Pulling fs layer
2025-02-16T00:22:57: 1735122ca526: Pulling fs layer
2025-02-16T00:22:57: 0efe336792bc: Pulling fs layer
2025-02-16T00:22:57: 6c47b4786d87: Pulling fs layer
2025-02-16T00:22:57: 661aa7a3d7b1: Waiting
2025-02-16T00:22:57: 00f885d273ab: Waiting
2025-02-16T00:22:57: e2fbc5971987: Waiting
2025-02-16T00:22:57: 1735122ca526: Waiting
2025-02-16T00:22:57: 0efe336792bc: Waiting
2025-02-16T00:22:57: 6c47b4786d87: Waiting
2025-02-16T00:22:57: 1a0f3a523f04: Waiting
2025-02-16T00:22:57: f0658fc0d198: Waiting
2025-02-16T00:22:58: 49ac0bbe6c8e: Download complete
2025-02-16T00:22:58: d1983a67e104: Download complete
2025-02-16T00:22:58: 1a0f3a523f04: Download complete
2025-02-16T00:22:59: 3386e6af03b0: Verifying Checksum
2025-02-16T00:22:59: 3386e6af03b0: Download complete
2025-02-16T00:22:59: 661aa7a3d7b1: Verifying Checksum
2025-02-16T00:22:59: 661aa7a3d7b1: Download complete
2025-02-16T00:23:00: 00f885d273ab: Verifying Checksum
2025-02-16T00:23:00: 00f885d273ab: Download complete
2025-02-16T00:23:01: f0658fc0d198: Download complete
2025-02-16T00:23:01: 3386e6af03b0: Pull complete
2025-02-16T00:23:01: 0efe336792bc: Verifying Checksum
2025-02-16T00:23:01: 0efe336792bc: Download complete
2025-02-16T00:23:02: e2fbc5971987: Verifying Checksum
2025-02-16T00:23:02: e2fbc5971987: Download complete
2025-02-16T00:23:02: 6c47b4786d87: Download complete
2025-02-16T00:23:04: 1735122ca526: Verifying Checksum
2025-02-16T00:23:04: 1735122ca526: Download complete
2025-02-16T00:23:04: 49ac0bbe6c8e: Pull complete
2025-02-16T00:23:04: d1983a67e104: Pull complete
2025-02-16T00:23:04: 1a0f3a523f04: Pull complete
2025-02-16T00:23:10: f0658fc0d198: Pull complete
2025-02-16T00:23:11: 661aa7a3d7b1: Pull complete
2025-02-16T00:23:14: 00f885d273ab: Pull complete
2025-02-16T00:23:18: e2fbc5971987: Pull complete
2025-02-16T00:23:22: 1735122ca526: Pull complete
2025-02-16T00:23:22: 0efe336792bc: Pull complete
2025-02-16T00:23:22: 6c47b4786d87: Pull complete
2025-02-16T00:23:22: Digest: sha256:dde960c143b301c9372611607f5d0e8f1e600ab1d40a8f1435090b02990cd51b
2025-02-16T00:23:22: Status: Downloaded newer image for mcr.microsoft.com/azureml/base:latest@sha256:dde960c143b301c9372611607f5d0e8f1e600ab1d40a8f1435090b02990cd51b
2025-02-16T00:23:22:  ---> 15b2e3b5af7a
2025-02-16T00:23:22: Step 2/17 : USER root
2025-02-16T00:23:23:  ---> Running in 13acc1ac97b3
2025-02-16T00:23:24: Removing intermediate container 13acc1ac97b3
2025-02-16T00:23:24:  ---> 7590afd37986
2025-02-16T00:23:24: Step 3/17 : RUN mkdir -p $HOME/.cache
2025-02-16T00:23:24:  ---> Running in f0618ff76395
2025-02-16T00:23:25: Removing intermediate container f0618ff76395
2025-02-16T00:23:25:  ---> 21ae423cdd93
2025-02-16T00:23:25: Step 4/17 : WORKDIR /
2025-02-16T00:23:25:  ---> Running in ecc97d995923
2025-02-16T00:23:26: Removing intermediate container ecc97d995923
2025-02-16T00:23:26:  ---> c7ad71240a2c
2025-02-16T00:23:26: Step 5/17 : COPY azureml-environment-setup/99brokenproxy /etc/apt/apt.conf.d/
2025-02-16T00:23:27:  ---> 8b2b6d3fc75c
2025-02-16T00:23:27: Step 6/17 : RUN if dpkg --compare-versions `conda --version | grep -oE '[^ ]+$'` lt 4.4.11; then conda install conda==4.4.11; fi
2025-02-16T00:23:27:  ---> Running in 5a36b40fd559
2025-02-16T00:23:29: Removing intermediate container 5a36b40fd559
2025-02-16T00:23:29:  ---> f2e6727910ca
2025-02-16T00:23:29: Step 7/17 : COPY azureml-environment-setup/mutated_conda_dependencies.yml azureml-environment-setup/mutated_conda_dependencies.yml
2025-02-16T00:23:30:  ---> 14c555061e4c
2025-02-16T00:23:30: Step 8/17 : RUN ldconfig /usr/local/cuda/lib64/stubs && conda env create -p /azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09 -f azureml-environment-setup/mutated_conda_dependencies.yml && rm -rf "$HOME/.cache/pip" && conda clean -aqy && CONDA_ROOT_DIR=$(conda info --root) && rm -rf "$CONDA_ROOT_DIR/pkgs" && find "$CONDA_ROOT_DIR" -type d -name __pycache__ -exec rm -rf {} + && ldconfig
2025-02-16T00:23:30:  ---> Running in 2d39d7ba6bb5
2025-02-16T00:25:09: Solving environment: ...working... done
2025-02-16T00:25:09: �[91m

2025-02-16T00:25:09: ==> WARNING: A newer version of conda exists. <==
2025-02-16T00:25:09:   current version: 4.5.11
2025-02-16T00:25:09:   latest version: 25.1.1

2025-02-16T00:25:09: Please update conda by running

2025-02-16T00:25:09:     $ conda update -n base -c defaults conda


2025-02-16T00:25:10: tk-8.6.14            | 3.4 MB    | ########## | 100% �[0m�[91m
2025-02-16T00:25:10: ca-certificates-2022 | 150 KB    | ########## | 100% �[0m�[91m
2025-02-16T00:25:10: expat-2.6.4          | 197 KB    | ########## | 100% �[0m�[91m
2025-02-16T00:25:10: libuuid-1.41.5       | 29 KB     | ########## | 100% �[0m�[91m
2025-02-16T00:25:16: python-3.12.9        | 37.3 MB   | ########## | 100% �[0m�[91m
2025-02-16T00:25:17: libstdcxx-ng-12.2.0  | 4.3 MB    | ########## | 100% �[0m�[91m
2025-02-16T00:25:17: _openmp_mutex-4.5    | 23 KB     | ########## | 100% �[0m�[91m
2025-02-16T00:25:17: libzlib-1.2.13       | 64 KB     | ########## | 100% �[0m�[91m
2025-02-16T00:25:17: libffi-3.4.2         | 57 KB     | ########## | 100% �[0m�[91m
2025-02-16T00:25:17: bzip2-1.0.8          | 484 KB    | ########## | 100% �[0m�[91m
2025-02-16T00:25:17: libgomp-12.2.0       | 455 KB    | ########## | 100% �[0m�[91m
2025-02-16T00:25:18: readline-8.1.2       | 291 KB    | ########## | 100% �[0m�[91m
2025-02-16T00:25:18: ld_impl_linux-64-2.3 | 759 KB    | ########## | 100% �[0m�[91m
2025-02-16T00:25:18: pip-22.3.1           | 1.5 MB    | ########## | 100% �[0m�[91m
2025-02-16T00:25:19: ncurses-6.4          | 1.1 MB    | ########## | 100% �[0m�[91m
2025-02-16T00:25:19: sqlite-3.45.3        | 1.6 MB    | ########## | 100% �[0m�[91m
2025-02-16T00:25:19: zlib-1.2.13          | 92 KB     | ########## | 100% �[0m�[91m
2025-02-16T00:25:19: wheel-0.38.4         | 32 KB     | ########## | 100% �[0m�[91m
2025-02-16T00:25:20: _libgcc_mutex-0.1    | 3 KB      | ########## | 100% �[0m�[91m
2025-02-16T00:25:20: tzdata-2022f         | 118 KB    | ########## | 100% �[0m�[91m
2025-02-16T00:25:20: libgcc-ng-12.2.0     | 931 KB    | ########## | 100% �[0m�[91m
2025-02-16T00:25:20: xz-5.6.4             | 567 KB    | ########## | 100% �[0m�[91m
2025-02-16T00:25:20: setuptools-65.5.1    | 731 KB    | ########## | 100% �[0m�[91m
2025-02-16T00:25:21: openssl-3.0.15       | 5.2 MB    | ########## | 100% �[0m
2025-02-16T00:25:21: Downloading and Extracting Packages
2025-02-16T00:25:22: Preparing transaction: ...working... done
2025-02-16T00:25:23: Verifying transaction: ...working... done
2025-02-16T00:26:20: Executing transaction: ...working... done
2025-02-16T00:26:20: �[91mERROR: Exception:
2025-02-16T00:26:20: Traceback (most recent call last):
2025-02-16T00:26:20:   File "/azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
2025-02-16T00:26:20:     status = run_func(*args)
2025-02-16T00:26:20:              ^^^^^^^^^^^^^^^
2025-02-16T00:26:20:   File "/azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
2025-02-16T00:26:20:     return func(self, options, args)
2025-02-16T00:26:20:            ^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-16T00:26:20:   File "/azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09/lib/python3.12/site-packages/pip/_internal/commands/install.py", line 326, in run
2025-02-16T00:26:20:     session = self.get_default_session(options)
2025-02-16T00:26:20:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-16T00:26:20:   File "/azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 98, in get_default_session
2025-02-16T00:26:20:     self._session = self.enter_context(self._build_session(options))
2025-02-16T00:26:20:                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-16T00:26:20:   File "/azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 125, in _build_session
2025-02-16T00:26:20:     session = PipSession(
2025-02-16T00:26:20:               ^^^^^^^^^^^
2025-02-16T00:26:20:   File "/azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09/lib/python3.12/site-packages/pip/_internal/network/session.py", line 343, in __init__
2025-02-16T00:26:20:     self.headers["User-Agent"] = user_agent()
2025-02-16T00:26:20:                                  ^^^^^^^^^^^^
2025-02-16T00:26:20:   File "/azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09/lib/python3.12/site-packages/pip/_internal/network/session.py", line 175, in user_agent
2025-02-16T00:26:20:     setuptools_dist = get_default_environment().get_distribution("setuptools")
2025-02-16T00:26:20:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-16T00:26:20:   File "/azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 188, in get_distribution
2025-02-16T00:26:20:     return next(matches, None)
2025-02-16T00:26:20:            ^^^^^^^^^^^^^^^^^^^
2025-02-16T00:26:20:   File "/azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 185, in <genexpr>
2025-02-16T00:26:20:     for distribution in self.iter_all_distributions()
2025-02-16T00:26:20:                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-16T00:26:20:   File "/azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09/lib/python3.12/site-packages/pip/_internal/metadata/base.py", line 612, in iter_all_distributions
2025-02-16T00:26:20:     for dist in self._iter_distributions():
2025-02-16T00:26:20:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-16T00:26:20:   File "/azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 176, in _iter_distributions
2025-02-16T00:26:20:     for dist in finder.find_eggs(location):
2025-02-16T00:26:20:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-16T00:26:20:   File "/azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 144, in find_eggs
2025-02-16T00:26:20:     yield from self._find_eggs_in_dir(location)
2025-02-16T00:26:20:   File "/azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 111, in _find_eggs_in_dir
2025-02-16T00:26:20:     from pip._vendor.pkg_resources import find_distributions
2025-02-16T00:26:20:   File "/azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09/lib/python3.12/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module>
2025-02-16T00:26:20:     register_finder(pkgutil.ImpImporter, find_on_path)
2025-02-16T00:26:20:                     ^^^^^^^^^^^^^^^^^^^
2025-02-16T00:26:20: AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
2025-02-16T00:26:20: �[0m�[91m

2025-02-16T00:26:20: CondaValueError: pip returned an error

2025-02-16T00:26:25: The command '/bin/sh -c ldconfig /usr/local/cuda/lib64/stubs && conda env create -p /azureml-envs/azureml_3f1df6da00c846b450f70fbe06959c09 -f azureml-environment-setup/mutated_conda_dependencies.yml && rm -rf "$HOME/.cache/pip" && conda clean -aqy && CONDA_ROOT_DIR=$(conda info --root) && rm -rf "$CONDA_ROOT_DIR/pkgs" && find "$CONDA_ROOT_DIR" -type d -name __pycache__ -exec rm -rf {} + && ldconfig' returned a non-zero code: 1

2025-02-16T00:26:25: �[0m

2025-02-16T00:26:25: CalledProcessError(1, ['docker', 'build', '-f', 'azureml-environment-setup/Dockerfile', '.', '-t', 'billmanhamlacr.azurecr.io/azureml/azureml_b8611fa83e4cf272933615613dc2fc6b', '-t', 'billmanhamlacr.azurecr.io/azureml/azureml_b8611fa83e4cf272933615613dc2fc6b:1'])

2025-02-16T00:26:25: Building docker image failed with exit code: 1

2025-02-16T00:26:25: Logging out of Docker registry: billmanhamlacr.azurecr.io
2025-02-16T00:26:25: Removing login credentials for https://index.docker.io/v1/


2025-02-16T00:26:25: Traceback (most recent call last):
  File "/mnt/azureml/cr/j/f24974886d0f496bbb734e82c6cf16cc/exe/wd/docker_utilities.py", line 152, in _docker_build_or_error
    docker_execute_function(docker_command, build_command, print_command_args=True)
  File "/mnt/azureml/cr/j/f24974886d0f496bbb734e82c6cf16cc/exe/wd/docker_utilities.py", line 23, in docker_execute_function
    return killable_subprocess.check_call(command_args, *popen_args,
  File "/mnt/azureml/cr/j/f24974886d0f496bbb734e82c6cf16cc/exe/wd/killable_subprocess.py", line 261, in check_call
    raise subprocess.CalledProcessError(process.returncode, cmd)
subprocess.CalledProcessError: Command '['docker', 'build', '-f', 'azureml-environment-setup/Dockerfile', '.', '-t', 'billmanhamlacr.azurecr.io/azureml/azureml_b8611fa83e4cf272933615613dc2fc6b', '-t', 'billmanhamlacr.azurecr.io/azureml/azureml_b8611fa83e4cf272933615613dc2fc6b:1']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "script.py", line 110, in <module>
    docker_utilities._docker_build_or_error(
  File "/mnt/azureml/cr/j/f24974886d0f496bbb734e82c6cf16cc/exe/wd/docker_utilities.py", line 156, in _docker_build_or_error
    _write_error_and_exit(error_msg, error_file_path=error_file_path)
  File "/mnt/azureml/cr/j/f24974886d0f496bbb734e82c6cf16cc/exe/wd/docker_utilities.py", line 217, in _write_error_and_exit
    sys.exit(1)
SystemExit: 1

@BillmanH
Copy link
Contributor Author

I was able to get an environment to work by just fiddling with the packages and running them again. In the case above I changed:

  • removed jsonschema from the env.yml
  • switched image to "mcr.microsoft.com/azureml/curated/sklearn-1.5:19"
name: billmanh-env
channels:
  - conda-forge
dependencies:
  - python=3.12.9
  - pip
  - pip:
      - mlflow
      - argparse
      - azure-ai-ml
      - azureml-mlflow
      - azureml-core
      - azure-identity
      - pandas
      - numpy
      - scikit-learn
      - matplotlib
      - joblib
      - pyyaml
      - uuid

I figured it out by randomly swapping things out until I got it to work. Basically, just button mashing. Having some insights into the mutated env file might make it easier?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. Machine Learning needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@BillmanH