Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

azure-ai-ml unable to download logs and outputs from Job #39325

Open
sharmuz opened this issue Jan 21, 2025 · 7 comments
Open

azure-ai-ml unable to download logs and outputs from Job #39325

sharmuz opened this issue Jan 21, 2025 · 7 comments
Assignees
Labels
bug This issue requires a change to an existing behavior in the product in order to be resolved. Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. issue-addressed Workflow: The Azure SDK team believes it to be addressed and ready to close. Machine Learning Service Attention Workflow: This issue is responsible by Azure service team.

Comments

@sharmuz
Copy link

sharmuz commented Jan 21, 2025

  • Package Name: azure-ai-ml
  • Package Version: 1.24.0
  • Operating System: macOS Sequoia 15.2 (24C101)
  • Python Version: 3.10.15

Describe the bug
I am unable to download the logs and outputs saved within a completed Job in Azure ML. Despite both the Job class and JobOperations class providing the functionality, neither seems to work.

I can confirm the files exist in both the Azure ML studio portal and the associated blob storage account container.

To Reproduce
Steps to reproduce the behavior:

  1. Submit a job to Azure ML, for which files will be saved to logs/ and outputs/
  2. Wait for job to complete
  3. Attempt to run the following:
from azure.ai.ml import MLClient

ml_client = MLClient(...)
job_name = "curious_badger_124abc"

# Option 1: via JobOperations
ml_client.jobs.download(
    name=job_name,
    download_path=".",
    all=True,  # also fails with all=False and output_name="my_file.txt"
)

# Option 2: via Job
my_job = ml_client.jobs.get(name=job_name)
print(my_job.log_files)

Option 1 raises a ServiceRequestError
Option 2 returns None

Expected behavior
Option 1 should download logs and outputs to local machine.
Option 2 should return a dict of file names and URLs.

Additional context
Looking through the associated code for azure-ai-ml I found this comment - is getting a job's logs/outputs via the python SDK meant to be possible?

@github-actions github-actions bot added Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. Machine Learning needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team. labels Jan 21, 2025
Copy link

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/azure-ml-sdk @azureml-github.

@achauhan-scc achauhan-scc self-assigned this Jan 21, 2025
@catalinaperalta catalinaperalta added bug This issue requires a change to an existing behavior in the product in order to be resolved. and removed question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Jan 22, 2025
@achauhan-scc
Copy link
Member

achauhan-scc commented Jan 29, 2025

@sharmuz - download is the correct way to download output and logs
ml_client.jobs.download(
name=job_name,
download_path=".",
all=True, # also fails with all=False and output_name="my_file.txt"
)
I just tried with latest version of SDK and it worked fine.

Image Image Image Please try again and check the access to storage account, if it fails, please share full error stack trace.

@achauhan-scc achauhan-scc added the needs-author-feedback Workflow: More information is needed from author to address the issue. label Jan 30, 2025
Copy link

Hi @sharmuz. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

@github-actions github-actions bot removed the needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team label Jan 30, 2025
Copy link

github-actions bot commented Feb 6, 2025

Hi @sharmuz, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

@github-actions github-actions bot added the no-recent-activity There has been no recent activity on this issue. label Feb 6, 2025
@sharmuz
Copy link
Author

sharmuz commented Feb 20, 2025

@achauhan-scc I can confirm ml_client.jobs.download is now working for me.

We had an internal DNS issue which was interfering with our AML. Resolving it made download work.

ml_client.jobs.get(name=job_name).log_files is still returning None though - is that a known issue given the comment?

@github-actions github-actions bot added needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team and removed needs-author-feedback Workflow: More information is needed from author to address the issue. no-recent-activity There has been no recent activity on this issue. labels Feb 20, 2025
@achauhan-scc
Copy link
Member

Thanks for confirming.
log_files seems like a documentation bug, But it seems like added long back I will talk to few folks and get it cleared.
Please use download (for completed job) or stream (for running job) to access the job logs.
Marking the issue addressed.

@achauhan-scc achauhan-scc added the issue-addressed Workflow: The Azure SDK team believes it to be addressed and ready to close. label Feb 21, 2025
@github-actions github-actions bot removed the needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team label Feb 21, 2025
Copy link

Hi @sharmuz. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue requires a change to an existing behavior in the product in order to be resolved. Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. issue-addressed Workflow: The Azure SDK team believes it to be addressed and ready to close. Machine Learning Service Attention Workflow: This issue is responsible by Azure service team.
Projects
None yet
Development

No branches or pull requests

3 participants