Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: retry LLM calls when HTTP requests fail #1415

Open
sxlijin opened this issue Feb 5, 2025 · 0 comments
Open

feature: retry LLM calls when HTTP requests fail #1415

sxlijin opened this issue Feb 5, 2025 · 0 comments

Comments

@sxlijin
Copy link
Collaborator

sxlijin commented Feb 5, 2025

User is reporting that OpenAI's o1-mini are flaky and inconsistently return this error:

"error\": {
    \"message\": \"Invalid prompt: your prompt was flagged as potentially violating our usage policy. Please try again with a different prompt: https://platform.openai.com/docs/guides/reasoning#advice-on-prompting\",
    \"type\": \"invalid_request_error\",
    \"param\": null,
    \"code\": \"invalid_prompt\"
  }

This is not an error that BAML can retry today. The implementation of our retry policies is entirely built around API availability ie we move forward along the retry strategy if and only if api.openai.com is hard down. We don't retry application errors, because usually that implies the BAML user's request is somehow malformed and retries won't make a difference.

But if reasoning model guardrails are going to cause flakiness on these calls, we need to add retries for this, and possibly expose the ability to define retry conditions to the user.

Original Discord thread: https://discord.com/channels/1119368998161752075/1253172394345107466/1336810187788386347

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant