Audio Live Mode with Tool Use

[dselman]

Description of the feature request:

The documentation is quite vague on this issue: https://ai.google.dev/api/multimodal-live#function-calling

Specifically, "Audio inputs and audio outputs negatively impact the model's ability to use function calling." — which is what I think I am seeing. When I use tools with audio input & output the tool responses sent back seem to be interpreted as part of the conversation, or at least the audio part of the conversation gets very confused.

Is this a bug? Or limitation? Should we expect improvements for this use case?

What problem are you trying to solve with this feature?

I would like to conduct an audio live mode interaction (audio In and Out) but with calls to some tools being performed in the background.

Any other information you'd like to share?

No response

[hapticdata]

#47 things are always getting better 😉