azure.ai.evaluation.evaluate returning invalid json for groundedness_pro evaluation

[soferreira]

• Package Name:
  azure-ai-evaluation 1.2.0

• Package Version:
  azure-ai-evaluation 1.2.0
  azure-ai-inference 1.0.0b8
  azure-ai-projects 1.0.0b5

• Operating System:
  linux

• Python Version:
  Python 3.10.8

Describe the bug

azure.ai.evaluation.evaluate is returning an invalid json for some of the groundedness_pro evaluations

To Reproduce
Steps to reproduce the behavior:

1. Run evaluate with evaluator:

    results = evaluate(
        evaluation_name=f"Eval-Run-{randomNum}-{model}",
        data="input_data.jsonl",
        target=ModelEndpoints(model_endpoints, model),
        evaluators=evaluators,
        azure_ai_project=azure_ai_project,
        evaluator_config=evaluator_config,
        output_path="./myevalresults.json"
    )

Evaluators:

evaluators = {}
groundedness_pro_evaluator = GroundednessProEvaluator(azure_ai_project=azure_ai_project, credential=credential)
evaluators["groundedness_pro"] = groundedness_pro_evaluator

Column mapping with atributes from input_data.jsonl and target response:

"column_mapping": {
    "context": "AML API only"",
    "response": "To provide you with the most accurate guidance, I need a bit more context about what you mean by \"create a run.\" The term \"run\" can be interpreted differently depending on the context. Here are a few possibilities:\n\n1. **Running as Exercise**: If you are referring to planning a running route for exercise, you can create a run by mapping out a route using apps like Google Maps, Strava, or MapMyRun. Choose a safe area, consider the distance you want to cover",
    "query": "How do you create a run?"",
},

Expected behavior
evaluate() is returning an invalid json

Screenshots

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @luigiw @needuv @singankit.