Ad hoc filters not working with Tempo

[Dexiion]

Describe the bug
Hi!

We are switching from ApplicationInisghts in Azure to Tempo in grafana using OpenTelemetry, in AppInsights we had on the TransactionSearch page some filters where we can find by HTTP Code, Custom Properties, Duration, Success, etc.

We found that in grafana we have this also and they are called Ad Hoc filters (image 1) but when we add them they do not filter nothing as you can see on the image 2

To Reproduce
Steps to reproduce the behavior:

1. Create a pannel in grafana with tempo data source
2. Add an ad hoc variable
3. Try to filter by some of the labels

Expected behavior
The filters on the ad hoc are applied correctly to the data source so in this specific case It will only show the traces with resource.service.name of deployment-tool-agent

Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm

[joe-elliott]

This seems to be a frontend issue. Is Grafana executing new queries when you apply the ad hoc filter? or is it doing live filtering in the frontend?

[Dexiion]

Well when I add the filter the data is being refreshed but whit no sense, you can see that I have it ordered by Start Time and when I add the service.name It is not ordered, When I click on refresh new data appears but It is from like 4 hours. It is realy weird

[joe-elliott]

I'm not super familiar with this feature in Grafana so I'm making a guess here. I think that when you select deployment-tool-agent Grafana is writing an executing a query like { resource.service.name = "deployment-tool-agent" }. This can be confirmed by opening the network tab and seeing what queries Grafana is executing.

The query above will find any span with the requested service name. The "service" listed in the table is the root service which is why it may be different. Expand the traces by clicking the drop down on the left:

See if the returned spans all have the expected service.

[Dexiion]

This is the query that grafana is executing

Well I said that it was not working because for example sima-jobs-api has nothing to do with deployment-tool-agent however I added the filter I clicked on the first trace and the span service.name was "centralconfigurationapi"