"too short binary" error while using Explore Logs with Multi-Tenant Loki.

[akshaykrjain]

Describe the bug
An error occurs when using the new “Grafana Explore Logs” feature with Loki running in multi-tenant mode. The plugin calls https://grafana.mycompany.net/api/datasources/uid/loki-global/resources/detected_labels? from the Grafana frontend, which in turn makes a request to https://loki.mycompany.net/loki/api/v1/detected_labels? and receives a 500 error from Loki.

Grafana shows this as "An error occurred within the plugin"

Versions-
• Grafana: v11.4.0 (b58701869e)
• Explore Log Plugin: v1.0.6
• Loki: v3.4.1

To Reproduce
Steps to reproduce the behavior:

1. Start the loki and enable multi-tenancy querier and configure a datasource in grafana with multiple tenants. (Using A|B format in X-Scope-OrgId header)

2. Navigate to Grafana Explore Home > Explore > Logs (Dont use traditional explore,select your loki multi-tenant datasource)

or call the Loki API directly

curl -i "https://loki.mycompany.net/loki/api/v1/detected_labels?query=&start=2025-02-13T09%3A16%3A57.260Z&end=2025-02-13T09%3A46%3A57.260Z" \ -H "Host: loki.svc.nepcloud.net" \ -H "X-Scope-OrgID: tenant-a|tenant-b"

it returns-

too short binary

Expected behavior
Either Grafana Plugin or Loki should handle this to avoid throwing plugin error. Perhaps detected_labels for multi-tenant should not respond with 500.

Screenshots, Promtail config, or terminal output
Logs from Loki (read)

level=debug ts=2025-02-14T14:30:19.306721629Z caller=multi_tenant_querier.go:321 msg="detected labels requested for multiple tenants, but not yet supported. returning static labels" tenantIDs=tenant-a|tenant-b
level=warn ts=2025-02-14T14:30:19.306958004Z caller=logging.go:128 orgID=tenant-a|tenant-b msg="GET /loki/api/v1/detected_labels?query=&start=2025-02-14T14%3A15%3A19.094Z&end=2025-02-14T14%3A30%3A19.094Z (500) 2.551893ms Response: \"too short binary\" ws: false; Accept: application/json, text/plain, */*; Accept-Encoding: gzip, deflate, br, zstd; Accept-Language: en-GB,en-US;q=0.9,en;q=0.8; Connection: close; Priority: u=1, i; Sec-Ch-Ua: \"Not(A:Brand\";v=\"99\", \"Google Chrome\";v=\"133\", \"Chromium\";v=\"133\"; Sec-Ch-Ua-Mobile: ?0; Sec-Ch-Ua-Platform: \"macOS\"; Sec-Fetch-Dest: empty; Sec-Fetch-Mode: cors; Sec-Fetch-Site: same-origin; User-Agent: Grafana/11.4.0; X-Amzn-Trace-Id: Root=1-67af537b-2cf0f3a31b421a2d70c917df; X-Datasource-Uid: loki-global; X-Forwarded-For: xxxx; X-Grafana-Id: exxxx; X-Grafana-Org-Id: 1; X-Grafana-Referer: https://grafana.svc.nepcloud.net/a/grafana-lokiexplore-app/explore?patterns=%5B%5D&var-primary_label=service_name%7C%3D~%7C.%2B&from=now-15m&to=now&timezone=browser&var-ds=loki-global&var-filters=&var-fields=&var-levels=&var-metadata=&var-patterns=&var-lineFilterV2=&var-lineFilters=; X-Plugin-Id: loki; X-Query-Tags: So

I am not sure where this issue should be posted either in Loki or in Grafana or in Plugin?