Cardinality limit on metrics (otel_metrics_overflow)

[xjanin]

Hi,
I've instrumented my application with opentelemetry-cpp (metrics only), and I use tags in my metrics. When testing with a low rate of metrics recording, I see my metrics with the expected tags in the prometheus export on the opentelemetry collector. However, when testing the application with a high rate of requests, and so a high rate of metrics recording, I get metrics aggregated in time series with no tags except "otel_metrics_overflow".

My understanding is that this should happens if the sdk collector has to collect metrics with a tag cardinality superior to 2000 in a collection cycle (by default).
Edit: https://opentelemetry.io/docs/specs/otel/metrics/sdk/#cardinality-limits

However, my tag cardinality doesn't increase with the number of request processed, and in the opentelemetry collector that my application uses, I don't see 2000 time series in the prometheus export.

So my questions are :

• Do I have a wrong understanding of what "otel_metrics_overflow" means ?
• Is there a way of monitoring the cardinality of the metrics produced by my application ?

Thank you and best regards,
Xavier

[lalitb]

Your understanding is correct. The overflow metrics series is created only if the unique metric series (i.e., request types with different attributes) within a collection interval exceed 2000. Can you confirm if the unique request types for your application stay below this limit? Note that the total number of requests can exceed 2000, but the unique types should not.

And also do you mean the only metrics series you are seeing within collection interval is overflow, and no other?

[xjanin]

And also do you mean the only metrics series you are seeing within collection interval is overflow, and no other?

No, indeed, I see all the metrics that are expected, and they are still increasing (I use histograms mostly), but most of the count is added to the overflow time series.

I can tell you that the prometheus export doesn't exceed 800 lines during the test, but I'm not sure that it matches exactly what otel calls a metric point (https://opentelemetry.io/docs/specs/otel/metrics/data-model/#metric-points). It looks like it.

Is there a way in the SDK to measure the number of metrics points ?

[xjanin]

Hello,
I dug into the issue a bit more .

It seems to comes down to the hashing of the attributes to keep track of them.
When recording a value, the function sdk::metrics::IsOverflowAttributes in attributes_hashmap.h checks that a map of all attributes combinations hash_map_ isn't larger than 2000, which then triggers the overflow tag or not.
This map is built by sdk::metrics::GetOrSetDefault and sdk::metrics::Set functions. They use hashes computed by sdk::common::GetHashForAttributeMap functions in attributemap_hash.h.
The first version of this function takes a std::map<std::string, OwnedAttributeValue> in input, and uses std::hash to compute the key hash. This hash is on the content of the key.
The second version takes a opentelemetry::common::KeyValueIterable which uses string_views as key, and gives the pointer to the string_view data to std::hash. This hash is on the memory location of the key.

inline size_t GetHashForAttributeMap(
    const opentelemetry::common::KeyValueIterable &attributes,
    nostd::function_ref<bool(nostd::string_view)> is_key_present_callback)
{
  AttributeConverter converter;
  size_t seed = 0UL;
  attributes.ForEachKeyValue(
      [&](nostd::string_view key, opentelemetry::common::AttributeValue value) noexcept {
        if (!is_key_present_callback(key))
        {
          return true;
        }
        GetHash(seed, key.data()); // here
        auto attr_val = nostd::visit(converter, value);
        nostd::visit(GetHashForAttributeValueVisitor(seed), attr_val);
        return true;
      });
  return seed;
}

Is this expected ?

Note that the values of the map are hashed as values and not as pointers.

Personally I expect that the sdk considers the content of my key values and not their memory location.

[lalitb]

Note that the values of the map are hashed as values and not as pointers.

Thanks for investigating @xjanin. Your analysis is correct - the hash should be on the key content, not its address. Creating an issue out of the discussion to be fixed.

[xjanin]

Thanks for the confirmation, I'll keep an eye on the issue then.