Kubernetes: How to populate resource attributes based on attributes, labels and transformation #1756

zeitlinger · 2025-01-17T12:53:53Z

Fixes #236

This is a new take on the issue. #349 has been created before and not completed.

So what's the difference now?

The proposed rules are already implemented in the OTel operator and work well: https://github.com/open-telemetry/opentelemetry-operator#configure-resource-attributes - the PR (at least in it's initial form) makes the operator implementation fully compatible with this spec PR.
The specification work on service.instance.id has been completed - which was effectively a blocker before.

Merge requirement checklist

CONTRIBUTING.md guidelines followed.
Change log entry added, according to the guidelines in When to add a changelog entry.
- If your PR does not need a change log, start the PR title with [chore]

…ributes

…tice

dashpole

Love the overall direction. This makes a lot of sense

docs/resource/k8s.md

dashpole · 2025-01-17T15:50:42Z

docs/resource/k8s.md

+Choose the first value found:
+
+- `pod.annotation[resource.opentelemetry.io/service.instance.id]`
+- `concat([k8s.namespace.name, k8s.pod.name, k8s.container.name], '.')`


nit: should we use a delimiter that is less common in names? Maybe / or :?

. is already typically used for namespacing in otel

This isn't really namespacing, though. Is it? I guess at least most k8s names seem to use dashes... e.g. kube-system or kube-addon-manager. So maybe . isn't too bad.

ChrsMark · 2025-01-21T09:16:14Z

docs/resource/k8s.md

@@ -354,3 +354,70 @@ A CronJob creates Jobs on a repeating schedule.
 <!-- endsemconv -->

 [DocumentStatus]: https://opentelemetry.io/docs/specs/otel/document-status
+
+## Specify resource attributes using Kubernetes annotations


I wonder if this should be an OTEP (or non-normative guide) instead.
In the current format it looks more like guidance not a spec.

Also I wonder if that could be part of the service.* definition. What we are trying to do here is to define how service.* attributes should be populated in a specific environment not how K8s attributes are defined/affected.

@open-telemetry/specs-semconv-approvers thoughts?

dmitryax · 2025-01-22T16:32:35Z

We briefly discussed this during the k8s SemConv WG meeting. There is no consensus on the document yet, but we decided to recommend moving it from the k8s resource section of the semantic conventions repo to service because it prescribes a way of populating values of service attributes, not attributes in k8s namespace.

jinja2 · 2025-01-23T21:09:47Z

docs/resource/k8s.md

+Choose the first value found:
+
+- `pod.annotation[resource.opentelemetry.io/service.name]`
+- `pod.label[app.kubernetes.io/name]` (well-known label


I think we should also look for the label app.kubernetes.io/instance. The value of this label is supposed to be a unique name to help differentiate multiple instances of the same application. In a helm chart, this will be same as the Release.Name. So if a user is installing an application twice for different usecases in the same namespace, this is the label which will differentiate the 2 services. And I think we should reference this before we check the more generic label app.kubernetes.io/name.

service.name isn't meant to differentiate multiple instances of the same application, right? The example from https://opentelemetry.io/docs/specs/semconv/attributes-registry/service/#service-attributes is "shoppingcart"

To clarify the usage of instance of application, let me use an example. I have to run 2 different statfulsets of the mysql db with greater than 1 pod in each in the same namespace. These are meant to be 2 distinct installations of the db, each maintaining its unique dataset.* Following the guidance of k8s std labels, i'll have below values

Statefulset 1 - app.kubernetes.io/name=mysql, app.kubernetes.io/instance: mysql-abc
Statefulset 2 - app.kubernetes.io/name=mysql, app.kubernetes.io/instance: mysql-xyz

My reading of the attributes definition - Logical name of the service., makes me think the unique mysql-abc and mysql-xyz names are more appropriate since i can differentiate which installation of the db the telemetry is coming from.

edited for more clarity

@open-telemetry/specs-semconv-approvers can someone help clarify the intention behind service.name here?

looking at https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/#applications-and-instances-of-applications - I would say that app.kubernetes.io/instance is the right translation for service.name - I didn't see that before.

Definition of service.name - see https://github.com/open-telemetry/semantic-conventions/blob/main/docs/attributes-registry/service.md

service.name: MUST be the same for all instances of horizontally scaled services

I'm wondering is we should fall back to app.kubernetes.io/name as the next best thing if app.kubernetes.io/instance is not found.

I'm wondering is we should fall back to app.kubernetes.io/name as the next best thing if app.kubernetes.io/instance is not found.

yeah, that sounds good to me

jinja2 · 2025-01-23T21:16:22Z

docs/resource/k8s.md

+- `pod.annotation[resource.opentelemetry.io/service.version]`
+- `pod.label[app.kubernetes.io/version]` (well-known label
+  [app.kubernetes.io/version](https://kubernetes.io/docs/reference/labels-annotations-taints/#app-kubernetes-io-version))
+- `if (contains(container docker image tag, '/') == false) container docker image tag`


Is there a reason we bail if the tag has /? The service attribute conventions don't define a format for version. If none of the other options work, wouldn't using the tag be better (not sure what the sdk defaults to)?

I'm unsure about this part of the spec.

I took this from the way the operator logic - collector does it a bit differently.

I've found the reason - it was a guard for a port number separator as can be seen here: https://github.com/distribution/reference/blob/284a39eaf3368476e0a4c6114a0eec61220acdd9/reference_test.go#L250

I've updated the spec to refer to that library - which is already used by the collector for container.image.id -

... with the notable difference that I think we should include the digest if available, it's the best source of truth for the service version

jinja2 · 2025-01-23T21:21:20Z

docs/resource/k8s.md

+Choose the first value found:
+
+- `pod.annotation[resource.opentelemetry.io/service.instance.id]`
+- `concat([k8s.namespace.name, k8s.pod.name, k8s.container.name], '.')`


We discussed service.instance.id in the semconv meeting and had some doubts about the scenario where a container is restarted without a pod recreate (e.g. kubelet restarting a container for OOM). The confusion was whether this id should change to reflect that a new instance of the container was started which is what would happen i think if we left this to an sdk.

The convention for service.instance.id states that it MUST be unique for each instance of the same service.namespace,service.name pair (in other words service.namespace,service.name,service.instance.id triplet MUST be globally unique). For a pod which is owned by a statefulset this is not the case. I guess the question is whether this id should be used to differentiate between different instances of a service at a given time or across all instances ever run. In case of a statefulset, the id is not unique enough to differentiate between instances of a pod with same sts ordinal but k8s will only run one instance of a pod with a given ordinal (the id will remain same after an sts pod is recreated). If we want a different id across pod recreates, switching to pod uid instead of name would work.

I've generally interpreted "globally unique" here to mean unique at a given time, rather than unique across all time. IMO this is somewhat moot since SDKs are going to start generating unique IDs for service.instance.id anyways.

IMO this is somewhat moot since SDKs are going to start generating unique IDs for service.instance.id anyways.

Are sdks going to ignore any service.instance.id values passed in with the env var OTEL_RESOURCE_ATTRIBUTES? My understanding is that the documented values are currently being calculated and set in this var by operator when auto-instrumenting.

IMO this is somewhat moot since SDKs are going to start generating unique IDs for service.instance.id anyways.

Are sdks going to ignore any service.instance.id values passed in with the env var OTEL_RESOURCE_ATTRIBUTES? My understanding is that the documented values are currently being calculated and set in this var by operator when auto-instrumenting.

no, SDKs must honor whatever attributes are set using OTEL_RESOURCE_ATTRIBUTES

I've generally interpreted "globally unique" here to mean unique at a given time, rather than unique across all time.

that is also my understanding

zeitlinger · 2025-01-27T13:28:15Z

We briefly discussed this during the k8s SemConv WG meeting. There is no consensus on the document yet, but we decided to recommend moving it from the k8s resource section of the semantic conventions repo to service because it prescribes a way of populating values of service attributes, not attributes in k8s namespace.

@open-telemetry/specs-semconv-approvers the service page is automatically generated - how can I move the content of this PR to the service page?

…d prod (deployment.envirnoment.name)

trask · 2025-01-28T21:40:45Z

@open-telemetry/specs-semconv-approvers the service page is automatically generated - how can I move the content of this PR to the service page?

maybe as subsection(s) under https://github.com/open-telemetry/semantic-conventions/blob/main/docs/resource/README.md#service

jinja2 · 2025-01-30T01:25:37Z

docs/resource/README.md

@@ -22,6 +22,11 @@ This document defines standard attributes for resources. These attributes are ty
  - [Semantic Attributes with Dedicated Environment Variable](#semantic-attributes-with-dedicated-environment-variable)
  - [Semantic Attributes with SDK-provided Default Value](#semantic-attributes-with-sdk-provided-default-value)
 - [Service](#service)
+  - [Service in Kubernetes](#service-in-kubernetes)
+    - [How `service.namespace` is calculated](#how-servicenamespace-is-calculated)


Since this is currently applicable to auto-instrumentation with operator, I think we should clarify that these are the guidelines suggested by conventions and adopted in operator. So maybe change the heading to

Suggested change

- [How `service.namespace` is calculated](#how-servicenamespace-is-calculated)

- [How `service.namespace` can be calculated](#how-servicenamespace-can-be-calculated)

good point

maybe "should" is better?

smithclay and others added 2 commits January 17, 2025 13:01

Add recommendation for well-known k8s labels to existing resource att…

d883b85

…ributes

update resource attribute transformations based on operator best prac…

0cab634

…tice

zeitlinger requested review from a team as code owners January 17, 2025 12:53

zeitlinger added 2 commits January 17, 2025 13:58

update resource attribute transformations based on operator best prac…

c25bef3

…tice

update resource attribute transformations based on operator best prac…

d29a822

…tice

zeitlinger requested a review from a team as a code owner January 17, 2025 13:01

zeitlinger mentioned this pull request Jan 17, 2025

[processor/k8sattributes] Operator resource attributes open-telemetry/opentelemetry-collector-contrib#37114

Open

dashpole reviewed Jan 17, 2025

View reviewed changes

ChrsMark added the area:k8s label Jan 20, 2025

ChrsMark requested a review from a team January 20, 2025 10:59

add fallback to k8s.namespace.name

3bdfb50

ChrsMark reviewed Jan 21, 2025

View reviewed changes

jinja2 reviewed Jan 23, 2025

View reviewed changes

remove part-of, which can sometimes be used to distinguish staging an…

3d15651

…d prod (deployment.envirnoment.name)

github-actions bot added the enhancement New feature or request label Jan 28, 2025

zeitlinger added 3 commits January 28, 2025 12:50

add app.kubernetes.io/instance

f561458

clarify how version is calculated

d4e7588

clarify how version is calculated

154d93f

move to service section

3652308

jinja2 reviewed Jan 30, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kubernetes: How to populate resource attributes based on attributes, labels and transformation #1756

Kubernetes: How to populate resource attributes based on attributes, labels and transformation #1756

zeitlinger commented Jan 17, 2025 •

edited

Loading

dashpole left a comment

dashpole Jan 17, 2025

zeitlinger Jan 20, 2025

dashpole Jan 21, 2025 •

edited

Loading

ChrsMark Jan 21, 2025

dmitryax commented Jan 22, 2025

jinja2 Jan 23, 2025

dashpole Jan 23, 2025

jinja2 Jan 23, 2025 •

edited

Loading

dashpole Jan 24, 2025

zeitlinger Jan 27, 2025

zeitlinger Jan 27, 2025

jinja2 Jan 30, 2025

jinja2 Jan 23, 2025

zeitlinger Jan 27, 2025

zeitlinger Jan 28, 2025

jinja2 Jan 23, 2025

jinja2 Jan 23, 2025 •

edited

Loading

dashpole Jan 24, 2025

jinja2 Jan 24, 2025 •

edited

Loading

zeitlinger Jan 27, 2025

zeitlinger Jan 27, 2025

zeitlinger commented Jan 27, 2025

trask commented Jan 28, 2025

jinja2 Jan 30, 2025

zeitlinger Jan 30, 2025

	- [How `service.namespace` is calculated](#how-servicenamespace-is-calculated)
	- [How `service.namespace` can be calculated](#how-servicenamespace-can-be-calculated)

Kubernetes: How to populate resource attributes based on attributes, labels and transformation #1756

Are you sure you want to change the base?

Kubernetes: How to populate resource attributes based on attributes, labels and transformation #1756

Conversation

zeitlinger commented Jan 17, 2025 • edited Loading

Merge requirement checklist

dashpole left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dashpole Jan 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmitryax commented Jan 22, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jinja2 Jan 23, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jinja2 Jan 23, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jinja2 Jan 24, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeitlinger commented Jan 27, 2025

trask commented Jan 28, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeitlinger commented Jan 17, 2025 •

edited

Loading

dashpole Jan 21, 2025 •

edited

Loading

jinja2 Jan 23, 2025 •

edited

Loading

jinja2 Jan 23, 2025 •

edited

Loading

jinja2 Jan 24, 2025 •

edited

Loading