[Donation Proposal]: compile time instrumentation for golang

[ralf0131]

Description

This is the formal donation proposal based on the previous discussion: #1961

Alibaba Cloud would like to donate the compile time instrumentation for golang to OpenTelmetry community. The project is a compile time instrumentation solution designed for Go applications. It empowers users to harness the capabilities of OpenTelemetry for enhanced observability without any manual modifications.

The core features of this project are:

• No user-side code modifications are required for the instrumentation to work.
• Support asynchronous context propagation, which does not require user to ensure the context is correctly passed.
• If users have previously added custom spans using the OTel-SDK within their code, the approach integrates these custom spans into the generated trace data, which significantly reduce the overhead for users to migrate from manual instrumentation to compile time instrumentation.
• Minor performance impact：5% cpu overhead, negligible memory footprint and < 1ms response time overhead
• Inspired by OpenTelemetry Java Agent, comprehensive test, like muzzle check, have been done to make sure the codes are working as expected

The side effect of this solution are:

• Increased the final binary size.
• Requires changes to compile command
• Extra compile time for applications

Benefits to the OpenTelemetry community

Currently there are several approaches to collect observability data for Golang applications:

1. Manual Instrumentation, which add custom instrumentation code through OTel-SDK
2. Compile time Instrumentation, which inject instrumentation code at compile time by leveraging the go build toolexec capability
3. eBPF based auto instrumentation, which inject instrumentation by injecting eBPF code into the kernel

For the OpenTelemetry community, it is important to have different approaches available to users, and each of the have its own advantage and disadvantage, and let users to choose the approach that best fix to their scenarios.
We found that the compile time instrumentation have not been gained enough awareness by the users and should be equally treated as an alternative to users.

Due to the lack of activity and maintenance, the existing compile time instrumentation Instrgen does not provide a production ready solution for users. The donation of Alibaba Cloud's compile time instrumentation will help the OpenTelemetry project to fill the gap in compile time instrumentation for golang. The OpenTelemetry users will gain yet another a feature rich and production ready solution for Golang and first class support as well as manual and eBPF based instrumentation.

In addition, with this donation, the OpenTelemetry community would benefit from having a team of observability engineers who works across different client instrumentations to (co-)maintain the Golang instrumentation and advance OpenTelemetry's influence especially in the APAC area.

Reasons for donation

Alibaba Cloud is actively embracing OpenTelemetry and the ecosystem, and would like to help expand the influence of OTel community to APAC area. We have completely rebuild our existing instrumentation based on OpenTelemetry, published the OpenTelemetry distribution for Java Agent, and open sourced the Golang compile time instrumentation, supporting OpenTelemetry natively in cloud service and contributing them back to the community.

Repository

https://github.com/alibaba/opentelemetry-go-auto-instrumentation

Existing usage

The project has been open sourced since Jan 2024, and the first release has been published on September. The commercial version which is built base the same core has been published on June and has been used by customers, and some of the customers are running in the production system.

Maintenance

In case of a successful donation Alibaba Cloud observability team would like to (co-)maintain this project and maintainers of existing approach are welcome to achieve a vendor neutral governance of the project.

Licenses

The project is licensed under the Apache 2.0 license.

Trademarks

The project has been open sourced with the aim to donate to OpenTelemetry, therefore it does not have any trademark other than OpenTelemetry.

Other notes

Roadmap of the project during/after the donation:

1. Support more libraries, such as gRPC / Gin / Kratos / Apache RocketMQ / Apache Dubbo etc.
2. Support runtime monitoring for golang, like garbage collection and etc.
3. Support the logging signal.
4. Support the profiling signal, including cpu and memory profiling, and support the latest OTel profiling data format.
5. Support trace and profiling correlation.
6. Extend the instrumentation rules to users to support customized instrumentation.
7. Support the GenAI semantic convention.