Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Version 1.34 metric oscillates #6229

Open
DasMagischeToastbrot opened this issue Jan 30, 2025 · 5 comments
Open

Version 1.34 metric oscillates #6229

DasMagischeToastbrot opened this issue Jan 30, 2025 · 5 comments
Labels
bug Something isn't working

Comments

@DasMagischeToastbrot
Copy link

Describe the bug

We transform the traces to metrics (histogram) via the otel-collectors. We updated otel in our go code to version 1.34 and now when the metric (histogram) gets a high and a low value, it oscillates around both values. Until v1.33 the metric just counts up. In v1.34 we don't see this anymore, is there a default ttl in place now?

Image

Environment

  • OS: Linux
  • Go Version: 1.23.0
  • Version: v1.34

Customization: We use explicit bucket boundaries:

const explicitBucketBoundaries = []float64{0.025, 0.05, 0.1, 0.25, 1.0, 5.0, 60.0}

 handleSuccessHisto, _ = meter.Int64Histogram(
  "my-application.handle.success",
  metric.WithDescription("Successfully handled message metric"),
  metric.WithExplicitBucketBoundaries(explicitBucketBoundaries...),
 )
@DasMagischeToastbrot DasMagischeToastbrot added the bug Something isn't working label Jan 30, 2025
@dmathieu
Copy link
Member

You mention these metrics are generated by the collector out of traces.
Do the traces have the same values as before? Is it possible the collector was upgraded around the same time as this?

0.34 includes an upgrade of the prometheus package, which include changes that could be surprising. Could this be the cause of your issue?
See the changelog for more details, and the mitigation.
https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md#added

@DasMagischeToastbrot
Copy link
Author

Yes the metrics get generated by a collector from traces. Yes the traces have the same value as before. We didn't upgrade the collector in the meantime. We

We assume that this upgrade of the prometheus package is the rootcause for this issue

@dmathieu
Copy link
Member

But you're not using prometheus, since you're only emitting traces?

@DasMagischeToastbrot
Copy link
Author

We are using prometheus, sorry I should have said it.

Our architecture is go application -> opentelemetry-agents -> opentelemetry-collector -> prometheus -> grafana

The opentelemetry-collector transforms the traces into metrics.

@dmathieu
Copy link
Member

Right, but you're not using the prometheus exporter from this SDK. So you're not using prometheus.
Traces are basically raw data which is then transformed by your collector.
Unless you do see a change in the traces data being emitted, your issue has no way of coming from this SDK, since exactly the same data is being emitted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants