-
Notifications
You must be signed in to change notification settings - Fork 96
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OTLP: 401 error sending trace to consumer #1181
Comments
To aid reproducibilityI'm attempting to run all this from a nix flake: # flake.nix
{
description = "Reproducing beyla issues";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
rust-overlay.url = "github:oxalica/rust-overlay";
flake-utils.url = "github:numtide/flake-utils";
};
outputs = {
nixpkgs,
rust-overlay,
flake-utils,
...
}:
flake-utils.lib.eachDefaultSystem (
system: let
beyla = pkgs.stdenv.mkDerivation {
pname = "grafana_beyla";
version = "v1.8.4-alpha";
src = pkgs.fetchzip {
url = "https://github.com/grafana/beyla/releases/download/v1.8.4-alpha/beyla-linux-amd64-v1.8.4-alpha.tar.gz";
hash = "sha256-hlXgm71bMhmP2QYBTMsU4sEQILpAxdGOJd1DUr6cdW8=";
stripRoot = false;
};
installPhase = ''
mkdir -p $out/bin
cp $src/beyla $out/bin
'';
};
rust = pkgs.rust-bin.selectLatestNightlyWith (toolchain: toolchain.default);
overlays = [(import rust-overlay)];
pkgs = import nixpkgs {
inherit system overlays;
};
in
with pkgs; {
devShells.default = mkShell {
buildInputs = [
beyla
go
rust
];
shellHook = ''
## quickstart guide stuff: https://grafana.com/docs/beyla/latest/quickstart/rust/
export BEYLA_OPEN_PORT=8080
export BEYLA_TRACE_PRINTER=text
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-gateway-prod-sa-east-1.grafana.net/otlp"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <api_token?>"
## beyla connection steps: https://onic.grafana.net/connections/add-new-connection/beyla
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="https://otlp-gateway-prod-sa-east-1.grafana.net/otlp/v1/metrics"
export OTEL_EXPORTER_OTLP_METRICS_HEADERS="Authorization=Basic <value1_returned_in_script>"
export BEYLA_OTEL_METRIC_FEATURES="application_span,application_service_graph,application,application_process"
export BEYLA_NETWORK_METRICS="true"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://otlp-gateway-prod-sa-east-1.grafana.net/otlp/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Basic <value2_returned_in_script>"
'';
};
}
);
} This could come in handy when trying to reproduce the issue. ps: I'm aware no one has packaged beyla in nixpkgs yet. If after I'm able to run beyla successfully no one has already done it, I'll open the PR |
Solution to quickstartI got it to work, in the end it was really just confusion over how to actually quickstart it. export BEYLA_OPEN_PORT=8080
export BEYLA_TRACE_PRINTER=text
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-gateway-prod-sa-east-1.grafana.net/otlp"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <token>" but the token needs to be from Maybe solution?I think the reason why my attempts at following https://onic.grafana.net/connections/add-new-connection/beyla didn't work was because of my config (or lack thereof) of alloy. Final remarkThe thing I think we can improve is the documentation at the quickstart, it took me 2 days to figure out where to get this specific api token from. |
Thanks for all of this feedback! I think we need to improve our documentation, if you would like to contribute a PR we'll gladly take it, if not I think this write up helps us a ton and we'll improve the docs. |
When following https://grafana.com/docs/beyla/latest/quickstart/rust/ or https://grafana.com/docs/beyla/latest/quickstart/golang/
I can't send data to Grafana Cloud from both metrics and traces.
Also, is this quickstart up-to-date? Can we do something to make more clear what is the value we need to send in
OTEL_EXPORTER_OTLP_HEADERS
?When attempting to add Beyla connection in https://onic.grafana.net/connections/add-new-connection/beyla
The env variables are directly returned in the script:
If we export those values, along with
BEYLA_OPEN_PORT=8080
andBEYLA_TRACE_PRINTER=text
.We now get metrics working with Grafana Cloud, yay!!!
But traces still outputs the following error:
Why does traces not work? Am I missing some piece of the puzzle?
Also, as mentioned, I attempted this with both Rust and Go, and I'm using the Beyla v1.8.4-alpha from Github releases.
edit: I also attempted all cases with and without
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
. At this point I'm trying all combinations of env variations, as I'm terribly confuse wether I follow the quickstart or the "beyla add connection guide".The text was updated successfully, but these errors were encountered: