Error: Attempted duplicate registration of API: context & propagation

[pieterjandebruyne]

• Package Name:
  @azure/monitor-opentelemetry
• Package Version:
  1.8.1
• Operating system:
  docker [- linux] node:20.9.0-alpine -> running in a container app
• nodejs
  20.9
• typescript
  5.7.2

Describe the bug
Error: @opentelemetry/api: Attempted duplicate registration of API: context
Error: @opentelemetry/api: Attempted duplicate registration of API: propagation

To Reproduce
Steps to reproduce the behavior:

import {
  type AzureMonitorOpenTelemetryOptions,
  useAzureMonitor,
} from '@azure/monitor-opentelemetry';
import { ManagedIdentityCredential } from '@azure/identity';

const isLocalEnv = process.env.ENVIRONMENT === 'local';

export const setupOpenTelemetry = () => {
  if (isLocalEnv) {
    return;
  }
  // Create a new ManagedIdentityCredential object.
  const credential = new ManagedIdentityCredential({
    clientId: process.env.AZURE_MANAGED_IDENTITY_ID,
  });

  const telemetryOptions: AzureMonitorOpenTelemetryOptions = {
    azureMonitorExporterOptions: {
      connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING,
      credential,
    },
    enableLiveMetrics: true,
    enableStandardMetrics: true,
    instrumentationOptions: {
      http: { enabled: true },
      azureSdk: { enabled: true },
      postgreSql: { enabled: true },
      redis: { enabled: true },
    },
  };
  useAzureMonitor(telemetryOptions);
};

Expected behavior
No errors

Additional context

First I get

@azure/opentelemetry-instrumentation-azure-sdk [
'Module @azure/core-tracing has been loaded before @azure/opentelemetry-instrumentation-azure-sdk so it might not work, please initialize it before requiring @azure/core-tracing'
]

But googling this error it seems like I can ignore it.

Then I get Errors:
@opentelemetry/api: Attempted duplicate registration of API: context
@opentelemetry/api: Attempted duplicate registration of API: propagation

I followed the documentation, tried without instrumentationOptions, also tried with additional registerInstrumentations but nothing seems to resolve these errors.

I also later get this error:

LongIntervalStatsbeat: metrics export failed (error RestError: {"itemsReceived":3,"itemsAccepted":0,"appId":null,"errors":[{"index":0,"statusCode":439,"message":"Daily quota exceeded"},{"index":1,"statusCode":439,"message":"Daily quota exceeded"},{"index":2,"statusCode":439,"message":"Daily quota exceeded"}]}) []

but I am unable to find any information about daily quota's, neither have I ever set one up.

[JacksonWeber]

@pieterjandebruyne These errors typically occur when the OpenTelemetry SDK has been initialized multiple times. Please make sure that you're not initializing multiple times. It would also help to see what your package.json file looks like.

[pieterjandebruyne]

Hi @JacksonWeber,

Thanks for looking into this. I don't think I am initialising it more then once, I only get the setup log 1 time at least and the error triggers from that first run.
I included the files that might be of interest. Let me know if you need anything else.

Thanks already.

// index.ts

/* eslint-disable simple-import-sort/imports */
import 'dotenv/config';
import './instrument';
import 'reflect-metadata';
/* eslint-enable simple-import-sort/imports */
import * as database from 'tf-database';
import * as Sentry from '@sentry/node';
import gracefulShutdown from 'http-graceful-shutdown';
import { logger } from 'tf-logger';
import { cronjobHandler } from './cronjob';
import { redisConnection } from 'tf-redis-instance';
import { addMiddlewares, getServers } from './server';
import { errorToObject } from 'tf-shared-utils';
import compression from 'compression';

const prisma = database.prismaClient;

const shutdownServices = async () => {
  await cronjobHandler.shutdownJobSheduler();
  await prisma.$disconnect();
  redisConnection.disconnect();
};

const main = async () => {
  const { app, apolloServer, httpServer } = await getServers();

  app.use(compression());

  await apolloServer.start();
  await new Promise<void>((resolve) => {
    httpServer.listen({ port: 4000 }, resolve);
  });

  addMiddlewares({ app, apolloServer, prisma });

  logger.info(`🚀 Server ready!`);

  const onShutdown = async () => {
    await shutdownServices();
    process.exit(0);
  };

  gracefulShutdown(httpServer, { onShutdown });
};

main()
  .then(async () => {
    await prisma.$disconnect();
  })
  .catch((error: unknown) => {
    logger.error({ error: errorToObject(error) }, `Server shutdown`);

    Sentry.captureException(error);
    setTimeout(async () => {
      await shutdownServices();
      process.exit(1);
    }, 3000);
  });

// instruments.ts

import { setupOpenTelemetry } from './setupOpenTelemetry';

setupOpenTelemetry();

// setupOpenTelemetry.ts

import {
  type AzureMonitorOpenTelemetryOptions,
  useAzureMonitor,
} from '@azure/monitor-opentelemetry';
import { ManagedIdentityCredential } from '@azure/identity';

const isLocalEnv = process.env.ENVIRONMENT === 'local';

export const setupOpenTelemetry = () => {
  console.log('setupOpenTelemetry')
  if (isLocalEnv) {
    return;
  }
  // Create a new ManagedIdentityCredential object.
  const credential = new ManagedIdentityCredential({
    clientId: process.env.AZURE_MANAGED_IDENTITY_ID,
  });

  const telemetryOptions: AzureMonitorOpenTelemetryOptions = {
    azureMonitorExporterOptions: {
      connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING,
      credential,
    },
    enableLiveMetrics: true,
    enableStandardMetrics: true,
    instrumentationOptions: {
      http: { enabled: true },
      azureSdk: { enabled: true },
      postgreSql: { enabled: true },
      redis: { enabled: true },
    },
  };
  useAzureMonitor(telemetryOptions);
};

//package.json

{
  "name": "backend",
  "packageManager": "[email protected]",
  "version": "0.0.0",
  "private": true,
  "jshintConfig": {
    "esversion": 6
  },
  "scripts": {
    "build": "tsc",
    "dev": "npm run watch",
    "watch": "nodemon",
  },
  "dependencies": {
    "@apollo/server": "^4.11.2",
    "@sentry/node": "^8.40.0",
    "axios": "^1.7.9",
    "bluebird": "^3.7.2",
    "body-parser": "^1.20.3",
    "class-validator": "^0.14.1",
    "compression": "^1.7.5",
    "cors": "^2.8.5",
    "cron": "^3.2.1",
    "dataloader": "^2.2.2",
    "dotenv": "^16.4.5",
    "express": "^4.21.1",
    "graphql": "^16.9.0",
    "graphql-middleware": "^6.1.35",
    "graphql-scalars": "^1.23.0",
    "graphql-tag": "^2.12.6",
    "graphql-type-json": "^0.3.2",
    "http-graceful-shutdown": "^3.1.13",
    "json-bigint-patch": "^0.0.8",
    "jsonwebtoken": "^9.0.2",
    "jwk-to-pem": "^2.0.7",
    "lodash": "^4.17.21",
    "luxon": "^3.5.0",
    "probe-image-size": "^7.2.3",
    "redis": "^4.7.0",
    "reflect-metadata": "^0.2.2",
    "shared-utils": "*",
    "typescript": "^5.7.2",
    "type-graphql": "2.0.0-rc.2",
    "uuid": "^11.0.3",
    "winston": "^3.17.0"
  },
  "devDependencies": {
    "@sentry/types": "^8.40.0",
    "@types/bluebird": "^3.5.42",
    "@types/compression": "^1.7.5",
    "@types/cors": "^2.8.17",
    "@types/express": "^4.17.21",
    "@types/jest": "^29.5.14",
    "@types/jsonwebtoken": "^9.0.7",
    "@types/jwk-to-pem": "^2.0.3",
    "@types/lodash": "^4.17.13",
    "@types/node": "20.9.5",
    "@types/uuid": "^10.0.0",
    "fs": "^0.0.2",
    "ioredis": "^5.4.1",
    "jest": "^29.7.0",
    "node-ts": "^6.1.3",
    "nodemon": "^3.1.7",
    "ts-jest": "^29.2.5",
    "ts-node": "^10.9.2",
    "tsconfig": "*"
  },
  "engines": {
    "node": "20.9.0",
    "npm": "^10.1.0"
  }
}

// tsconfig.json

{
  "extends": "tsconfig/node.json",
  "main": "src/index.ts",
  "include": ["."],
  "exclude": ["node_modules", "build", "codegen.ts"],

  "compilerOptions": {
   "declaration": true,
    "rootDir": "./",
    "baseUrl": "./src",
    "sourceRoot": "/",
    "outDir": "./build",
    "lib": [
      "ESNext"
    ],
    "module": "CommonJS",
    "target": "ES2015",
    "types": ["node", "jest"],
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "paths": {
      "*": ["node_modules/*", "src/*"]
    },
    "typeRoots": [
      "src/@types",
      "node_modules/@types",
      "../../node_modules/@types"
    ]
  }
}

[pieterjandebruyne]

As a note to the additional packages you see in package.json

    "@opentelemetry/api": "^1.9.0",
    "@opentelemetry/instrumentation": "^0.57.1",
    "@opentelemetry/instrumentation-http": "^0.57.1",
    "@opentelemetry/sdk-trace-node": "^1.30.1",

I had these added before because I wanted to add additional isntrumentation tools:
My ideal setup in the end would be this: but I am not sure if I'm using conflicting libaries here.. I think azure is trying to push towards their opentelemetry distro?

/* eslint-disable no-process-env */
// docs: https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-add-modify?tabs=nodejs
// https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-help-support-feedback?tabs=nodejs#why-should-i-use-the-azure-monitor-opentelemetry-distro
// available packages: https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/plugins/node

import {
  type AzureMonitorOpenTelemetryOptions,
  useAzureMonitor,
} from '@azure/monitor-opentelemetry';
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { ManagedIdentityCredential } from '@azure/identity';
import { DataloaderInstrumentation } from '@opentelemetry/instrumentation-dataloader';
import { GraphQLInstrumentation } from '@opentelemetry/instrumentation-graphql';
import { IORedisInstrumentation } from '@opentelemetry/instrumentation-ioredis';
import { PgInstrumentation } from '@opentelemetry/instrumentation-pg';
import { logger } from 'tf-logger';
import { registerInstrumentations } from '@opentelemetry/instrumentation';
import { metrics } from '@opentelemetry/api';
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http';
import { ExpressInstrumentation } from '@opentelemetry/instrumentation-express';

const isLocalEnv = process.env.ENVIRONMENT === 'local';

export const setupOpenTelemetry = () => {
  console.log('setupOpenTelemetry');
  if (isLocalEnv) {
    return;
  }
  console.log('setupOpenTelemetry-log');
  logger.info('setupOpenTelemetry');

  // Create a new ManagedIdentityCredential object.
  const credential = new ManagedIdentityCredential({
    clientId: process.env.AZURE_MANAGED_IDENTITY_ID,
  });

  const telemetryOptions: AzureMonitorOpenTelemetryOptions = {
    azureMonitorExporterOptions: {
      connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING,
      credential,
    },
    enableLiveMetrics: true,
    enableStandardMetrics: true,
    instrumentationOptions: {
      http: { enabled: true },
      azureSdk: { enabled: true },
      postgreSql: { enabled: true },
      redis: { enabled: true },
    },
  };
  useAzureMonitor(telemetryOptions);

  const tracerProvider = new NodeTracerProvider();
  tracerProvider.register();
  const meterProvider = metrics.getMeterProvider();
  registerInstrumentations({
    instrumentations: [
      new HttpInstrumentation(), // http
      new ExpressInstrumentation(), // Express
      new DataloaderInstrumentation(), // dataloader
      new GraphQLInstrumentation({
        // graphql
        mergeItems: true,
        depth: -1,
        allowValues: true,
      }),
      new IORedisInstrumentation(), // redis
      new PgInstrumentation({
        // postgres
        enhancedDatabaseReporting: true,
      }),
    ],
    tracerProvider,
    meterProvider,
  });
  logger.info('setup telemetry finished');
};

I stripped the instrumentations part before to have a more minimal setup to test first, but that was already giving me errors.

[JacksonWeber]

@pieterjandebruyne Ah, your issue is overriding the global tracer provider when you're doing:

const tracerProvider = new NodeTracerProvider();
tracerProvider.register();

You shouldn't need to register a global tracerProvider here, or a meterProvider as the @azure/monitor-opentelemetry package will do that for you. If you take that out your code should stop throwing those warnings. Let me know if that works as expected for you.

[pieterjandebruyne]

@JacksonWeber I already stripped that part off my code for initial testing, but it kept throwing those errors. As you can see in my original post, I did not include any other code than useAzureMonitor(telemetryOptions); and it was still giving me the duplicate registration errors and also daily quota error (I can not find any information about this quota either, and I never set any limits myself)

In regards to the tracerProvider, isn't it the same code that is currently in the documentation, should that also be updated?
https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-add-modify?tabs=nodejs

[JacksonWeber]

@pieterjandebruyne The 439 Daily Quota error appears to be a possible issue on the ingestion side. It should be transitory and is considered retriable so you should not lose any telemetry. If you do experience telemetry loss, please let me know.

The code in the example you provided is getting a delegate tracer provider not creating a new tracer provider.

Just so I can understand the issue better, can you please provide me the minimum relevant pieces of code? I see your original example but I only ever get the warning you've mentioned.