"No authentication handlers are registered" but there is one

[jeffputz]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

The CI build of POP Forums seems to have an issue where over time, the registered auth handler becomes "deregistered" until I cycle the Azure App Service. I can't reproduce this in other instances I have running, but it happens frequently on this one. Everything runs as expected, until it doesn't.

The Program of the app calls an extension method which registers the auth:

		services.AddAuthentication()
			.AddCookie(PopForumsAuthorizationDefaults.AuthenticationScheme, option =>
			{
				option.ExpireTimeSpan = new TimeSpan(365, 0, 0, 0);
				option.LoginPath = "/Forums/Account/Login";
				// TODO: This is lame because of fx, see: https://github.com/dotnet/aspnetcore/issues/9039
				option.Events.OnRedirectToAccessDenied = context =>
				{
					context.Response.StatusCode = 403;
					return Task.CompletedTask;
				};
			});

Another extension method registers middleware...

app.UseMiddleware<PopForumsAuthorizationMiddleware>();

And here's the middleware. The exception is thrown on the context.AuthenticateAsync call:

	public async Task InvokeAsync(HttpContext context, IUserService userService, IProfileService profileService, ISetupService setupService, IConfig config, IOAuthOnlyService oAuthOnlyService)
	{
		var isSetupAndConnectionGood = setupService.IsRuntimeConnectionAndSetupGood();
		if (!isSetupAndConnectionGood)
		{
			await _next.Invoke(context);
			return;
		}
		var authResult = await context.AuthenticateAsync(PopForumsAuthorizationDefaults.AuthenticationScheme);
		if (authResult.Principal?.Identity is ClaimsIdentity identity)
		{
			var user = await userService.GetUserByName(identity.Name);
			if (user != null)
			{
				foreach (var role in user.Roles)
					identity.AddClaim(new Claim(PopForumsAuthorizationDefaults.ForumsClaimType, role));
				identity.AddClaim(new Claim(PopForumsAuthorizationDefaults.ForumsUserIDType, user.UserID.ToString()));
				context.Items["PopForumsUser"] = user;
				var profile = await profileService.GetProfile(user);
				context.Items["PopForumsProfile"] = profile;
				context.User = new ClaimsPrincipal(identity);
				if (config.IsOAuthOnly && user.TokenExpiration < DateTime.UtcNow)
				{
					var isSuccess = await oAuthOnlyService.AttemptTokenRefresh(user);
					if (!isSuccess)
						context.Response.Redirect("/Forums/Account/Login");
				}
			}
		}
		await _next.Invoke(context);
	}

Exception:

InvalidOperationException: No authentication handlers are registered. Did you forget to call AddAuthentication().AddSomeAuthHandler?

at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme)
at PopForums.Mvc.Areas.Forums.Authorization.PopForumsAuthorizationMiddleware.InvokeAsync(HttpContext context, IUserService userService, IProfileService profileService, ISetupService setupService, IConfig config, IOAuthOnlyService oAuthOnlyService) in /home/vsts/work/1/s/src/PopForums.Mvc/Areas/Forums/Authorization/PopForumsAuthorizationMiddleware.cs:line 20
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

This code has been mostly unchanged going back to .NET v5. Again, when the app starts, everything is fine, but when I come back to it after a few days, it's returning 500's with the above exception and stack trace. I can't think of any reason that after the app is started, the auth scheme would no longer be in place.

Expected Behavior

The auth scheme is durably registered.

Steps To Reproduce

• Run the app in a Linux Azure App Service.
• Observe that it runs fine.
• Wait some amount of time.
• The above exceptions start happening.
• Restart the app service.
• Normal operation resumes.

Exceptions (if any)

InvalidOperationException: No authentication handlers are registered. Did you forget to call AddAuthentication().AddSomeAuthHandler?

at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme)
at PopForums.Mvc.Areas.Forums.Authorization.PopForumsAuthorizationMiddleware.InvokeAsync(HttpContext context, IUserService userService, IProfileService profileService, ISetupService setupService, IConfig config, IOAuthOnlyService oAuthOnlyService) in /home/vsts/work/1/s/src/PopForums.Mvc/Areas/Forums/Authorization/PopForumsAuthorizationMiddleware.cs:line 20
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

.NET Version

8.0

Anything else?

No response

[halter73]

Unfortunately, these repro steps are not sufficient to demonstrate that there is a bug in the framework and not in the app. If you can provide a minimal repro that demonstrates just this issue and a specific trigger for the InvalidOperationException more specific than "wait some amount of time," that points to this being a framework issue, we'll look into this further.

[jeffputz]

I suspected that this would be the response, which is why I almost didn't submit this. I can't repro it, I can only show you that it happens. The auth code has always been complex, which is why I also didn't look at the source, because I wouldn't know where to start. From what I described, you could logically concluded that a minimal repo wouldn't be useful. That repo would be the code samples in the docs, which would then have to run for some amount of time in an Azure Linux App Service before it broke. That I can't do this does not mean that there's no problem.

My assumption is that the place where the registration of handlers is stored is actually volatile after startup. So why is that?

[halter73]

My assumption is that the place where the registration of handlers is stored is actually volatile after startup. So why is that?

I'm not sure why we decided to make it possible to mutate authentication schemes after app start. That decision to allow for that much flexibility was before my time, but my guess is that it was support very dynamic multitenant auth scenarios that might be desired by framework or CMS authors. I haven't seen it used outside of our tests and samples like this one though.

aspnetcore/src/Security/samples/DynamicSchemes/Controllers/AuthController.cs

Lines 23 to 28 in 8efe5b0

	public IActionResult Remove(string scheme)
	{
	_schemeProvider.RemoveScheme(scheme);
	_optionsCache.TryRemove(scheme);
	return Redirect("/");
	}

Assuming that the issue really is the authentication scheme being removed after application start, and not the application restarting without registering any schemes, you should be able to see it by registering a custom IAuthenticationSchemeProvider that logs the calls to add and remove schemes:

builder.Services.AddSingleton<IAuthenticationSchemeProvider, LoggingAuthSchemeProvider>();

// ...

public class LoggingAuthSchemeProvider(IOptions<AuthenticationOptions> options, ILogger<LoggingAuthSchemeProvider> logger)
    : AuthenticationSchemeProvider(options)
{
    public override bool TryAddScheme(AuthenticationScheme scheme)
    {
        logger.LogInformation("Adding scheme '{SchemeName}'.", scheme.Name);
        return base.TryAddScheme(scheme);
    }

    public override void RemoveScheme(string name)
    {
        logger.LogCritical("Removing scheme '{SchemeName}'!!!{StackTrace}", name, Environment.StackTrace);
        base.RemoveScheme(name);
    }
}

[jeffputz]

Finally getting around to adding this logging... I can see the registration on startup being logged (on the Azure Linux app service at https://popforumsdev.azurewebsites.net/forums/). About five or six minutes later, the health check agent pings the service, and I see this logged when it hits the above middleware:
InvalidOperationException: No authentication handlers are registered. Did you forget to call AddAuthentication().Add[SomeAuthHandler]("PopForumsAuthScheme",...)?

I doubt it matters (unless there's some weird caching thing going on), but at first the only headers (I'm adding the IP) I record are:

IP: ::ffff:169.254.134.1
Accept: */*
Host: 169.254.134.2:8080
User-Agent: HealthCheck/1.0

Five seconds after that I get more headers:

IP: ::ffff:169.254.134.1
Host: popforumsdev.azurewebsites.net
User-Agent: ReadyForRequest/1.0 (LocalCache)
disguised-host: popforumsdev.azurewebsites.net
X-ARR-SSL: 2048|256|C=US, S=Washington, L=Redmond, O=Microsoft Corporation, OU=Microsoft IT, CN=Microsoft IT TLS CA 5|CN=popforumsdev.azurewebsites.net
x-ms-request-id: 9a8dfbb4-e574-460d-982d-ca1893135c35
X-LocalCache-Ready-Check: 1
WAS-USE-PLACEHOLDER: 1
X-Original-For: [::ffff:169.254.134.1]:58665
X-Original-Proto: http

I can confirm no app restarts, no add/remove scheme hits, and when I go to the app it runs as expected. I only bring this up because I've seen things behave differently in Linux versus Windows. I'm running two instances, and I wonder if that has something to do with it.

[atbez-miview]

Not relevant to your problem, but to clarify on the 2 messages you posted. It's not "more headers" per se, it is different messages coming from different parts of the service. The User-Agent header it the important distinction there. The first one (HealthCheck/1.0) is because have HealthCheck feature turned on. The second (ReadyForRequest/1.0) is one of many that come from the app service host to confirm your app is running. They could arrive in either order. HealthCheck should be periodic. ReadyForRequest usually only shows up when an instance is restarted/scaled.

[jeffputz]

OK, the thinking out loud helped... With the logging in place, I realized that it should be logging the added scheme twice, once for each instance. It was not, so it stood to reason that only one instance was broken. Sure enough, I looked up the instances with az webapp list-instances and changed the ARRAffinity cookie to look at the other instance. Sure enough, No authentication handlers are registered. Did you forget... every time.

I'm not sure then if this is a fx problem or Azure. I have several sites running on the same app plan, and I can switch instances on them every time. Would not using AddDataProtection on startup cause this? If so, why would it only break one instance?

EDIT: I don't think it's the data protection. Even with it in place, the second instance never registers the auth handler.

[jeffputz]

While I can't explain why this was an issue on just one of two running nodes, it does seem to be related to the order of middleware registration in the app. I changed it, and the problem went away. Also hard to say why it started happening specifically in v8.