App authentication woes on SharePoint (Token request failed. The remote server returned an error: (401) Unauthorized.)

This article explains how to get rid of sudden and unexplainable 401 Access Denied errors when trying to authenticate against a fairly fresh Microsoft 365 / SharePoint Online tenant. This issue seems to be caused by a long-ish project to finally retire ACS - Azure Access Control service) on SharePoint (it's retired everywhere else already!)

Note: This might still be an updating story, as the situation with ACS is definitely... Developing. Yeah, let's call it that. It's a developing situation.

Problem

In the beginning of September (2020), a lot of threads, chats and tickets started popping up about apps and scripts suddenly failing to authenticate against SharePoint Online. It seemed somewhat widespread and just worsening.

One way to reproduce this issue is to use app-only authentication in OfficeDevPnP's AuthenticationManager:

var authman = new OfficeDevPnP.Core.AuthenticationManager();
using (Microsoft.SharePoint.Client.ClientContext ctx = authman .GetAppOnlyAuthenticatedContext(appUrl, clientId, clientSecret))
{
  Web web = ctx.Web;
  context.Load(web, w => w.Id, w => w.Title);
  context.ExecuteQueryRetry();
}

The very low-key and non-descriptive error was simply:

"The remote server returned an error: (401) Unauthorized."

Sometimes with an additional insult to injury:

{"error":"invalid_request","error_description":"Token type is not allowed."}

The weirdest part? The same code or script might work just fine for other tenants. The only difference should be the age of the tenant you're authenticating against - it should be provisioned during or after August 2020.

Reason

This has been a long time coming I suppose, but Microsoft is pushing users away from ACS and using Client Id & Client Secret -combo to authenticate against Microsoft 365. Some time during 2020, Microsoft added a new tenant-level property called "DisableCustomAppAuthentication" to SharePoint Online. This property was first surfaced in the August 2020 release of SharePoint Online CSOM. This release is available as a NuGet package with version 16.1.20412.12000.

This property should be pretty useful for a gradual move away from ACS - your organization can approach whatever deadline Microsoft sets for disabling ACS completely by "soft-disabling" it first with this property, and seeing if something breaks. Furthermore, disabling ACS is definitely a security improvement, as leaking a client id and client secret might've lead to anyone being able to access the tenant from anyone with zero oversight and very bad governance in general.

However, this is soiled pretty badly by the weird rollout. Unfortunately, someone at Microsoft made the decision to set this property to be true by default, and it'll affect any tenants provisioned after sometime late August, 2020.

That'll break a lot of custom functionality like apps or PowerShell scripts that work on any older tenants.

And obviously, the whole security aspect is also completely destroyed by the fact that since you'll now have to use Azure AD, which doesn't contain granular, site-level permissions for app authentication. At all. So any app registration either gets everything, or nothing.

With ACS, you could assign permissions on a per- site collection level.

So much for security improvements!

But at least good folks at Microsoft seem to be aware of the issues with this default setting:

Solution

Well, essentially, you've got 2 options. Let me explain them below!

Time needed: 15 minutes.

How to fix "401 Unauthorized" when using app authentication on a SharePoint tenant that was provisioned after August 2020?

1. Move away from the old, app-only authentication using Client Id and Client Secret

   This would be the better way forward - for an application authentication scenarios, you'd need to register your app in Azure Active Directory, but in that case you can't manage permissions granularly, at all.

2. Enforce TLS 1.2

   There are a few ways to do this - you might be able to do something like this in the code:
   const System.Net.SecurityProtocolType Tls12 = (System.Net.SecurityProtocolType)((System.Security.Authentication.SslProtocols)0x00000C00); System.Net.ServicePointManager.SecurityProtocol = Tls12;

   Or you might need to simply update to .NET Framework 4.7.2, if possible (and if you're still on the Framework-train instead of the Core mainstream .NET one).

   Alternatively, you might need to apply some interesting configuration tricks - I explain them closer here:
   How to force an outdated .NET project to use TLS 1.2?

3. Set the property DisableCustomAppAuthentication to false.

   You can also enable custom app authentication by disabling the tenant property "DisableCustomAppAuthentication".

   "Disable disable" => enable. Double negative and all that.

   You'll need to have at least SharePoint Administrator permissions to run this.

   First of all, update your SharePoint Online PowerShell module to the latest version. After that, authenticate, and then run this below:

   Set-SPOTenant -DisableCustomAppAuthentication $false

   Or alternatively, you can run this PnP commandlet:

   Set-PnPTenant -DisableCustomAppAuthentication $false

   Either of these will enable you to still use the ACS for the time being. Remember to hatch a plan of some kind for the future when it won't be available anymore!

   Oh - the change won't be instant. It might take up to 24 hours to actually update on your tenant (thanks, timer job of some sort).

... aaand you should be done! Until it breaks again.

References

• Credit where credit is due:

  • My colleague Jouni Pohjolainen brought this one to my attention (wish he had a blog, would make it easier to attribute stuff like this! 😄)

  • Gautam, another one of my colleagues (is there a pattern here?) pointed out I was missing the (newly available) switch PnP commandlets have to enable ACS - thanks!

• Other people having this issue:

  • SharePoint App-Only Add-ins throwing 401 Unauthorized on newly created O365 tenants

    • Thread on Microsoft Q&A

  • SharePoint Online authorization issue 'Token type is not allowed'

    • https://sharepoint.stackexchange.com/questions/284402/sharepoint-online-authorization-issue-token-type-is-not-allowed/284734

  • People enjoying the (well, partially) same issue on GitHub:

    • https://github.com/SharePoint/sp-dev-docs/issues/6903

Also worthwhile to see this Twitter conversation by my colleague Vardhaman:

So interesting new development for SharePoint App-Only permissions. The Tenant.DisableCustomAppAuthentication property introduced in latest version of CSOM can be used to disable granting permissions through appregnew.aspx/appinv.aspx https://t.co/Rog1ZhhmUT

— Vardhaman Deshpande (@vrdmn) September 21, 2020