Using Sustainsys.Saml2 in Sitecore Identity Plugin fails to find 'Microsoft.IdentityModel.Tokens.Saml, Version=5.2.4.0'

I'm trying to implement a SAML IdentityProvider plugin for Sitecore Identity Server. My project settings are:

I have referenced the following packages in my solution:

I have configured the Sustainsys.Saml2.IdentityProvider to work with our Idp and hooked it up using:

new Microsoft.AspNetCore.Authentication.AuthenticationBuilder(services).AddSaml2(...)

When I copy the build output .dll and configuration to my IdentityServer instance I can see the following error in the log when the server starts up:

540 System.IO.FileLoadException: Could not load file or assembly 
'Microsoft.IdentityModel.Tokens.Saml, Version=5.2.4.0, Culture=neutral, 
PublicKeyToken=31bf3856ad364e35'. Could not find or load a specific file. (Exception from 
HRESULT: 0x80131621)
541 File name: 'Microsoft.IdentityModel.Tokens.Saml, Version=5.2.4.0, Culture=neutral, 
PublicKeyToken=31bf3856ad364e35'
542    at Sustainsys.Saml2.Configuration.SPOptions.set_EntityId(EntityId value)
543    at XYZ.IdentityProvider.Saml2.ConfigureSitecore. 
<ConfigureServices>b__3_0(Saml2Options options)
544    at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
545    at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
546    at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean 
useDefaultConstructor)
547    at System.Lazy`1.CreateValue()
...

My runtime on the server is:

When I check the runtime folder all Microsoft.IdentityModel.* assemblies are version 5.2.0.

How can I get this plugin to run, even though the Sustainsys package references a newer version of the Microsoft.Identity.* assemblies?