w32tm not syncing my DC's time like I would expect

I have an issue where my domain controller (a HyperV VM) is not syncing the time with the external time server as expected. The status shows it has recent syncs going on but the time still never updates correctly. However, if I update it manually running w32tm it updates the time perfectly. The at some point down the line it gets out of sync again.

Note: I have had my VM server set to NOT sync the host with the system time. I have just recently set up the VM server to update its time using my DC's time server.

Any ideas what I'm doing wrong? Anything I can check etc?

My config from w32tm...

[Configuration]

EventLogFlags: 2 (Local)
AnnounceFlags: 5 (Local)
TimeJumpAuditOffset: 28800 (Local)
MinPollInterval: 6 (Local)
MaxPollInterval: 10 (Local)
MaxNegPhaseCorrection: 172800 (Local)
MaxPosPhaseCorrection: 172800 (Local)
MaxAllowedPhaseOffset: 300 (Local)

FrequencyCorrectRate: 4 (Local)
PollAdjustFactor: 5 (Local)
LargePhaseOffset: 50000000 (Local)
SpikeWatchPeriod: 900 (Local)
LocalClockDispersion: 10 (Local)
HoldPeriod: 5 (Local)
PhaseCorrectRate: 7 (Local)
UpdateInterval: 100 (Local)


[TimeProviders]

NtpClient (Local)
DllName: C:\Windows\system32\w32time.DLL (Local)
Enabled: 1 (Local)
InputProvider: 1 (Local)
AllowNonstandardModeCombinations: 1 (Local)
ResolvePeerBackoffMinutes: 15 (Policy)
ResolvePeerBackoffMaxTimes: 7 (Policy)
CompatibilityFlags: 2147483648 (Local)
EventLogFlags: 0 (Policy)
LargeSampleSkew: 3 (Local)
SpecialPollInterval: 3600 (Policy)
Type: NTP (Policy)
NtpServer: 0.north-america.pool.ntp.org,0x9 1.north-america.pool.ntp.org,0x9 2.north-america.pool.ntp.org,0x9 3.north-america.pool.ntp.org,0x9 (Policy)

NtpServer (Local)
DllName: C:\Windows\system32\w32time.DLL (Local)
Enabled: 1 (Local)
InputProvider: 0 (Local)
AllowNonstandardModeCombinations: 1 (Local)

VMICTimeProvider (Local)
DllName: C:\Windows\System32\vmictimeprovider.dll (Local)
Enabled: 1 (Local)
InputProvider: 1 (Local)

My status from w32tm...

Leap Indicator: 3(last minute has 61 seconds)
Stratum: 2 (secondary reference - syncd by (S)NTP)
Precision: -6 (15.625ms per tick)
Root Delay: 0.0468690s
Root Dispersion: 0.0292386s
ReferenceId: 0xCE6C0083 (source IP:  206.108.0.131)
Last Successful Sync Time: 2/11/2021 11:01:09 AM
Source: 3.north-america.pool.ntp.org,0x9
Poll Interval: 6 (64s)

A small test I ran on the sync server...

>w32tm /stripchart /computer:0.north-america.pool.ntp.org /dataonly /samples:5
Tracking 0.north-america.pool.ntp.org [23.131.160.7:123].
Collecting 5 samples.
The current time is 2/11/2021 11:55:13 AM.
11:55:13, +876.3220145s
11:55:15, error: 0x800705B4
11:55:18, error: 0x800705B4
11:55:21, +876.2797538s
11:55:23, error: 0x800705B4

Here is what happens when I manually update the time...

Results: Changes system time from 12:09PM to 12:23PM.

>w32tm /resync
Sending resync command to local computer
The computer did not resync because no time data was available.