dotnet pack version without build: Could not load file or assembly

I am getting the following error when setting the version using dotnet pack instead on dotnet build when 'overriding' the package

System.IO.FileLoadException: Could not load file or assembly 'My.Package, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040) File name: 'My.Package, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'

Let's say I have the following 'dependecy graph'

• My.Package 1.0.0.1
• My.OtherPackage 1.0.0.1
  • NewtonSoft.Json 12.0.1
  • My.Package 1.0.0.0 <---- notice that we override this version

When I build / pack as follows, it works

dotnet build MySolution.sln --version-suffix 1.0.0.1 -c Release
dotnet pack My.Package/My.Package.csproj --no-build /p:Version=$VERSION=1.0.0.1 -c Release

However (as I am using docker) I would like to avoid putting the version at build time and rather only at pack time

dotnet build MySolution.sln -c Release
dotnet pack My.Package/My.Package.csproj --no-build /p:Version=$VERSION=1.0.0.1

This also works, as long as I do not have to 'override' the package version in the dependency graph.

1. Why does it not work ? I compared the files generated in the .nupkg file and I could not see a difference
2. Can I make it work without having to specify the version at build time ?