Creating a bitfield class that points to arbitrary data- is this safe?

Context

I am creating a Bitfield class that is responsible for providing access to a contiguous set of bits in a UInt32. The source data is not managed by the Bitfield, but instead another object. In practice, the same object that owns the source data will also own any Bitfield instances that point to it, so the pointer lifetime will never exceed that of the source data. All parameters passed to the Bitfield constructor are determined at runtime. My current approach is as follows:

public class Bitfield
{
    private int offset;
    private uint mask;
    unsafe private uint* data;

    unsafe public Bitfield(uint* data, int msb, int lsb)
    {
        this.data = data;
        mask = (uint)(((1UL << (msb + 1)) - 1) ^ ((1UL << lsb) - 1));
        offset = lsb;
    }
    unsafe public void Set(uint value) => *data = ((value << offset) & mask) | (*data & ~mask);
    unsafe public uint Get() => (*data & mask) >> offset;
}

In the using application, a Bitfield might be employed as below:

class Program
{
    static uint sourceData = 0xDEADBEEF;

    unsafe static void Main(string[] args)
    {
        Bitfield high;
        Bitfield low;

        fixed (uint* data = &sourceData)
        {
            high = new Bitfield(data, 31, 16);
            low = new Bitfield(data, 15, 0);
        }

        Console.WriteLine($"DEAD ?= {high.Get():X}");
        Console.WriteLine($"BEEF ?= {low.Get():X}");
        high.Set(0xFEED);
        Console.WriteLine($"FEEDBEEF ?= {sourceData:X}");

        Console.ReadKey();
    }
}

Main question

Is this a sound approach or should I seek a different strategy?

Other considerations

I read that the Garbage Collector may rearrange memory, hence the fixed body. When this happens, I worry that the pointer will not be updated to match sourceData's new location. Therefore, high and low will operate on invalid data, rendering my approach dangerous. Can someone confirm this? I could pass the source data to the Get/Set methods with ref and achieve the same result as the pointer, but then the caller must keep track of which source data to pass to which Bitfields (This will vary by owning object at runtime).

Side questions:

Is there perhaps a Reflection construct that would work similarly? (I don't know much about Reflection.)

Why does the Garbage Collector rearrange memory? Is it to combat fragmentation?

I agree that the approach is not sound, and can lead to data corruption and/or invalid memory access. It seems that my dream of operating on arbitrary data would require some feature that ensures the referenced memory can't be moved without a pointer to it noticing. Unfortunately, neither BitVector32 nor BitArray quite meet this original intent either, but they did start me thinking.

Upon reviewing the use cases, I have settled on a solution by which I refer to the source data. Like the other suggestions, it doesn't meet the original intent. However, the source data will always be an IList<uint>, and never a lone value. With this in mind, I created a class that encapsulates the source data and manages a dictionary of field descriptors. It will then provide access to both individual fields and to the source data array. In this way, the fields remain flexible and can point to different data at runtime using an reference and index instead of a pointer.

// This is now just a descriptor of the field parameters
public struct Bitfield
{
    public int index;
    public int offset;
    public uint mask;

    public Bitfield(int index, int msb, int lsb)
    {
        this.index = index;
        mask = (uint)(((1UL << (msb + 1)) - 1) ^ ((1UL << lsb) - 1));
        offset = lsb;
    }
}

// The data is now encapsulated in its own class
public class DataArray
{
    public IList<uint> data;

    private Dictionary<string, Bitfield> fields = new Dictionary<string, Bitfield>();

    public DataArray(IList<uint> sourceData)
    {
        // I don't care if this is a copy or reference assignment as long as
        // I use DataArray.data to access the array from now on
        data = sourceData;
    }
    public void AddField(string name, int index, int msb, int lsb)
    {
        fields[name] = new Bitfield(index, msb, lsb);
    }
    public uint Get(string name)
    {
        uint result = 0;
        if(fields.TryGetValue(name, out Bitfield field))
        {
            result = (data[field.index] & field.mask) >> field.offset;
        }
        else
        {
            // throw invalid name
        }
        return result;
    }
    public void Set(string name, uint value)
    {
        if(fields.TryGetValue(name, out Bitfield field))
        {
            data[field.index] = ((value << field.offset) & field.mask) | (data[field.index] & ~field.mask);
        }
        else
        {
            // throw invalid name
        }
    }
}