Concurrent Read/Write MySQL EF Core

Using EF Core 2.2.6 and Pomelo.EntityFrameworkCore.MySql 2.2.6 (with MySqlConnector 0.59.2)). I have a model for UserData:

public class UserData
{
    [DatabaseGenerated(DatabaseGeneratedOption.None)]
    public ulong ID { get; private set; }

    [Required]
    public Dictionary<string, InventoryItem> Inventory { get; set; }

    public UserData()
    {
        Data = new Dictionary<string, string>();
    }
}

I have a REST method that can be called that will add items to the user inventory:

using (var transaction = context.Database.BeginTransaction())
{
    UserData data = await context.UserData.FindAsync(userId);

    // there is code here to detect duplicate entries/etc, but I've removed it for brevity
    foreach (var item in items) data.Inventory.Add(item.ItemId, item);

    context.UserData.Update(data);
    await context.SaveChangesAsync();

    transaction.Commit();
}

If two or more calls to this method are made with the same user id then I get concurrent accesses (despite the transaction). This causes the data to sometimes be incorrect. For example, if the inventory is empty and then two calls are made to add items simultaneously (item A and item B), sometimes the database will only contain either A or B, and not both. From logging it appears that it is possible for EF to read from the database while the other read/write is still occurring, causing the code to have the incorrect state of the inventory for when it tries to write back to the db. So I tried marking the isolation level as serializable.

using (var transaction = context.Database.BeginTransaction(System.Data.IsolationLevel.Serializable))

Now I sometimes see an exception:

MySql.Data.MySqlClient.MySqlException (0x80004005): Deadlock found when trying to get lock; try restarting transaction

I don't understand how this code could deadlock... Anyways, I tried to proceed by wrapping this whole thing in a try/catch, and retry:

public static async Task<ResponseError> AddUserItem(Controller controller, MyContext context, ulong userId, List<InventoryItem> items, int retry = 5)
{
    ResponseError result = null;

    try
    {
        using (var transaction = context.Database.BeginTransaction(System.Data.IsolationLevel.Serializable))
        {
            UserData data = await context.UserData.FindAsync(userId);

            // there is code here to detect duplicate entries/etc, but I've removed it for brevity
            foreach (var item in items) data.Inventory.Add(item.ItemId, item);

            context.UserData.Update(data);
            await context.SaveChangesAsync();

            transaction.Commit();
        }
    }
    catch (Exception e)
    {
        if (retry > 0)
        {
            await Task.Delay(SafeRandomGenerator(10, 500));
            return await AddUserItem(controller, context, userId, items, retry--);
        }
        else
        {
            // store exception and return error
        }
    }

    return result;
}

And now I am back to the data being sometimes correct, sometimes not. So I think the deadlock is another problem, but this is the only method accessing this data. So, I'm at a loss. Is there a simple way to read from the database (locking the row in the process) and then writing back (releasing the lock on write) using EF Core? I've looked at using concurrency tokens, but this seems overkill for what appears (on the surface to me) to be a trivial task.

I added logging for mysql connector as well as asp.net server and can see the following failure:

fail: Microsoft.EntityFrameworkCore.Database.Command[20102]
  => RequestId:0HLUD39EILP3R:00000001 RequestPath:/client/AddUserItem => Server.Controllers.ClientController.AddUserItem (ServerSoftware)
  Failed executing DbCommand (78ms) [Parameters=[@p1='?' (DbType = UInt64), @p0='?' (Size = 4000)], CommandType='Text', CommandTimeout='30']
  UPDATE `UserData` SET `Inventory` = @p0
  WHERE `ID` = @p1;
  SELECT ROW_COUNT();

A total hack is to just delay the arrival of the queries by a bit. This works because the client is most likely to generate these calls on load. Normally back-to-back calls aren't expected, so spreading them out in time by delaying on arrival works. However, I'd rather find a correct approach, since this just makes it less likely to be an issue:

ResponseError result = null;
await Task.Delay(SafeRandomGenerator(100, 500));
using (var transaction = context.Database.BeginTransaction(System.Data.IsolationLevel.Serializable))
// etc

This isn't a good answer, because it isn't what I wanted to do, but I'll post it here as it did solve my problem. My problem was that I was trying to read the database row, modify it in asp.net, and then write it back, all within a single transaction and while avoiding deadlocks. The backing field is JSON type, and MySQL provides some JSON functions to help modify that JSON directly in the database. This required me to write SQL statements directly instead of using EF, but it did work.

The first trick was to ensure I could create the row if it didn't exist, without requiring a transaction and lock.

INSERT INTO UserData VALUES ({0},'{{}}','{{}}') ON DUPLICATE KEY UPDATE ID = {0};

I used JSON_REMOVE to delete keys from the JSON field:

UPDATE UserData as S set S.Inventory = JSON_REMOVE(S.Inventory,{1}) WHERE S.ID = {0};

and JSON_SET to add/modify entries:

UPDATE UserData as S set S.Inventory = JSON_SET(S.Inventory,{1},CAST({2} as JSON)) WHERE S.ID = {0};

Note, if you're using EF Core and want to call this using FromSql then you need to return the entity as part of your SQL statement. So you'll need to add something like this to each SQL statement:

SELECT * from UserData where ID = {0} LIMIT 1;

Here is a full working example as an extension method:

public static async Task<UserData> FindOrCreateAsync(this IQueryable<UserData> table, ulong userId)
{
    string sql = "INSERT INTO UserData VALUES ({0},'{{}}','{{}}') ON DUPLICATE KEY UPDATE ID = {0}; SELECT * FROM UserData WHERE ID={0} LIMIT 1;";
    return await table.FromSql(sql, userId).SingleOrDefaultAsync();
}

public static async Task<UserData> JsonRemoveInventory(this DbSet<UserData> table, ulong userId, string key)
{
    if (!key.StartsWith("$.")) key = $"$.\"{key}\"";
    string sql = "UPDATE UserData as S set S.Inventory = JSON_REMOVE(S.Inventory,{1}) WHERE S.ID = {0}; SELECT * from UserData where ID = {0} LIMIT 1;";
    return await table.AsNoTracking().FromSql(sql, userId, key).SingleOrDefaultAsync();
}

Usage:

var data = await context.UserData.FindOrCreateAsync(userId);
await context.UserData.JsonRemoveInventory(userId, itemId);