The network path was not found when add member to group

I'm using the following code to add user to group

using (PrincipalContext context = new PrincipalContext(ContextType.Machine, ip))
{
    using (GroupPrincipal group = GroupPrincipal.FindByIdentity(context, groupName))
    {
        using (UserPrincipal user = UserPrincipal.FindByIdentity(context, userName))
        {
            group.Members.Add(user); // Exception throw
            group.Save();
        }
    }
}

When the group is empty, it works as expected.

However, if the group has any member already, it will throw

System.Runtime.InteropServices.COMException (0x80070035): The network path was not found.

at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail) at System.DirectoryServices.DirectoryEntry.Bind()
at System.DirectoryServices.DirectoryEntry.get_SchemaClassName() at System.DirectoryServices.AccountManagement.SAMUtils.DirectoryEntryAsPrincipal(DirectoryEntry de, StoreCtx storeCtx) at System.DirectoryServices.AccountManagement.SAMMembersSet.get_CurrentAsPrincipal() at System.DirectoryServices.AccountManagement.PrincipalCollectionEnumerator.MoveNext() at System.DirectoryServices.AccountManagement.PrincipalCollection.ContainsEnumTest(Principal principal) at System.DirectoryServices.AccountManagement.PrincipalCollection.Add(Principal principal) at SKAR.Database.LocalGroup.AddLocalUserToLocalGroup(String machineName, String groupName, String userName)

I don't understand why & how "if the group is empty or not" could impact the add operation at all.

Anyway, I tried another approach

string userPath = string.Format("WinNT://{0}/{1},user", ip, userName);
string groupPath = string.Format("WinNT://{0}/{1},group", ip, groupName);
using (DirectoryEntry group = new DirectoryEntry(groupPath))
{
    group.Invoke("Add", userPath);
    group.CommitChanges();
}

Which throws

Error occur: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Runtime.InteropServices.COMException: A member could not be added to or removed from the local group because the member does not exist.

--- End of inner exception stack trace --- at System.DirectoryServices.DirectoryEntry.Invoke(String methodName, Object[] args) at SKAR.Database.LocalGroup.AddLocalUserToLocalGroup(String machineName, String groupName, String userName) at Prerequisite.Program.Main(String[] args)

Not sure if it's since we are not supposed to use IP in the second usage.

In short, I have two questions
1. Why the first approach throws me the exception if the group has member already?
2. How can I make it work?

I'm using windows 2012 R2, trying to add a user from one server to the local group in another server.
Both servers are in the same workgroup(not domain).
Both sides have the same account with the same password.