System.Net.Http.HttpRequestException for Send grid send email

We are facing following exception in calling Sendgrid Send Email function in some case. We get this exception for 20-30 emails out of 50,000 in a day.

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: Unknown error (0xffffffff)     
at System.Net.Sockets.Socket.BeginConnectEx(EndPoint remoteEP, Boolean flowContext, AsyncCallback callback, Object state)     
at System.Net.Sockets.Socket.UnsafeBeginConnect(EndPoint remoteEP, AsyncCallback callback, Object state)     
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)     --- End of inner exception stack trace ---     
at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)     
at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)     --- End of inner exception stack trace ---     
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     
at SendGrid.Helpers.Reliability.RetryDelegatingHandler.<SendAsync>d__4.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     
at SendGrid.BaseClient.<MakeRequest>d__20.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     
at SendGrid.BaseClient.<RequestAsync>d__21.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     
at SendGrid.BaseClient.<SendEmailAsync>d__22.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    

My code to send email is

var client = new SendGridClient("API Key");
var response = await client.SendEmailAsync(message);

Can anyone please help me how can I add more verification to avoid such situation.

The Retry logic did work for me so far.

catch (Exception ex)
{
    if (--retryMax == 0)
    {
        throw;
    }
    else
    {
        await Task.Delay(1000);
    }
    return null;
}