Renci.SshNet A connection attempt failed because connection was terminated by the client

I'm trying to upload generated word documents to a SFTP server using Renci.SshNet:

if (protocol == "SFTP")
                {
                    if (connectorSettings.DebugMode) properties.AddInfoMessage(_actionName + " " + "File Transfer via SFTP Protocol" + ".");
                    //SFTP
                    using (var sftpClient = new SftpClient(host, port, username, password))
                    {
                        //Connect clients to server
                        sftpClient.Connect();

                        if (connectorSettings.DebugMode) properties.AddInfoMessage(_actionName + " " + "Connected to " + host + ".");

                        try
                        {
                            //Create remote directory
                            if (createDirectory && !sftpClient.Exists(directory))
                            {
                                if (connectorSettings.DebugMode) properties.AddInfoMessage(_actionName + " " + "Directory Creation Start: " + directory + ".");
                                sftpClient.CreateDirectory(directory);

                                if (connectorSettings.DebugMode) properties.AddInfoMessage(_actionName + " Directory Creation Success");
                            }

                            sftpClient.ChangeDirectory(directory);
                            if (connectorSettings.DebugMode) properties.AddInfoMessage(_actionName + " " + "Destination directory: " + directory + ".");

                            using (var docStream = properties.GetDocumentStream(doc))
                            {
                                if (connectorSettings.DebugMode) properties.AddInfoMessage(_actionName + " " + "File Upload Start: " + doc.DisplayName + doc.Extension + ".");
                                sftpClient.BufferSize = 4 * 1024; // bypass Payload error large files
                                sftpClient.UploadFile(docStream, doc.DisplayName + doc.Extension);
                            }

                            if (connectorSettings.DebugMode)  properties.AddInfoMessage(_actionName + " " + "Upload " + doc.DisplayName + doc.Extension + " " + " Completed.");
                        }
                        finally
                        {
                            sftpClient.Disconnect();

                            if (connectorSettings.DebugMode)  properties.AddInfoMessage(_actionName + " " + "Disconnect from " + host + ".");
                        }
                    }

Now this piece of code works fine with all clients I have been working with in conneting to their server and upload multiple files one by one. However, yesterday with a new client I got this weird issue:

I can connect and upload multiples documents successfully to their server from this code locally (local application). But when I publish the web app to Azure, I can only connect and upload multiple documents in the first 1 MINUTE, after that somehow I got blocked for 1 hour interval and my application can no longer connect, so the code failed at sftpClient.Connect(); giving me this exception error:

Error: System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond ....

at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)

at Renci.SshNet.Session.SocketConnect(String host, Int32 port)

at Renci.SshNet.Session.Connect()

at Renci.SshNet.BaseClient.Connect()

at FileTransferActionProvider.FileTransferAction.Run(ActionProperties properties)

The server the client is using is Bitvise SSH Server. Reading their documentation I found this:

“Another way Bitvise SSH Server tries to thwart attackers is through automatic blocking of IP addresses that have recently initiated multiple failed login attempts. In default settings, the SSH server will block for 1 hour any IP address that initiates more than 20 failed login attempts in 5 minutes.”

which is weird since my application has clearky made a successful login in the first few attempts to upload files. Is this something wrong with the library i'm using (Renci.ssh) or something with Bitvise SSH Server ?

Please help. Thanks