Timeout unable to read data from the transport connection

Starting Saturday 2nd Feb 2020 around 10 am we have been experiencing random intermittent connectivity issues between our EC2 instance and RDS instance running MySQL 5.6.44.

Our EC2 instance is running Windows 2012 Server R2. We have both ASMX and WCF web services, both of which are experiencing this issue.

Each server makes different calls to the database.

The connectivity issues haven't been related to just one database call/query, and no application or database changes have been deployed since 24th Jan 2020. Also, we aren't seeing any excessive metrics like connections, CPU, etc. These all appear to be average compared to the past 2 weeks.

I've tried restarting all instances, including the RDS instance. But still getting these errors.

Some example exceptions are all of which are random. Some of the queries are very simple SELECT statements returning just 1 row of data.

System.TimeoutException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a while, or established connection failed because connected host has failed to respond. ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after some time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a while, or established connection failed because connected host has failed to respond

and

MySql.Data.MySqlClient.MySqlException (0x80004005): Timeout expired. The timeout period elapsed before completion of the operation or the server is not responding. ---> System.TimeoutException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a while, or established connection failed because connected host has failed to respond. ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

and

error host="WIN-H7RM59S7UK7" type="System.Exception"
message="Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. requestGuid: db2badaf-8f17-4181-8e59-cbe656271491"
source="ProjectName" detail="System.Exception: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. requestGuid: db2badaf-8f17-4181-8e59-cbe656271491 at ProjectName.BookingServices.GetExistingResponsiveBookingPage(String requestId, String cultureCode, String coupon) in c:\VSProjects\ProjectName\ProjectName\BookingServices.asmx.cs:line 158" time="2020-02-05T09:07:17.7473532Z" />

and

error host="WIN-H7RM59S7UK7"
type="System.Net.Sockets.SocketException" message="A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond" source="System"
detail="System.TimeoutException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) --- End of inner exception stack trace --- at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) at MySql.Data.Common.MyNetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count) --- End of inner exception stack trace --- at MySql.Data.Common.MyNetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count) at MySql.Data.MySqlClient.TimedStream.Read(Byte[] buffer, Int32 offset, Int32 count) at System.IO.BufferedStream.Read(Byte[] array, Int32 offset, Int32 count) at MySql.Data.MySqlClient.MySqlStream.ReadFully(Stream stream, Byte[] buffer, Int32 offset, Int32 count) at MySql.Data.MySqlClient.MySqlStream.LoadPacket() at MySql.Data.MySqlClient.MySqlStream.ReadPacket() at MySql.Data.MySqlClient.NativeDriver.ReadOk(Boolean read)
at MySql.Data.MySqlClient.MySqlConnection.ChangeDatabase(String databaseName) at MySql.Data.MySqlClient.MySqlConnection.Open() at TMSShared.MySQL.General.CallMysqlSPReturnReader(String sName, List1 oParameters, Action1 fn) in c:\VSProjects\TMSShared\TMSShared\MySQL\General.cs:line 163 at API.DAL.POCO.Availability.MealPlan.GetAllHotelMealPlanBySupplierId(Int32 iSupplierId) in c:\VSProjects\ProjectNameAPI\API\DAL\POCO\Availability\MealPlan.cs:line 227 at API.DAL.POCO.Availability.GlobalAvailability.Get(GlobalAvailabilityRequest globalAvailabilityRequest) in c:\VSProjects\ProjectName\API\DAL\POCO\Availability\GlobalAvailability.cs:line 409 at API.DAL.BusinessLogic.Request.GlobalAvailability(GlobalAvailabilityRequest globalAvailabilityRequest) in c:\VSProjects\ProjectName\API\DAL\BusinessLogic\Request.cs:line 463"
time="2020-02-05T09:52:19.6013482Z" />

and

error host="WIN-H7RM59S7UK7"
type="System.Threading.ThreadAbortException" message="Thread was being aborted." source="MySql.Data"
detail="System.Threading.ThreadAbortException: Thread was being aborted. at MySql.Data.Common.MyNetworkStream.HandleOrRethrowException(Exception e) at MySql.Data.Common.MyNetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count) at MySql.Data.MySqlClient.TimedStream.Read(Byte[] buffer, Int32 offset, Int32 count) at System.IO.BufferedStream.Read(Byte[] array, Int32 offset, Int32 count) at MySql.Data.MySqlClient.MySqlStream.ReadFully(Stream stream, Byte[] buffer, Int32 offset, Int32 count) at MySql.Data.MySqlClient.MySqlStream.LoadPacket() at MySql.Data.MySqlClient.MySqlStream.ReadPacket() at MySql.Data.MySqlClient.NativeDriver.GetResult(Int32& affectedRow, Int64& insertedId) at MySql.Data.MySqlClient.Driver.NextResult(Int32 statementId, Boolean force) at MySql.Data.MySqlClient.MySqlDataReader.NextResult() at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(CommandBehavior behavior) at TMSShared.MySQL.General.CallMysqlSPReturnReader(String sName, List1 oParameters, Action1 fn) in c:\VSProjects\TMSShared\TMSShared\MySQL\General.cs:line 163 at TMSShared.Utils.Message.GetLoginMessage(LoginMessage iStatusCode, LanguageCultureName languageCultureName) in c:\VSProjects\TMSShared\TMSShared\Utils\Message.cs:line 170 at API.DAL.APIUser.CheckCredentials(String sEmail, String sPass, String sUserIP, Int32 iUserType) in c:\VSProjects\ProjectName\API\DAL\APIUser.cs:line 107 at API.HotelAPI.PostRoomUpdate() in c:\VSProjects\ProjectName\API\Service.asmx.cs:line 97"
time="2020-02-03T11:10:18.2941871Z" />

EDIT ADDITIONAL INFORMATION

The problem started 2020-02-02 09:45:10 and has been ongoing ever since this date. In our IIS HTTPErr log file for this day, here is a snippet of the entries before and after the time we started to experience these issues:

I've masked out the external IP addresses

2020-02-02 09:36:28 *.*.*.* 62969 10.0.0.12 443 - Timer_ConnectionIdle
2020-02-02 09:38:29 *.*.*.* 58836 10.0.0.12 443 - Timer_ConnectionIdle
2020-02-02 09:41:44 *.*.*.* 38060 10.0.0.12 443 - Timer_ConnectionIdle
2020-02-02 09:43:35 *.*.*.* 39069 10.0.0.12 443 - Timer_ConnectionIdle
2020-02-02 09:43:35 *.*.*.* 36349 10.0.0.12 443 - Timer_ConnectionIdle

And then when the problem first occurred we were seeing

2020-02-02 09:45:10 *.*.*.* 53951 10.0.0.12 443 HTTP/1.1 POST /API/V1.04/AdvService.asmx/GetAdvGlobalAvailability - 1 Connection_Abandoned_By_ReqQueue mydomain.com+(API+V1.04)
2020-02-02 09:45:41 *.*.*.* 53967 10.0.0.12 443 HTTP/1.1 POST /API/V1.04/AdvService.asmx/GetAdvGlobalAvailability - 1 Connection_Abandoned_By_ReqQueue mydomain.com+(API+V1.04)
2020-02-02 09:46:04 *.*.*.* 53971 10.0.0.12 443 HTTP/1.1 POST /API/V1.04/AdvService.asmx/GetAdvGlobalAvailability - 1 Connection_Abandoned_By_ReqQueue mydomain.com+(API+V1.04)
2020-02-02 09:46:29 *.*.*.* 51898 10.0.0.12 443 HTTP/1.1 POST /API/V1.04/AdsHotelService.asmx/getRoomAvailability 503 1 AppOffline mydomain.com+(API+V1.04)
...
...
...
2020-02-02 12:47:26 *.*.*.* 64846 10.0.0.12 443 HTTP/1.1 POST /API/V1.04/AdsHotelService.asmx/getRoomAvailability 503 1 AppOffline mydomain.com+(API+V1.04)
2020-02-02 12:47:33 *.*.*.* 40169 10.0.0.12 443 HTTP/1.1 POST /API/V1.04/Service.asmx/PostRoomUpdate 503 1 AppOffline mydomain.com+(API+V1.04)
2020-02-02 12:47:33 *.*.*.* 18340 10.0.0.12 443 HTTP/1.1 POST /API/V1.04/AdsHotelService.asmx/getRoomAvailability 503 1 AppOffline mydomain.com+(API+V1.04)
2020-02-02 12:47:35 *.*.*.* 18426 10.0.0.12 443 HTTP/1.1 POST /API/V1.04/AdsHotelService.asmx/getRoomAvailability 503 1 AppOffline mydomain.com+(API+V1.04)
2020-02-02 12:47:38 *.*.*.* 18938 10.0.0.12 443 HTTP/1.1 POST /API/V1.04/AdsHotelService.asmx/getRoomAvailability 503 1 AppOffline mydomain.com+(API+V1.04)
2020-02-02 12:47:38 *.*.*.* 18950 10.0.0.12 443 HTTP/1.1 POST /API/V1.04/AdsHotelService.asmx/getRoomAvailability 503 1 AppOffline mydomain.com+(API+V1.04)
2020-02-02 12:47:40 *.*.*.* 19124 10.0.0.12 443 HTTP/1.1 POST /API/V1.04/AdsHotelService.asmx/getRoomAvailability 503 1 AppOffline mydomain.com+(API+V1.04)
2020-02-02 12:47:40 *.*.*.* 64424 10.0.0.12 443 HTTP/1.1 POST /API/V1.01/AdvService.asmx/GetAdvAvailability 503 1 Disabled mydomain.com+(API)
2020-02-02 12:56:07 *.*.*.* 38634 10.0.0.12 80 HTTP/1.0 GET / 404 - NotFound -
#Software: Microsoft HTTP API 2.0
#Version: 1.0
#Date: 2020-02-02 13:05:22
#Fields: date time c-ip c-port s-ip s-port cs-version cs-method cs-uri sc-status s-siteid s-reason s-queuename
2020-02-02 13:05:21 *.*.*.* 58945 10.0.0.12 443 - - - - - Timer_ConnectionIdle -
2020-02-02 13:05:21 *.*.*.* 58948 10.0.0.12 443 - - - - - Timer_ConnectionIdle -
2020-02-02 13:05:21 *.*.*.* 58944 10.0.0.12 443 - - - - - Timer_ConnectionIdle -
2020-02-02 13:05:26 *.*.*.* 58946 10.0.0.12 443 - - - - - Timer_ConnectionIdle -
2020-02-02 13:06:27 *.*.*.* 58984 10.0.0.12 443 - - - - - Timer_ConnectionIdle -
2020-02-02 13:06:31 *.*.*.* 38084 10.0.0.12 443 - - - - - Timer_ConnectionIdle -
2020-02-02 13:06:31 *.*.*.* 33065 10.0.0.12 443 - - - - - Timer_ConnectionIdle -
2020-02-02 13:09:23 *.*.*.* 46315 10.0.0.12 443 - - - - - Timer_ConnectionIdle -
2020-02-02 13:10:02 *.*.*.* 58386 10.0.0.12 443 - - - - - Timer_MinBytesPerSecond -
2020-02-02 13:13:48 *.*.*.* 58382 10.0.0.12 443 - - - - - Timer_MinBytesPerSecond -

I'm not seeing the 503 errors in the log file now, but we are experiencing connectivity issues between our EC2 instance running IIS and our RDS database.

2020-02-04 09:00:26 *.*.*.* 45371 10.0.0.12 443 - - - - - Timer_ConnectionIdle -
2020-02-04 09:03:22 *.*.*.* 53461 10.0.0.12 443 - - - - - Timer_ConnectionIdle -
2020-02-04 09:04:07 *.*.*.* 55266 10.0.0.12 443 - - - - - Timer_ConnectionIdle -
2020-02-04 09:04:07 *.*.*.* 33402 10.0.0.12 443 - - - - - Timer_ConnectionIdle -
2020-02-04 09:04:07 *.*.*.* 48068 10.0.0.12 443 - - - - - Timer_ConnectionIdle -
2020-02-04 09:04:17 *.*.*.* 44998 10.0.0.12 443 - - - - - Timer_ConnectionIdle -
2020-02-04 09:06:38 *.*.*.* 38659 10.0.0.12 443 - - - - - Timer_ConnectionIdle -
2020-02-04 09:06:58 *.*.*.* 35861 10.0.0.12 443 - - - - - Timer_ConnectionIdle -

I’m not sure if this issue is related to Windows, IIS, MySQL or connectivity between the two instances.