Getting Win32Exception: Connection Timeout Expired even on small webserver load

I've configured EF 2.1 in my ASP.NET Core 2.1 app like this:

 services.AddDbContext<TacsDbContext>(
     options => options.UseSqlServer(connectionString, sql => sql.EnableRetryOnFailure()),
     ServiceLifetime.Transient, ServiceLifetime.Singleton);

Every time I restart the app, after couple of days running in production I'm getting this for all requests:

System.Data.SqlClient.SqlException (0x80131904): Connection Timeout Expired. The timeout period elapsed while attempting to consume the pre-login handshake acknowledgement. This could be because the pre-login handshake failed or the server was unable to respond back in time. The duration spent while attempting to connect to this server was - [Pre-Login] initialization=1470; handshake=1; ---> System.ComponentModel.Win32Exception (258): The wait operation timed out

 at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
 at System.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()
 --- End of stack trace from previous location where exception was thrown ---
 at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnectionAsync(Boolean errorsExpected, CancellationToken cancellationToken)
 at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)
 at Microsoft.EntityFrameworkCore.Query.Internal.AsyncQueryingEnumerable`1.AsyncEnumerator.BufferlessMoveNext(DbContext _, Boolean buffer, CancellationToken cancellationToken)
 at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
 at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
 at Microsoft.EntityFrameworkCore.Query.Internal.AsyncQueryingEnumerable`1.AsyncEnumerator.MoveNext(CancellationToken cancellationToken)
 at System.Linq.AsyncEnumerable.SelectEnumerableAsyncIterator`2.MoveNextCore(CancellationToken cancellationToken)
 at System.Linq.AsyncEnumerable.AsyncIterator`1.MoveNext(CancellationToken cancellationToken)
 at System.Linq.AsyncEnumerable.FirstOrDefault_[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken)
 at Microsoft.EntityFrameworkCore.Query.Internal.AsyncLinqOperatorProvider.TaskResultAsyncEnumerable`1.Enumerator.MoveNext(CancellationToken cancellationToken)
 at System.Linq.AsyncEnumerable.SelectEnumerableAsyncIterator`2.MoveNextCore(CancellationToken cancellationToken)
 at System.Linq.AsyncEnumerable.AsyncIterator`1.MoveNext(CancellationToken cancellationToken)
 at Microsoft.EntityFrameworkCore.Query.Internal.AsyncLinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext(CancellationToken cancellationToken)
 at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteSingletonAsyncQuery[TResult](QueryContext queryContext, Func`2 compiledQuery, IDiagnosticsLogger`1 logger, Type contextType)
 at TACS.People.PeopleRepository.GetPersonInformation(String enterpriseId) in D:\a\1\s\TACS.People\PeopleRepository.cs:line 39
 at TACS.People.PeopleRepositoryExtensions.EnsurePerson(IPeopleRepository peopleRepository, String enterpriseId) in D:\a\1\s\TACS.People\PeopleRepositoryExtensions.cs:line 13
 at TACS.Web.AppCode.Auth.TacsClaimsTransformation.AddPersonId(ClaimsPrincipal principal) in D:\a\1\s\TACS.Web\AppCode\Auth\TacsClaimsTransformation.cs:line 77
 at TACS.Web.AppCode.Auth.TacsClaimsTransformation.TransformAsync(ClaimsPrincipal principal) in D:\a\1\s\TACS.Web\AppCode\Auth\TacsClaimsTransformation.cs:line 52
 at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme)
 at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
 at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
 at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
 at TACS.Web.Startup.<>c__DisplayClass6_0.<<Configure>b__0>d.MoveNext() in D:\a\1\s\TACS.Web\Startup.cs:line 172
 --- End of stack trace from previous location where exception was thrown ---
 at Microsoft.AspNetCore.Server.IISIntegration.IISMiddleware.Invoke(HttpContext httpContext)
 at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
 ClientConnectionId:89feb963-e494-4f12-94fb-57aa18a2c116
 Error Number:-2,State:0,Class:11

It looks like there is too many connection being opened. Is this because I have used Transient service life time?

I believed it is good practice to create new DbContext each time I want to access database. Using scoped lifetime would cause, that single instance of DbContext would be used per request, and that looks like antipattern, since there might be multiple Unit of works.