lately I have noticed that usually the web application that is in production (online) goes wrong and I am forced to restart it manually.
To try to understand the error in the web.config file I enabled the following entry: - stdoutLogEnabled = "true"
Just today I noticed (randomly) that the application was down and opening the log file this was its content:
crit: Microsoft.AspNetCore.Hosting.Diagnostics[6]
Application startup exception
MySql.Data.MySqlClient.MySqlException (0x80004005): User 'nome_database' has exceeded the 'max_user_connections' resource (current value: 10)
at MySqlConnector.Core.ServerSession.SwitchAuthenticationAsync(ConnectionSettings cs, PayloadData payload, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 535
at MySqlConnector.Core.ServerSession.TryResetConnectionAsync(ConnectionSettings cs, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 498
at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ConnectionPool.cs:line 72
at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ConnectionPool.cs:line 141
at MySql.Data.MySqlClient.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Nullable`1 ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 645
at MySql.Data.MySqlClient.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 312
at MySql.Data.MySqlClient.MySqlConnection.Open() in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 280
at Hangfire.MySql.MySqlStorage.CreateAndOpenConnection()
at Hangfire.MySql.MySqlStorage.UseTransaction[T](Func`2 func, Nullable`1 isolationLevel)
at Hangfire.MySql.MySqlStorage.UseTransaction(Action`1 action)
at Hangfire.MySql.MySqlWriteOnlyTransaction.Commit()
at Hangfire.RecurringJobManager.AddOrUpdate(String recurringJobId, Job job, String cronExpression, RecurringJobOptions options)
at Hangfire.RecurringJobManagerExtensions.AddOrUpdate(IRecurringJobManager manager, String recurringJobId, Job job, String cronExpression, TimeZoneInfo timeZone, String queue)
at Hangfire.RecurringJob.AddOrUpdate[T](String recurringJobId, Expression`1 methodCall, String cronExpression, TimeZoneInfo timeZone, String queue)
at FFM3._1.Utility.HangfireJobScheduler.ScheduleRecurringJobs(IWebHostEnvironment env) in C:\Users\Xxxx\Xxxx\XXX\XXX\XXX\Utility\HangfireJobScheduler.cs:line 23
at FFM3._1.Startup.Configure(IApplicationBuilder app, IWebHostEnvironment env, IServiceProvider serviceProvider) in C:\Users\Xxxx\Xxxx\XXX\XXX\XXX\Startup.cs:line 170
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.AspNetCore.Hosting.ConfigureBuilder.Invoke(Object instance, IApplicationBuilder builder)
at Microsoft.AspNetCore.Hosting.ConfigureBuilder.<>c__DisplayClass4_0.<Build>b__0(IApplicationBuilder builder)
at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass13_0.<UseStartup>b__2(IApplicationBuilder app)
at Microsoft.AspNetCore.Mvc.Filters.MiddlewareFilterBuilderStartupFilter.<>c__DisplayClass0_0.<Configure>g__MiddlewareFilterBuilder|0(IApplicationBuilder builder)
at Microsoft.AspNetCore.Server.IIS.Core.IISServerSetupFilter.<>c__DisplayClass2_0.<Configure>b__0(IApplicationBuilder app)
at Microsoft.AspNetCore.HostFilteringStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app)
at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
Unhandled exception. MySql.Data.MySqlClient.MySqlException (0x80004005): User 'nome_database' has exceeded the 'max_user_connections' resource (current value: 10)
at MySqlConnector.Core.ServerSession.SwitchAuthenticationAsync(ConnectionSettings cs, PayloadData payload, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 535
at MySqlConnector.Core.ServerSession.TryResetConnectionAsync(ConnectionSettings cs, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 498
at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ConnectionPool.cs:line 72
at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ConnectionPool.cs:line 141
at MySql.Data.MySqlClient.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Nullable`1 ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 645
at MySql.Data.MySqlClient.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 312
at MySql.Data.MySqlClient.MySqlConnection.Open() in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 280
at Hangfire.MySql.MySqlStorage.CreateAndOpenConnection()
at Hangfire.MySql.MySqlStorage.UseTransaction[T](Func`2 func, Nullable`1 isolationLevel)
at Hangfire.MySql.MySqlStorage.UseTransaction(Action`1 action)
at Hangfire.MySql.MySqlWriteOnlyTransaction.Commit()
at Hangfire.RecurringJobManager.AddOrUpdate(String recurringJobId, Job job, String cronExpression, RecurringJobOptions options)
at Hangfire.RecurringJobManagerExtensions.AddOrUpdate(IRecurringJobManager manager, String recurringJobId, Job job, String cronExpression, TimeZoneInfo timeZone, String queue)
at Hangfire.RecurringJob.AddOrUpdate[T](String recurringJobId, Expression`1 methodCall, String cronExpression, TimeZoneInfo timeZone, String queue)
at FFM3._1.Utility.HangfireJobScheduler.ScheduleRecurringJobs(IWebHostEnvironment env) in C:\Users\Xxxx\Xxxx\XXX\XXX\XXX\Utility\HangfireJobScheduler.cs:line 23
at FFM3._1.Startup.Configure(IApplicationBuilder app, IWebHostEnvironment env, IServiceProvider serviceProvider) in C:\Users\Xxxx\Xxxx\XXX\XXX\XXX\Startup.cs:line 170
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.AspNetCore.Hosting.ConfigureBuilder.Invoke(Object instance, IApplicationBuilder builder)
at Microsoft.AspNetCore.Hosting.ConfigureBuilder.<>c__DisplayClass4_0.<Build>b__0(IApplicationBuilder builder)
at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass13_0.<UseStartup>b__2(IApplicationBuilder app)
at Microsoft.AspNetCore.Mvc.Filters.MiddlewareFilterBuilderStartupFilter.<>c__DisplayClass0_0.<Configure>g__MiddlewareFilterBuilder|0(IApplicationBuilder builder)
at Microsoft.AspNetCore.Server.IIS.Core.IISServerSetupFilter.<>c__DisplayClass2_0.<Configure>b__0(IApplicationBuilder app)
at Microsoft.AspNetCore.HostFilteringStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app)
at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
at FFM3._1.Program.Main(String[] args) in C:\Users\Xxxx\Xxxx\XXX\XXX\XXX\Program.cs:line 16
I believe the problem is in the use of hangfire (or perhaps in its incorrect configuration).
I am working with Asp.Net Core 3.1 and I am using MySql as a DB.
Below I report what I entered in the ConfigureServices method of the Startup.cs file
services.AddHangfire(configuration => configuration.SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
.UseSimpleAssemblyNameTypeSerializer()
.UseRecommendedSerializerSettings()
.UseStorage(new MySqlStorage(Configuration.GetConnectionString("DefaultConnection"), new MySqlStorageOptions()
{
TablesPrefix = "Hangfire"
})));
services.AddHangfireServer();
Below I report what I entered in the Configure method of the Startup.cs file
app.UseHangfireDashboard("/hangfire", new DashboardOptions
{
Authorization = new[] { new HangfireAuthorizationFilter() }
});
HangfireJobScheduler.ScheduleRecurringJobs(env);
From the log I hope you can give me some indications to help me solve this problem.
I thank you in advance.
Check the connection pooling related connection string options.
By default, connection pooling is enabled in MySqlConnector (which is the underlying library used by Pomelo) and opens up until 100 connections (the maxpoolsize
connection string option).
If your MySQL database is configured in a way, that it cannot handle that many connections, and you recently discovered these exceptions, then you might got more traffic recently then before and are exceeding the configured values (more concurrent connections than max_connections
).
To resolve this issue, either just configure the connection pooling in a way, that you don't exceed the MySQL's max_connections
value or configure MySQL in a way, to allow more connections (see max_connections) or (as a last resort) to forcefully close already open (and probably stale) connections after a certain amount of time (see wait_timeout and interactive_timeout).
User contributions licensed under CC BY-SA 3.0