Hangfire - Asp.Net Core 3.1 - Error log Application startup exception

0

lately I have noticed that usually the web application that is in production (online) goes wrong and I am forced to restart it manually.

To try to understand the error in the web.config file I enabled the following entry: - stdoutLogEnabled = "true"

Just today I noticed (randomly) that the application was down and opening the log file this was its content:

crit: Microsoft.AspNetCore.Hosting.Diagnostics[6]

      Application startup exception

MySql.Data.MySqlClient.MySqlException (0x80004005): User 'nome_database' has exceeded the 'max_user_connections' resource (current value: 10)

   at MySqlConnector.Core.ServerSession.SwitchAuthenticationAsync(ConnectionSettings cs, PayloadData payload, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 535

   at MySqlConnector.Core.ServerSession.TryResetConnectionAsync(ConnectionSettings cs, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 498

   at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ConnectionPool.cs:line 72

   at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ConnectionPool.cs:line 141

   at MySql.Data.MySqlClient.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Nullable`1 ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 645

   at MySql.Data.MySqlClient.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 312

   at MySql.Data.MySqlClient.MySqlConnection.Open() in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 280

   at Hangfire.MySql.MySqlStorage.CreateAndOpenConnection()

   at Hangfire.MySql.MySqlStorage.UseTransaction[T](Func`2 func, Nullable`1 isolationLevel)

   at Hangfire.MySql.MySqlStorage.UseTransaction(Action`1 action)

   at Hangfire.MySql.MySqlWriteOnlyTransaction.Commit()

   at Hangfire.RecurringJobManager.AddOrUpdate(String recurringJobId, Job job, String cronExpression, RecurringJobOptions options)

   at Hangfire.RecurringJobManagerExtensions.AddOrUpdate(IRecurringJobManager manager, String recurringJobId, Job job, String cronExpression, TimeZoneInfo timeZone, String queue)

   at Hangfire.RecurringJob.AddOrUpdate[T](String recurringJobId, Expression`1 methodCall, String cronExpression, TimeZoneInfo timeZone, String queue)

   at FFM3._1.Utility.HangfireJobScheduler.ScheduleRecurringJobs(IWebHostEnvironment env) in C:\Users\Xxxx\Xxxx\XXX\XXX\XXX\Utility\HangfireJobScheduler.cs:line 23

   at FFM3._1.Startup.Configure(IApplicationBuilder app, IWebHostEnvironment env, IServiceProvider serviceProvider) in C:\Users\Xxxx\Xxxx\XXX\XXX\XXX\Startup.cs:line 170

   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)

   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

   at Microsoft.AspNetCore.Hosting.ConfigureBuilder.Invoke(Object instance, IApplicationBuilder builder)

   at Microsoft.AspNetCore.Hosting.ConfigureBuilder.<>c__DisplayClass4_0.<Build>b__0(IApplicationBuilder builder)

   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass13_0.<UseStartup>b__2(IApplicationBuilder app)

   at Microsoft.AspNetCore.Mvc.Filters.MiddlewareFilterBuilderStartupFilter.<>c__DisplayClass0_0.<Configure>g__MiddlewareFilterBuilder|0(IApplicationBuilder builder)

   at Microsoft.AspNetCore.Server.IIS.Core.IISServerSetupFilter.<>c__DisplayClass2_0.<Configure>b__0(IApplicationBuilder app)

   at Microsoft.AspNetCore.HostFilteringStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app)

   at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)

Unhandled exception. MySql.Data.MySqlClient.MySqlException (0x80004005): User 'nome_database' has exceeded the 'max_user_connections' resource (current value: 10)

   at MySqlConnector.Core.ServerSession.SwitchAuthenticationAsync(ConnectionSettings cs, PayloadData payload, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 535

   at MySqlConnector.Core.ServerSession.TryResetConnectionAsync(ConnectionSettings cs, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 498

   at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ConnectionPool.cs:line 72

   at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ConnectionPool.cs:line 141

   at MySql.Data.MySqlClient.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Nullable`1 ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 645

   at MySql.Data.MySqlClient.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 312

   at MySql.Data.MySqlClient.MySqlConnection.Open() in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 280

   at Hangfire.MySql.MySqlStorage.CreateAndOpenConnection()

   at Hangfire.MySql.MySqlStorage.UseTransaction[T](Func`2 func, Nullable`1 isolationLevel)

   at Hangfire.MySql.MySqlStorage.UseTransaction(Action`1 action)

   at Hangfire.MySql.MySqlWriteOnlyTransaction.Commit()

   at Hangfire.RecurringJobManager.AddOrUpdate(String recurringJobId, Job job, String cronExpression, RecurringJobOptions options)

   at Hangfire.RecurringJobManagerExtensions.AddOrUpdate(IRecurringJobManager manager, String recurringJobId, Job job, String cronExpression, TimeZoneInfo timeZone, String queue)

   at Hangfire.RecurringJob.AddOrUpdate[T](String recurringJobId, Expression`1 methodCall, String cronExpression, TimeZoneInfo timeZone, String queue)

   at FFM3._1.Utility.HangfireJobScheduler.ScheduleRecurringJobs(IWebHostEnvironment env) in C:\Users\Xxxx\Xxxx\XXX\XXX\XXX\Utility\HangfireJobScheduler.cs:line 23

   at FFM3._1.Startup.Configure(IApplicationBuilder app, IWebHostEnvironment env, IServiceProvider serviceProvider) in C:\Users\Xxxx\Xxxx\XXX\XXX\XXX\Startup.cs:line 170

   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)

   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

   at Microsoft.AspNetCore.Hosting.ConfigureBuilder.Invoke(Object instance, IApplicationBuilder builder)

   at Microsoft.AspNetCore.Hosting.ConfigureBuilder.<>c__DisplayClass4_0.<Build>b__0(IApplicationBuilder builder)

   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass13_0.<UseStartup>b__2(IApplicationBuilder app)

   at Microsoft.AspNetCore.Mvc.Filters.MiddlewareFilterBuilderStartupFilter.<>c__DisplayClass0_0.<Configure>g__MiddlewareFilterBuilder|0(IApplicationBuilder builder)

   at Microsoft.AspNetCore.Server.IIS.Core.IISServerSetupFilter.<>c__DisplayClass2_0.<Configure>b__0(IApplicationBuilder app)

   at Microsoft.AspNetCore.HostFilteringStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app)

   at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)

   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)

   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)

   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)

   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)

   at FFM3._1.Program.Main(String[] args) in C:\Users\Xxxx\Xxxx\XXX\XXX\XXX\Program.cs:line 16

I believe the problem is in the use of hangfire (or perhaps in its incorrect configuration).

I am working with Asp.Net Core 3.1 and I am using MySql as a DB.

Below I report what I entered in the ConfigureServices method of the Startup.cs file

services.AddHangfire(configuration => configuration.SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
                                                               .UseSimpleAssemblyNameTypeSerializer()
                                                               .UseRecommendedSerializerSettings()
                                                               .UseStorage(new MySqlStorage(Configuration.GetConnectionString("DefaultConnection"), new MySqlStorageOptions()
                                                               {
                                                                   TablesPrefix = "Hangfire"
                                                               })));
            services.AddHangfireServer();

Below I report what I entered in the Configure method of the Startup.cs file

app.UseHangfireDashboard("/hangfire", new DashboardOptions
            {
                Authorization = new[] { new HangfireAuthorizationFilter() }
            });
            HangfireJobScheduler.ScheduleRecurringJobs(env);

From the log I hope you can give me some indications to help me solve this problem.

I thank you in advance.

mysql
asp.net-core
hangfire
pomelo-entityframeworkcore-mysql
asked on Stack Overflow May 19, 2020 by Gargano

1 Answer

0

Check the connection pooling related connection string options.

By default, connection pooling is enabled in MySqlConnector (which is the underlying library used by Pomelo) and opens up until 100 connections (the maxpoolsize connection string option).

If your MySQL database is configured in a way, that it cannot handle that many connections, and you recently discovered these exceptions, then you might got more traffic recently then before and are exceeding the configured values (more concurrent connections than max_connections).

To resolve this issue, either just configure the connection pooling in a way, that you don't exceed the MySQL's max_connections value or configure MySQL in a way, to allow more connections (see max_connections) or (as a last resort) to forcefully close already open (and probably stale) connections after a certain amount of time (see wait_timeout and interactive_timeout).

answered on Stack Overflow May 27, 2020 by lauxjpn

User contributions licensed under CC BY-SA 3.0