scale-out signalr server with redis backplane not working correctly (vue front end)

Appreciation for any answers!

I created a signalr backend server based on .net core 3.1 running on docker(debian). It works well when i only create single server deployment on kubernetes. But when i increase replicas to more than 1 ,it works incorrectly. It looks like the redis backplane not working ,and leads to unreachable communication between multiple servers .

Following the official documention,i installed nuget package:

<PackageReference Include="Microsoft.AspNetCore.SignalR.StackExchangeRedis" Version="3.1.13" />

this is the backend server startup.cs code :

public void ConfigureServices(IServiceCollection services)
{
// ...
  services
    .AddSignalR(option =>
    {
        option.EnableDetailedErrors = true;
    })
    .AddNewtonsoftJsonProtocol()
    .AddStackExchangeRedis(option =>
    {
    option.ConnectionFactory = async writer =>
    {
        var config = new ConfigurationOptions
        {
            AbortOnConnectFail = false,
            ChannelPrefix = "devopshub",
            ServiceName = "devopshub",
            ClientName = "devopshub"
        };
        config.DefaultDatabase = 13;
        var connection = await ConnectionMultiplexer.ConnectAsync("redis host:6379,allowAdmin=true,defaultdatabase=13", writer);
        connection.ConnectionFailed += (_, e) =>
        {
            Console.WriteLine("SignalR Redis Server Connection Failed.");
        };
        if (connection.IsConnected)
            Console.WriteLine("SignalR Redis Server Connected.");
        else
            Console.WriteLine("SignalR Redis Server Not Connected.");
        return connection;
    };
// ... 
}

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
// ...
    app.UseRouting();
    app.UseCors(builder => builder
    .SetIsOriginAllowed(_ => true)
    .AllowAnyMethod()
    .AllowAnyHeader()
    .AllowCredentials());
    app.UseEndpoints(builder =>
    {
        builder.MapHub<LogHub>("/api/loghub").RequireCors(t => t.SetIsOriginAllowed(_ => true).AllowAnyMethod().AllowAnyHeader().AllowCredentials());
        builder.MapHub<ProcessDetailStatusSyncHub>("/api/processdetailstatussynchub").RequireCors(t => t.SetIsOriginAllowed(_ => true).AllowAnyMethod().AllowAnyHeader().AllowCredentials());
        builder.MapControllers();
        builder.MapHealthChecks("/health");
    });
// ...
}

All codes seems ok,and it does work correctly when the server is deployed single.once i add the replicas ,the websocket connection would reset or disconnect immediately or after several seconds. After that,the front end (vue)project would throw an unhandle error ,and then it would crash.

Like this:

Error: read ECONNRESET
    at TCP.onStreamRead (internal/stream_base_commons.js:205:27)
Emitted 'error' event on Socket instance at:
    at emitErrorNT (internal/streams/destroy.js:92:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  errno: 'ECONNRESET',
  code: 'ECONNRESET',
  syscall: 'read'
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! frontendProjectName@0.0.1 dev: `vue-cli-service serve`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the frontendProjectName@0.0.1 dev script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\ASUS\AppData\Roaming\npm-cache\_logs\2021-03-18T03_13_05_435Z-debug.log
终端进程“C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -Command npm run dev”已终止，退出代码: 1。

The last line chinese words means"process stoped, exit with code 1"

And this is the log file content :

0 info it worked if it ends with ok
1 verbose cli [
1 verbose cli   'D:\\nodejs\\node.exe',
1 verbose cli   'D:\\nodejs\\node_modules\\npm\\bin\\npm-cli.js',
1 verbose cli   'run',
1 verbose cli   'dev'
1 verbose cli ]
2 info using npm@6.13.4
3 info using node@v12.16.1
4 verbose run-script [ 'predev', 'dev', 'postdev' ]
5 info lifecycle frontendProjectName@0.0.1~predev: frontendProjectName@0.0.1
6 info lifecycle frontendProjectName@0.0.1~dev: frontendProjectName@0.0.1
7 verbose lifecycle frontendProjectName@0.0.1~dev: unsafe-perm in lifecycle true
8 verbose lifecycle frontendProjectName@0.0.1~dev: PATH: D:\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin;E:\workspace\CICD\hualv-devops-frontend\node_modules\.bin;C:\Users\ASUS\AppData\Roaming\npm;C:\ProgramData\DockerDesktop\version-bin;C:\Program Files\Docker\Docker\Resources\bin;C:\Python27\;C:\Python27\Scripts;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\dotnet\;C:\Program Files (x86)\Microsoft SQL Server\140\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\140\DTS\Binn\;D:\Program Files\platform-tools;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\Microsoft SQL Server\140\Tools\Binn\;C:\Program Files\Microsoft SQL Server\140\DTS\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\130\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\Client SDK\ODBC\130\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\140\Tools\Binn\ManagementStudio\;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Git\cmd;C:\Program Files\TortoiseGit\bin;D:\nodejs\;C:\Program Files (x86)\ATI Technologies\ATI.ACE\Core-Static;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;E:\Nuget;C:\Users\ASUS\AppData\Roaming\npm;C:\Users\ASUS\AppData\Local\Microsoft\WindowsApps;D:\Program Files\Fiddler;D:\Docker Toolbox;C:\Users\ASUS\.dotnet\tools;D:\JetBrains Rider\bin;;D:\Microsoft VS Code\bin;E:\Nuget;C:\Users\ASUS\AppData\Local\Programs\Fiddler;C:\Users\ASUS\AppData\Local\Microsoft\WindowsApps;D:\JetBrains Rider 2020.3.2\bin;
9 verbose lifecycle frontendProjectName@0.0.1~dev: CWD: E:\workspace\CICD\hualv-devops-frontend
10 silly lifecycle frontendProjectName@0.0.1~dev: Args: [ '/d /s /c', 'vue-cli-service serve' ]
11 silly lifecycle frontendProjectName@0.0.1~dev: Returned: code: 1  signal: null
12 info lifecycle frontendProjectName@0.0.1~dev: Failed to exec dev script
13 verbose stack Error: frontendProjectName@0.0.1 dev: `vue-cli-service serve`
13 verbose stack Exit status 1
13 verbose stack     at EventEmitter.<anonymous> (D:\nodejs\node_modules\npm\node_modules\npm-lifecycle\index.js:332:16)
13 verbose stack     at EventEmitter.emit (events.js:311:20)
13 verbose stack     at ChildProcess.<anonymous> (D:\nodejs\node_modules\npm\node_modules\npm-lifecycle\lib\spawn.js:55:14)
13 verbose stack     at ChildProcess.emit (events.js:311:20)
13 verbose stack     at maybeClose (internal/child_process.js:1021:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)
14 verbose pkgid frontendProjectName@0.0.1
15 verbose cwd E:\workspace\CICD\hualv-devops-frontend
16 verbose Windows_NT 10.0.19041
17 verbose argv "D:\\nodejs\\node.exe" "D:\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "run" "dev"
18 verbose node v12.16.1
19 verbose npm  v6.13.4
20 error code ELIFECYCLE
21 error errno 1
22 error frontendProjectName@0.0.1 dev: `vue-cli-service serve`
22 error Exit status 1
23 error Failed at the frontendProjectName@0.0.1 dev script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]

And the server console log something like this:

Socket connection closed prematurely.
System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake.
 ---> System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake.
   at System.Net.WebSockets.ManagedWebSocket.ThrowIfEOFUnexpected(Boolean throwOnPrematureClosure)
   at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken, Boolean throwOnPrematureClosure)
   at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TWebSocketReceiveResultGetter,TWebSocketReceiveResult](Memory`1 payloadBuffer, CancellationToken cancellationToken, TWebSocketReceiveResultGetter resultGetter)
   at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TWebSocketReceiveResultGetter,TWebSocketReceiveResult](Memory`1 payloadBuffer, CancellationToken cancellationToken, TWebSocketReceiveResultGetter resultGetter)
   at Microsoft.AspNetCore.Http.Connections.Internal.Transports.WebSocketsServerTransport.StartReceiving(WebSocket socket)

Okay,all is my fault.After carefully read the official documention again ,i found out the reason .I ignored the last one point:

Configure your server farm load balancing software for sticky sessions.

The solution for my situation is to configure nginx ingress to enable Sticky Session,like this:

    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/affinity-mode: balanced
    nginx.ingress.kubernetes.io/session-cookie-expires: "86400"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "86400"

Add config above to metadata:annotations node in ingress yaml