MYSQL crashing with signal 11

1

Previously I added same crashing with different error log. This time it shows different error. Could someone please help with what can be the failure here? I'm using NDB cluster and master master replication in same database.

And also I'm doing a load test which handles 2000TPS as peek time transaction. Having 2 mysql servers, 2 NDBD(Data Servers) and 1 server for cluster manager.

13:11:15 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=2501
max_threads=2500
thread_count=2008
connection_count=2002
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1124431 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f6560048a70
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
2018-02-19T20:12:02.670945+07:00 0 [Note] /usr/sbin/mysqld (mysqld 5.7.19-ndb-7.5.7-cluster-gpl-log) starting as process 18000 ...
2018-02-19T20:12:02.681263+07:00 0 [Note] InnoDB: PUNCH HOLE support available
2018-02-19T20:12:02.681325+07:00 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-02-19T20:12:02.681343+07:00 0 [Note] InnoDB: Uses event mutexes
2018-02-19T20:12:02.681353+07:00 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-02-19T20:12:02.681360+07:00 0 [Note] InnoDB: Compressed tables use zlib 1.2.3
2018-02-19T20:12:02.681377+07:00 0 [Note] InnoDB: Using Linux native AIO
2018-02-19T20:12:02.682863+07:00 0 [Note] InnoDB: Number of pools: 1
2018-02-19T20:12:02.683018+07:00 0 [Note] InnoDB: Using CPU crc32 instructions
2018-02-19T20:12:02.685593+07:00 0 [Note] InnoDB: Initializing buffer pool, total size = 15G, instances = 15, chunk size = 128M
2018-02-19T20:12:03.761039+07:00 0 [Note] InnoDB: Completed initialization of buffer pool
2018-02-19T20:12:03.878409+07:00 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2018-02-19T20:12:03.895334+07:00 0 [Note] InnoDB: Highest supported file format is Barracuda.
2018-02-19T20:12:03.974159+07:00 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 738148980146
2018-02-19T20:12:04.237405+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738154222592
2018-02-19T20:12:04.505813+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738159465472
2018-02-19T20:12:04.782026+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738164708352
2018-02-19T20:12:05.056884+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738169951232
2018-02-19T20:12:05.332870+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738175194112
2018-02-19T20:12:05.609434+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738180436992
2018-02-19T20:12:05.888033+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738185679872
2018-02-19T20:12:06.166643+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738190922752
2018-02-19T20:12:06.444935+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738196165632
2018-02-19T20:12:06.724184+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738201408512
2018-02-19T20:12:07.003352+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738206651392
2018-02-19T20:12:07.279073+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738211894272
2018-02-19T20:12:07.555518+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738217137152
2018-02-19T20:12:07.831187+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738222380032
2018-02-19T20:12:08.106033+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738227622912
2018-02-19T20:12:08.380741+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738232865792
2018-02-19T20:12:08.655991+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738238108672
2018-02-19T20:12:08.928785+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738243351552
2018-02-19T20:12:09.214037+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738248594432
2018-02-19T20:12:09.598199+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738253837312
2018-02-19T20:12:09.966482+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738259080192
2018-02-19T20:12:10.346543+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738264323072
2018-02-19T20:12:10.728390+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738269565952
2018-02-19T20:12:11.111287+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738274808832
2018-02-19T20:12:11.488422+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738280051712
2018-02-19T20:12:11.867041+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738285294592
2018-02-19T20:12:12.244510+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738290537472
2018-02-19T20:12:12.628907+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738295780352
2018-02-19T20:12:13.012033+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738301023232
2018-02-19T20:12:13.395482+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738306266112
2018-02-19T20:12:13.778862+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738311508992
2018-02-19T20:12:14.162134+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738316751872
2018-02-19T20:12:14.332603+07:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 738319200295
2018-02-19T20:12:14.333107+07:00 0 [Note] InnoDB: Database was not shutdown normally!
2018-02-19T20:12:14.333130+07:00 0 [Note] InnoDB: Starting crash recovery.
2018-02-19T20:12:15.342683+07:00 0 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
2018-02-19T20:12:18.278539+07:00 0 [Note] InnoDB: Apply batch completed
2018-02-19T20:12:18.278599+07:00 0 [Note] InnoDB: Last MySQL binlog file position 0 1248259, file name mysql-bin.000489
2018-02-19T20:12:18.386336+07:00 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2018-02-19T20:12:18.386380+07:00 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2018-02-19T20:12:18.386427+07:00 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2018-02-19T20:12:18.418626+07:00 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2018-02-19T20:12:18.419531+07:00 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2018-02-19T20:12:18.419562+07:00 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2018-02-19T20:12:18.419984+07:00 0 [Note] InnoDB: Waiting for purge to start
2018-02-19T20:12:18.470102+07:00 0 [Note] InnoDB: 5.7.19 started; log sequence number 738319200295
2018-02-19T20:12:18.470117+07:00 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 14592ms. The settings might not be optimal. (flushed=0 and evicted=0, during the time.)
2018-02-19T20:12:18.470764+07:00 0 [Note] InnoDB: Loading buffer pool(s) from /logs/mysql-new-/ib_buffer_pool
2018-02-19T20:12:18.471031+07:00 0 [Note] Plugin 'FEDERATED' is disabled.
2018-02-19T20:12:18.680524+07:00 0 [Note] NDB: NodeID is 4, management server '10.81.133.5:1186'
2018-02-19T20:12:19.047371+07:00 0 [Note] InnoDB: Buffer pool(s) load completed at 180219 20:12:19
2018-02-19T20:12:19.081036+07:00 0 [Note] NDB[0]: NodeID: 4, all storage nodes connected
2018-02-19T20:12:19.081218+07:00 0 [Note] NDB Binlog: Starting...
2018-02-19T20:12:19.081241+07:00 0 [Note] NDB Util: Starting...
2018-02-19T20:12:19.081321+07:00 0 [Note] NDB Index Stat: Starting...
2018-02-19T20:12:19.081392+07:00 1 [Note] NDB Binlog: Started
2018-02-19T20:12:19.081401+07:00 0 [Note] NDB Index Stat: Wait for server start completed
2018-02-19T20:12:19.081416+07:00 1 [Note] NDB Binlog: Setting up
2018-02-19T20:12:19.081401+07:00 2 [Note] NDB Util: Wait for server start completed
2018-02-19T20:12:19.081550+07:00 1 [Note] NDB Binlog: Created schema Ndb object, reference: 0x80040004, name: 'Ndb Binlog schema change monitoring'
2018-02-19T20:12:19.081617+07:00 1 [Note] NDB Binlog: Created injector Ndb object, reference: 0x80050004, name: 'Ndb Binlog data change monitoring'
2018-02-19T20:12:19.081631+07:00 1 [Note] NDB Binlog: Setup completed
2018-02-19T20:12:19.081665+07:00 1 [Note] NDB Binlog: Wait for server start completed
2018-02-19T20:12:19.158141+07:00 0 [Note] Recovering after a crash using mysql-bin
2018-02-19T20:12:19.166999+07:00 0 [Note] Starting crash recovery...
2018-02-19T20:12:19.167061+07:00 0 [Note] Crash recovery finished.
2018-02-19T20:12:19.182863+07:00 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2018-02-19T20:12:19.183128+07:00 0 [Warning] CA certificate ca.pem is self signed.
2018-02-19T20:12:19.184762+07:00 0 [Note] Server hostname (bind-address): '*'; port: 3306
2018-02-19T20:12:19.184813+07:00 0 [Note] IPv6 is available.
2018-02-19T20:12:19.184847+07:00 0 [Note]   - '::' resolves to '::';
2018-02-19T20:12:19.184878+07:00 0 [Note] Server socket created on IP: '::'.
2018-02-19T20:12:19.191456+07:00 0 [Warning] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-02-19T20:12:19.191531+07:00 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-02-19T20:12:19.191572+07:00 0 [Warning] 'user' entry 'sqladmin@localhost' ignored in --skip-name-resolve mode.
2018-02-19T20:12:19.191603+07:00 0 [Warning] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-02-19T20:12:19.191616+07:00 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-02-19T20:12:19.191625+07:00 0 [Warning] 'db' entry 'css_cluster_ndb sqladmin@localhost' ignored in --skip-name-resolve mode.
2018-02-19T20:12:19.198119+07:00 0 [Warning] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-02-19T20:12:19.198160+07:00 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-02-19T20:12:19.824532+07:00 3 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
2018-02-19T20:12:19.825150+07:00 4 [Warning] Slave SQL for channel '': If a crash happens this configuration does not guarantee that the relay log info will be consistent, Error_code: 0
2018-02-19T20:12:19.828701+07:00 3 [Note] Slave I/O thread for channel '': connected to master 'root@10.80.172.199:3306',replication started in log 'mysql-bin.000443' at position 602368754
2018-02-19T20:12:19.830389+07:00 0 [Note] Event Scheduler: Loaded 1 event
2018-02-19T20:12:19.830518+07:00 5 [Note] Event Scheduler: scheduler thread started with id 5
2018-02-19T20:12:19.831053+07:00 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.7.19-ndb-7.5.7-cluster-gpl-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Cluster Community Server (GPL)
2018-02-19T20:12:19.831112+07:00 0 [Note] NDB Index Stat: Wait for cluster to start
2018-02-19T20:12:19.831134+07:00 2 [Note] NDB Util: Wait for cluster to start
2018-02-19T20:12:19.831155+07:00 1 [Note] NDB Binlog: Check for incidents
2018-02-19T20:12:19.831174+07:00 0 [Note] NDB Index Stat: Started
2018-02-19T20:12:19.831192+07:00 1 [ERROR] mysqld startup An incident event has been written to the binary log which will stop the slaves.
2018-02-19T20:12:19.831219+07:00 2 [Note] NDB Binlog: Ndb tables initially read only.
2018-02-19T20:12:19.831234+07:00 2 [Note] NDB Util: Started
2018-02-19T20:12:19.831301+07:00 0 [Note] NDB Index Stat: created Ndb object 'Ndb Index Stat', ref: 0x80070004
mysql
asked on Stack Overflow Feb 19, 2018 by chinthakaS

0 Answers

Nobody has answered this question yet.


User contributions licensed under CC BY-SA 3.0