MySQL Cluster error: 1005: Can't create table 'ndb_schema' (errno: 13)
I'm having troubles with my MySQL Cluster Test-setup:
I have two machines running Ubuntu and Docker (using a multi-host network) and each node (ndb_mgm, ndbd and mysqld) runs via Docker. The setup is as follows:
Machine 1:
- ndb_mgmd
- ndbd1
- mysqld1
Machine 2:
- ndbd2
- mysqld2
When I start the mysqld on one machine, it connects to the manager and wants to initiate the cluster, but its unable to create the table, The whole NDB log on a mysqld instance after a fresh start:
2016-08-02 09:08:57 135 [Note] NDB: Changed global value of binlog_format from STATEMENT to MIXED
2016-08-02 09:08:58 135 [Note] NDB: NodeID is 4, management server 'ndb_mgmd:1186'
2016-08-02 09:08:58 135 [Note] NDB[0]: NodeID: 4, all storage nodes connected
2016-08-02 09:08:58 135 [Warning] NDB: server id set to zero - changes logged to bin log with server id zero will be logged with another server id by slave mysqlds
2016-08-02 09:08:58 135 [Note] NDB Binlog: Starting...
2016-08-02 09:08:58 135 [Note] NDB Binlog: Started
2016-08-02 09:08:58 135 [Note] NDB Binlog: Setting up
2016-08-02 09:08:58 135 [Note] NDB Binlog: Created schema Ndb object, reference: 0x80040004, name: 'Ndb Binlog schema change monitoring'
2016-08-02 09:08:58 135 [Note] NDB Binlog: Created injector Ndb object, reference: 0x80050004, name: 'Ndb Binlog data change monitoring'
2016-08-02 09:08:58 135 [Note] NDB Binlog: Setup completed
2016-08-02 09:08:58 135 [Note] NDB Binlog: Wait for server start completed
2016-08-02 09:08:58 135 [Note] NDB Util: Starting...
2016-08-02 09:08:58 135 [Note] NDB Util: Wait for server start completed
2016-08-02 09:08:58 135 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: bf07c00a-5890-11e6-83a3-02420a000a1e.
2016-08-02 09:08:58 135 [Note] NDB Index Stat: Starting...
2016-08-02 09:08:58 135 [Note] NDB Index Stat: Wait for server start completed
2016-08-02 09:08:58 135 [Note] Server hostname (bind-address): '*'; port: 3306
2016-08-02 09:08:58 135 [Note] IPv6 is available.
2016-08-02 09:08:58 135 [Note] - '::' resolves to '::';
2016-08-02 09:08:58 135 [Note] Server socket created on IP: '::'.
2016-08-02 09:08:58 135 [Note] Event Scheduler: Loaded 0 events
2016-08-02 09:08:58 135 [Note] /usr/local/mysql-cluster-gpl-7.4.12-linux-glibc2.5-x86_64/bin/mysqld: ready for connections.
Version: '5.6.31-ndb-7.4.12-cluster-gpl' socket: '/tmp/mysql.sock' port: 3306 MySQL Cluster Community Server (GPL)
2016-08-02 09:08:58 135 [Note] NDB Util: Wait for cluster to start
2016-08-02 09:08:58 135 [Note] NDB Util: Started
2016-08-02 09:08:58 135 [Note] NDB Binlog: Check for incidents
2016-08-02 09:08:58 135 [Note] NDB Binlog: Wait for cluster to start
2016-08-02 09:08:58 135 [Note] NDB Index Stat: Wait for cluster to start
2016-08-02 09:08:58 135 [Note] NDB Index Stat: Started
2016-08-02 09:08:58 135 [Note] created index stats Ndb object: reference 0x80070004, name: 'Ndb Index Statistics monitoring'
2016-08-02 09:08:58 135 [Note] NDB: Creating mysql.ndb_schema
2016-08-02 09:08:58 135 [ERROR] NDB: Query 'CREATE TABLE IF NOT EXISTS mysql.ndb_schema ( db VARBINARY(63) NOT NULL,name VARBINARY(63) NOT NULL,slock BINARY(32) NOT NULL,query BLOB NOT NULL,node_id INT UNSIGNED NOT NULL,epoch BIGINT UNSIGNED NOT NULL,id INT UNSIGNED NOT NULL,version INT UNSIGNED NOT NULL,type INT UNSIGNED NOT NULL,PRIMARY KEY USING HASH (db,name) ) ENGINE=NDB CHARACTER SET latin1' failed, error: 1005: Can't create table 'ndb_schema' (errno: 13)
The last line shows, that its unable to create a table:
NDB: Query 'CREATE TABLE ...' failed, error: 1005: Can't create table 'ndb_schema' (errno: 13)
If i checkout the ndb_mgm configuration, every node seem connected correctly:
ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=2 @10.0.10.20 (mysql-5.6.31 ndb-7.4.12, Nodegroup: 0, *)
id=3 @10.0.10.21 (mysql-5.6.31 ndb-7.4.12, Nodegroup: 0)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @10.0.10.10 (mysql-5.6.31 ndb-7.4.12)
[mysqld(API)] 2 node(s)
id=4 @10.0.10.30 (mysql-5.6.31 ndb-7.4.12)
id=5 @10.0.10.31 (mysql-5.6.31 ndb-7.4.12)
The configuration for ndb_mgmd:
[ndbd default]
NoOfReplicas=2
DataMemory=80M
IndexMemory=18M
[ndb_mgmd]
hostname=10.0.10.10
datadir=/var/lib/mysql-cluster
[ndbd]
hostname=10.0.10.20
datadir=/usr/local/mysql/data
[ndbd]
hostname=10.0.10.21
datadir=/usr/local/mysql/data
[mysqld]
hostname=10.0.10.30
[mysqld]
hostname=10.0.10.31
Each mysqld is configured as followed:
[mysqld]
ndbcluster
log-error=/var/log/mysql/error.log
Running the status of the cluster on a mysqld instance shows no errors:
mysql> SHOW ENGINE NDB STATUS
-> ;
+------------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| Type | Name | Status |
+------------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| ndbcluster | connection | cluster_node_id=4, connected_host=ndb_mgmd, connected_port=1186, number_of_data_nodes=2, number_of_ready_data_nodes=2, connect_count=0 |
| ndbcluster | NdbTransaction | created=2, free=2, sizeof=368 |
| ndbcluster | NdbOperation | created=4, free=4, sizeof=944 |
| ndbcluster | NdbIndexScanOperation | created=0, free=0, sizeof=1152 |
| ndbcluster | NdbIndexOperation | created=0, free=0, sizeof=952 |
| ndbcluster | NdbRecAttr | created=0, free=0, sizeof=88 |
| ndbcluster | NdbApiSignal | created=16, free=16, sizeof=144 |
| ndbcluster | NdbLabel | created=0, free=0, sizeof=200 |
| ndbcluster | NdbBranch | created=0, free=0, sizeof=32 |
| ndbcluster | NdbSubroutine | created=0, free=0, sizeof=72 |
| ndbcluster | NdbCall | created=0, free=0, sizeof=24 |
| ndbcluster | NdbBlob | created=0, free=0, sizeof=496 |
| ndbcluster | NdbReceiver | created=0, free=0, sizeof=128 |
| ndbcluster | NdbLockHandle | created=0, free=0, sizeof=48 |
| ndbcluster | binlog | latest_epoch=0, latest_trans_epoch=200377404227594, latest_received_binlog_epoch=0, latest_handled_binlog_epoch=0, latest_applied_binlog_epoch=0 |
+------------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
15 rows in set (0.00 sec)
If I want to create a table manually on the cluster, i get the following error:
mysql> CREATE DATABASE cluster;
Query OK, 1 row affected (0.00 sec)
mysql> CREATE TABLE cluster_test (name VARCHAR(20), value VARCHAR(20)) ENGINE=ndbcluster;
ERROR 1296 (HY000): Got error 4009 'Cluster Failure' from NDBCLUSTER
mysql> show warnings;
+-------+------+--------------------------------------------------+
| Level | Code | Message |
+-------+------+--------------------------------------------------+
| Error | 1296 | Got error 4009 'Cluster Failure' from NDBCLUSTER |
+-------+------+--------------------------------------------------+
1 row in set (0.00 sec)
I don't know where to get a detailed error description. If I'm right Errno 13 is usually some kind of permission error, but every node has the right permissions to the folders (they actually wrote some files). If you need a detailed folder permission list, please let me know.
Thanks in advice!
asked on Stack Overflow Aug 2, 2016 by 
manixx
manixx1 Answer
Currently the problem is not solved, but after some research the problem seems the AUFS storage inside Docker. I described here the issue.
I got it working using less layers inside the image. I combined a lot of commands inside one RUN directive.
Here are the Dockerfiles that work for me:
mysql daemon:
FROM ubuntu
RUN apt-get update && \
apt-get install -y libcam-pdf-perl build-essential libaio1 libaio-dev && \
cpan File::Copy \
Sys::Hostname \
Data::Dumper
// .. install mysql cluster files to /usr/local/mysql
WORKDIR /usr/local/mysql
RUN groupadd mysql && \
useradd -g mysql -s /bin/false mysql && \
chown -R root . && \
chown -R mysql data && \
chgrp -R mysql . && \
scripts/mysql_install_db --user=mysql
ENV PATH=/usr/local/mysql/bin:$PATH
USER mysql
ndbd:
FROM ubuntu
// .. install mysql cluster files to /usr/local/bin
RUN chmod +x /usr/local/bin/ndb* && \
mkdir -p /usr/local/mysql/data
ndb_mgm:
FROM ubuntu
// .. install mysql cluster files to /usr/local/bin
RUN chmod +x /usr/local/bin/ndb_mgm* && \
mkdir -p /var/lib/mysql-cluster
Maybe this helps someone!
answered on Stack Overflow Oct 3, 2016 by manixx
manixxUser contributions licensed under CC BY-SA 3.0