Rejoining or Bootstrapping MySQL Galera Cluster Nodes After Shutdown
I have a 3 Node MySQL Galera Cluster that faced a shutdown on all 3 nodes at the same time, luckily this is only a testing environment, but at that time it was down and did not want to start up.
When trying to start MySQL the only error visible was:
$ /etc/init.d/mysql restart
* MySQL server PID file could not be found!
........ * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
* Failed to restart server.
At that time I can see that the galera port is started, but not mysql:
$ ps aux | grep mysql
root 23580 0.0 0.0 45081800 pts/0 S 00:37 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/run/mysqld/mysqld.pid
mysql 24144 0.7 22.2 1185116455660 pts/0 Sl 00:38 0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 --wsrep_start_position=long:string
$ netstat -tulpn
Active Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 00 0.0.0.0:4567 0.0.0.0:* LISTEN 25507/mysqld
More in detail is explained on a SeveralNines Blog Post, but due to the fact that all the nodes left the cluster, one of the nodes needs to be started as a referencing point, before the other nodes can rejoin or bootstrapped to the cluster.
Rejoining the Cluster
Consult the blog for more information, but from my end, I had a look at the node with the highest seqno and then updated safe_to_bootstrap to 1: