--- title: "MariaDB MaxScale 6.0 Native Clustering" publish_date: 2022-08-24 updated_date: 2022-08-26 author: "Pon Suresh Pandian" tags: - name: "Clustering" url: "/resources/blog/tag/clustering.md" - name: "High Availability" url: "/resources/blog/tag/high-availability.md" - name: "MaxScale" url: "/resources/blog/tag/maxscale.md" --- # MariaDB MaxScale 6.0 Native Clustering [MaxScale 6.0](https://staging-mdb.com/kb/en/mariadb-maxscale-6-mariadb-maxscale-60-release-notes-2021-06-30/) introduced native clustering, allowing several [MaxScale](https://staging-mdb.com/products/maxscale/) instances to communicate and share configuration updates with each other. In this blog post, We are going to explain how the feature works and demonstrate how to implement it. ### Common MaxScale Topologies **Centralized MaxScale Server** This is the most common installation, where MaxScale is between application servers and the database. It is simple, but without any high availability. If MaxScale goes down, the application loses all connectivity to the database. ![Centralized MaxScale server](https://staging-mdb.com/wp-content/uploads/2022/08/maxscale-clustering-blog-img1.png) **MaxScale HA setup** Another common setup is to install MaxScale onto each application server or seperate servers. The benefit of this setup is that the loss of one MaxScale/app server will not bring down the entire application. ![MaxScale HA setup](https://staging-mdb.com/wp-content/uploads/2022/08/maxscale-clustering-blog-img2.png) ### Dynamic Configuration Changes MaxScale allows runtime config changes via the MaxScale GUI, MaxCtrl or the REST API. Those runtime changes are saved on disk, inside the [persistdir](https://staging-mdb.com/docs/reference/mxs/global-parameters/persistdir/) directory. If you have multiple MaxScales you have to do any runtime changes on every MaxScale manually. **Internal workflow** Basically this config sync involves six stages. ![Internal workflow](https://staging-mdb.com/wp-content/uploads/2022/08/maxscale-clustering-blog-img3.png) **Runtime Changes** All the runtime changes are done by MaxScale GUI, MaxCtrl or the REST API. Those changes are saved on disk. **Data Storage** The configuration information is stored in “maxscale\_config” table under mysql schema. It contains: - the cluster name (the cluster name from MaxScale configuration), - the configuration version number. This version number will increment for each change, - the actual configuration data which is stored in JSON format. The version number starts from 0 and the first dynamic change causes it to be set to 1. This means the first update stored in the database will have the version number 1 and all subsequent configuration updates increment the version. **Updating the Configuration** Before applying configuration changes, the current config version is selected from the database by using `SELECT ... FOR UPDATE` query. If the config version is the same as the version stored in MaxScale, the update can proceed. If the config version is not the same, an error is returned to the client. **Version Check** Once modifications are done, MaxScale attempts to update the row on the “maxscale\_config” table. Here, the `SELECT ... FOR UPDATE` query reads the config version. If it succeeds, it will update the row. If the version check fails due to deadlock or any conflicts in that transaction MaxScale returns error to the client. It guarantees that only one MaxScale succeeds in updating the configuration. A MaxScale that fails to perform the update will return an error to the client and read the new configuration from the database. The runtime configuration changes are stored in JSON format on disk. This is the “cached configuration”. If this cached configuration is available, it will be used instead of the static configuration files(/etc/maxscale.cnf). This allows the configuration changes to persist through a crash as well as through temporary outages. If the transition from the current configuration to the configuration stored in the cluster fails, the cached configuration is discarded. This guarantees that only one attempt is made to start with a cached configuration and all subsequent startups use the static configuration files. **Configuration Synchronization** MaxScale periodically polls the database to get the latest configuration version. If the database config version is newer than the current MaxScale config, MaxScale checks the difference between the new and current configurations and updates the configuration by applying only the new changes. If the configuration update is successful, MaxScale is in sync with other MaxScale nodes. If the configuration update fails, MaxScale attempts to roll back the partial changes. **Rollback** If the configuration update completes on MaxScale but fails to be committed to the database, MaxScale will attempt to revert the local configuration change. If this attempt fails, MaxScale will discard the cached configuration and abort the process. When synchronizing with the cluster, if MaxScale fails to apply a configuration retrieved from the cluster, it attempts to revert the configuration to the previous version. If MaxScale is successful in reverting the configuration to the previous version, the failed configuration update is ignored. If the failed configuration update cannot be reverted, the MaxScale configuration will be in an indeterminate state, and MaxScale will discard the cached configuration and abort the process. As an example, let’s assume my current configuration version is 2. - **If the configuration update is successful**, my configuration version will be 3 . - **If the update fails, but rollback is successful**, the configuration version will change from 3 to 2. - **If both update and rollback fail**, MaxScale discards all the versions and restarts the process using the static configuration file (/etc/maxscale.cnf). **Implementation** To start we need to get at least 2 nodes. Download and install MaxScale 6.0 or higher and start a clean version. Here we have a 3 node Galera cluster and MaxScale 6.4.0 was installed on all the database nodes. `node-1 - 172.31.83.144` `node-2 - 172.31.81.34` `node-3 - 172.31.85.115` `Virtual IP - 172.31.xx.xx` ![Implementation](https://staging-mdb.com/wp-content/uploads/2022/08/maxscale-clustering-blog-img4-1024x333.png) When configuring MaxScale synchronization for the first time, the same static configuration files should be used on all MaxScale instances that use same cluster value of “[config\_sync\_cluster](https://staging-mdb.com/docs/reference/mxs/module-parameters/config_sync_cluster/)” must be the same on all MaxScale instances and the cluster (i.e. the monitor) pointed by it and its servers must be the same in every configuration. For making a MaxScale cluster we need to create new users on database nodes to synchronize configuration changes across MaxScale instances. This user must have the following grants: ``` MariaDB> grant SELECT, INSERT, UPDATE, CREATE ON mysql.maxscale_config to 'sync_user'@'%'; ``` And we need to add the above user details on MaxScale config under `[maxscale]` tag. Example: ``` config_sync_cluster="Galera-Monitor" config_sync_user=sync_user config_sync_password=R00t@123 ``` **Sample Config** This is our sample config. The Configuration file should be the same on all MaxScale nodes. ``` $ cat /etc/maxscale.cnf # Global parameters [maxscale] threads=auto config_sync_cluster="Galera-Monitor" config_sync_user=sync_user config_sync_password=R00t@123 admin_secure_gui=false admin_host=0.0.0.0 # Server definitions [node1] type=server address=172.31.83.144 port=3306 protocol=MariaDBBackend [node2] type=server address=172.31.81.34 port=3306 protocol=MariaDBBackend [node3] type=server address=172.31.85.115 port=3306 protocol=MariaDBBackend # Monitor for the servers [Galera-Monitor] type=monitor module=galeramon servers=node1,node2,node3 user=max_user password=R00t@123 monitor_interval=2000 # Service definitions ``` Here we are using a read/write split router to route the application traffic. MaxScale node1 handles all the writes and node2 and node3 will handle all the reads. ``` # ReadWriteSplit [Read-Write-Service] type=service router=readwritesplit servers=node1,node2,node3 user=max_user password=R00t@123 # Listener definitions for the services [Read-Write-Listener] type=listener service=Read-Write-Service protocol=MariaDBClient port=4006 [root@ip-172-31-83-144 centos]# maxctrl show maxscale | grep -i 'config_sync' │ │ "config_sync_cluster": "Galera-Monitor", │ │ │ "config_sync_interval": "5000ms", │ │ │ "config_sync_password": "*****", │ │ │ "config_sync_timeout": "10000ms", │ │ │ "config_sync_user": "max_user", │ ``` Currently we are using default values for these `config_sync_interval` and `config_sync_timeout` parameters. The config has been added. Internally, all nodes are monitoring and communicating with all the other MaxScale nodes. Let’s add a new node to `[maxscale]` without restarting the service and see how this configuration changes replicating to other nodes. ``` [root@ip-172-31-83-144 centos]# maxctrl create server node4 172.31.80.237 3306 OK ``` New node is added under `[maxscale]`, but MaxScale won’t perform any health checks on this node. We need to add this node4 to “galera-monitor”. ``` [root@ip-172-31-83-144 centos]# maxctrl link monitor Galera-Monitor node4 OK ``` Node4 was added under “galera-monitor”. Let’s add this node to the Read-Write split service. ``` [root@ip-172-31-83-144 centos]# maxctrl link service Read-Write-Service node4 OK [root@ip-172-31-83-144 centos]# maxctrl list servers ┌────────┬───────────────┬──────┬─────────────┬─────────────────┬──────────────┐ │ Server │ Address       │ Port │ Connections │ State           │ GTID         │ ├────────┼───────────────┼──────┼─────────────┼─────────────────┼──────────────┤ │ node4  │ 172.31.80.237 │ 3306 │ 0           │ Slave, Running  │              │ ├────────┼───────────────┼──────┼─────────────┼─────────────────┼──────────────┤ │ node3  │ 172.31.85.115 │ 3306 │ 0           │ Slave, Running  │ 3-3-5,1-1-47 │ ├────────┼───────────────┼──────┼─────────────┼─────────────────┼──────────────┤ │ node2  │ 172.31.81.34  │ 3306 │ 0           │ Slave, Running  │ 2-2-5,1-1-47 │ ├────────┼───────────────┼──────┼─────────────┼─────────────────┼──────────────┤ │ node1  │ 172.31.83.144 │ 3306 │ 0           │ Master, Running │ 1-1-47       │ └────────┴───────────────┴──────┴─────────────┴─────────────────┴──────────────┘ [root@ip-172-31-83-144 centos]# maxctrl list services ┌────────────────────┬────────────────┬─────────────┬───────────────────┬────────────────────────────┐ │ Service            │ Router         │ Connections │ Total Connections │ Targets                    │ ├────────────────────┼────────────────┼─────────────┼───────────────────┼────────────────────────────┤ │ Read-Write-Service │ readwritesplit │ 0           │ 0                 │ node1, node2, node3, node4 │ └────────────────────┴────────────────┴─────────────┴───────────────────┴────────────────────────────┘ ``` Whenever the MaxScale configuration is modified at runtime, the latest configuration is stored in the database cluster in the `mysql.maxscale_config` table. A local copy of the configuration is stored in the data directory to allow MaxScale to function even if a connection to the cluster cannot be made. By default this file is stored at */var/lib/maxscale/maxscale-config.json*. While starting, the MaxScale service checks if a local version of this configuration exists. If it does and it is a valid cached configuration, the static configuration file as well as any other generated configuration files are ignored. The exception is the `[maxscale]` section of the main static configuration file which is always read. Each configuration has a version number with the initial configuration being version 0. Each time the configuration is modified, the version number is incremented. This version number is used to detect when MaxScale needs to update its configuration. Now new nodes were added under `[maxscale]`. Let’s verify this change is replicated to other MaxScale nodes. From MaxScale node2 and MaxScale node3: ``` # maxctrl list servers ┌────────┬───────────────┬──────┬─────────────┬─────────────────┬──────────────┐ │ Server │ Address       │ Port │ Connections │ State           │ GTID         │ ├────────┼───────────────┼──────┼─────────────┼─────────────────┼──────────────┤ │ node4  │ 172.31.80.237 │ 3306 │ 0           │ Slave, Running  │              │ ├────────┼───────────────┼──────┼─────────────┼─────────────────┼──────────────┤ │ node3  │ 172.31.85.115 │ 3306 │ 0           │ Slave, Running  │ 3-3-5,1-1-47 │ ├────────┼───────────────┼──────┼─────────────┼─────────────────┼──────────────┤ │ node2  │ 172.31.81.34  │ 3306 │ 0           │ Slave, Running  │ 2-2-5,1-1-47 │ ├────────┼───────────────┼──────┼─────────────┼─────────────────┼──────────────┤ │ node1  │ 172.31.83.144 │ 3306 │ 0           │ Master, Running │ 1-1-47       │ └────────┴───────────────┴──────┴─────────────┴─────────────────┴──────────────┘ ``` From the node2 and node3 MaxScale log ``` 2022-06-19 08:09:03   notice : (ConfigManager); Updating to configuration version 5 2022-06-19 08:28:43   notice : (ConfigManager); Updating to configuration version 6 2022-06-19 08:28:43   notice : (ConfigManager); Added 'node4' to 'Galera-Monitor' 2022-06-19 08:28:43   notice : 'node4' sent version string '10.6.8-MariaDB'. Detected type: 'MariaDB', version: 10.6.8. ``` **How to Verify the MaxScale Cluster is Synced?** The output of `maxctrl show maxscale` contains the `Config Sync` field with information about the current configuration state of the local MaxScale as well as the state of any other nodes using this cluster. ``` [root@ip-172-31-83-144 centos]# maxctrl show maxscale | grep -i 'Config Sync' -A10 │ Config Sync  │ {                                                           │ │              │     "checksum": "e2f0041a344c41065dc94f5c6d8802f40c396f2a", │ │              │     "nodes": {                                              │ │              │         "ip-172.31.83.144.ec2.internal": "OK",              │ │              │         "ip-172.31.81.34.ec2.internal": "OK",               │ │              │         "ip-172.31.85.115.ec2.internal": "OK",              │ │              │         "ip-172.31.80.237.ec2.internal": "OK"               │ │              │     },                                                      │ │              │     "origin": "172.31.83.144.ec2.internal",                 │ │              │     "status": "OK",                                         │ │              │     "version": 8                                            │ │              │ }                                                           │ ``` The version field is the logical configuration version and the origin is the node that originates the latest configuration change. The checksum field is the checksum of the logical configuration and can be used to compare whether two MaxScale instances are in the same configuration state. The nodes field contains the status of each MaxScale instance mapped to the hostname of the server. This field is updated whenever MaxScale reads the configuration from the cluster and can thus be used to detect which MaxScale have updated their configuration. **Let’s do Changes from MaxScale GUI** Currently the `monitor_interval` is 2000ms which means around 2 seconds. We are going to change this `monitor_interval` from 2000 to 5000 ms on the GUI. Before update: ![Monitor interval before update](https://staging-mdb.com/wp-content/uploads/2022/08/maxscale-clustering-blog-img5.png) ``` [root@ip-172-31-83-144 centos]# maxctrl show monitors | grep -i monitor_interval │                     │     "monitor_interval": 2000,                       │ ``` ![Monitor interval update](https://staging-mdb.com/wp-content/uploads/2022/08/maxscale-clustering-blog-img6.png) After update: ![Monitor interval after update](https://staging-mdb.com/wp-content/uploads/2022/08/maxscale-clustering-blog-img7.png) From all the MaxScale nodes: ``` #  maxctrl show monitors | grep -i monitor_interval │                     │     "monitor_interval": 5000,                       │ [root@ip-172-31-83-144 centos]# maxctrl show maxscale | grep -i 'Config Sync' -A10 │ Config Sync  │ {                                                           │ │              │     "checksum": "aea6f6ef5a2fa869041981b93493aa371d7eb8f5", │ │              │     "nodes": {                                              │ │              │         "ip-172.31.83.144.ec2.internal": "OK",              │ │              │         "ip-172.31.81.34.ec2.internal": "OK",               │ │              │         "ip-172.31.85.115.ec2.internal": "OK",              │ │              │         "ip-172.31.80.237.ec2.internal": "OK"               │ │              │     },                                                      │ │              │     "origin": "172.31.83.144.ec2.internal",                 │ │              │     "status": "OK",                                         │ │              │     "version": 9                                            │ │              │ }                                                           │ ``` **Disabling MaxScale Cluster Synchronization** To disable configuration synchronization, remove config\_sync\_cluster from the configuration file or set it to an empty string for example, `config_sync_cluster=""`. This can be done at runtime with MaxCtrl by passing an empty string to `config_sync_cluster`. Example ``` [root@ip-172-31-83-144 centos]# maxctrl alter maxscale config_sync_cluster "" OK ``` To reset the MaxScale synchronization, - Stop all MaxScale instances. - Remove the cached configuration file stored at `/var/lib/maxscale/maxscale-config.json` on all MaxScale instances. - Drop the `mysql.maxscale_config` table. - Start all MaxScale instances. **Limitations** The synchronization only affects the MaxScale configuration. The state of objects or any external files (e.g. Cache filter rules, TLS certificates, data masking rules) are not synchronized by this mechanism. ### For More Information - [MariaDB MaxScale documentation](https://staging-mdb.com/docs/products/mariadb-maxscale/) - [MariaDB MaxScale Release Notes 6.4.1](https://staging-mdb.com/docs/release-notes/mariadb-maxscale-6/6-4-1/) - [MariaDB MaxScale Release Notes 22.08.0 (beta)](https://staging-mdb.com/docs/release-notes/mariadb-maxscale-22-08/22-08-0/)