Zero Downtime: Upgrading MySQL Server in a Tungsten Cluster

Introduction

Tungsten Clustering allows for many types of maintenance to happen with no downtime at all.

This blog post will explore how to upgrade the actual MySQL Server on all cluster nodes with zero downtime. These same steps may be used to upgrade the MySQL server or patch the OS.

Zero-Downtime Upgrade Steps

This is a zero-downtime upgrade:

• Pick a replica node and shun it, and take all replication offline
  • cctrl> datasource NODENAME shun
  • tungsten@shell> trepctl -all-services offline
• Upgrade the MySQL Server on that node
• Bring up replication and verify health
  • tungsten@shell> trepctl -all-services online
  • tungsten@shell> trepctl services
• Ensure that the Replicator has time to catch up after being offline for a period of time
  • tungsten@shell> trepctl services
• Welcome the replica node back into the cluster
  • cctrl> datasource NODENAME welcome
• Repeat for the other Replica node
• Perform a switch of the Primary to one of the upgraded Replicas
  • cctrl> switch to NODENAME
• Upgrade the old Primary with the same steps
• Done!

The complete procedure is detailed here.

Zero-Downtime Best Practices

• The best practice is to leave the cluster in Automatic mode, because shunning a node takes care of preventing both failover to the node and Connector reads from it.

Using Maintenance mode degrades the service level, because in Maintenance mode, there is no High Availability (HA) providing automatic failover if the MySQL Server dies.

In Automatic mode, there is HA/automatic failover, because the Manager still provides quorum even though the node is shunned.

• Take the Replicator offline gracefully using `trepctl offline` before taking down the database.
• Bring the Replicator online and verify status using `trepctl online` and `trepctl services` or `trepctl status`.

Wrap-Up

In this post we summarized the zero-downtime upgrade procedure for MySQL Server in a Tungsten Cluster. We also covered best practices for upgrading.

Smooth sailing!