Solving Customer Problems
One of the things I love to do as part of my job at Continuent is to solve customer issues. Recently, we were asked whether there was a way to temporarily remove a node from a cluster. The cctrl command datasource shun was one answer, but it could definitely be improved: shunning a node is immediate and will kill all ongoing client connections to it.
Flexible Proxy
Our Database proxy, Tungsten Connector, is able to control connections by sending them to a particular node in a cluster. You see where I’m going: it can also prevent new connections from being established to a given node, and wait for all ongoing connections to finish or kill them after a timeout, just like it does for the switch command.
There we go, all the bits and pieces were available for us to create the datasource {nodename} drain command:
cctrl> datasource db2 drain 30
DataSource 'u1@nyc' set to SHUNNED
Another Issue, Another Solution
In the example above, we ask all connectors to stop routing connection to db2, wait at most 30 seconds and if there are any remaining connections, sever them. Without a timeout, the connectors would wait indefinitely for connections to disconnect by themselves. This brings us to a question: how do I know how many connections are left on this data source?
This is where we introduced the command datasource {nodename} connections:
cctrl> datasource db2 connections
17
The output is intentionally rough, allowing easy scripting with the response. For those humans who want more details, the -l flag (for “long” output) comes to the rescue:
> datasource db2 connections -l
17
connector@db3[16305] (10)
connector@db2[20304] (5)
connector@db1[12353] (2)
There you can see that the Connector on db1 holds 2 connections, the Connector on db2 has 5, and there are 10 on db3.
Bringing Back the Node
As you might have noticed, the data source that is “drained” is actually turned into the SHUNNED state, but a special flag (“lastShunReason”) is set to “DRAIN-CONNECTIONS”.
The way to bring it back to the cluster and allow new connections to it is the same as for a regular shunned node - you welcome it back into the cluster with datasource {nodename} welcome:
> datasource db2 welcome
WARNING: This is an expert-level command:
Incorrect use may cause data corruption
or make the cluster unavailable.
Do you want to continue? (y/n)> y
DataSource 'u1@nyc' is now OFFLINE
The automatic policy will put it online right away, and connectors will route connections back to it.
Through the APIv2
For those who’d like to integrate this in their programmatic environment, the corresponding API calls have been added to the Manager, for example:
POST /api/v2/manager/control/service/west/datasource/db2/drain?timeoutSecs=30
All docs available on our website:
- https://docs.continuent.com/apiv2docs/manager/resource_ControlResourceV2.html#resource_ControlResourceV2_drainDatasource_POST
- https://docs.continuent.com/apiv2docs/manager/resource_ControlResourceV2.html#resource_ControlResourceV2_welcomeDatasource_POST
Docs and Beyond
Note that the drain and connections commands have been introduced in version 7.0.2 released on December 9th. The documentation for the drain command is available here:
https://docs.continuent.com/tungsten-clustering-7.0/cmdline-tools-cctrl-commands.html#cmdline-tools-cctrl-commands-datasource-drain
Worth mentioning, there is a similar command for the maintenance of a Tungsten Connector node: connector drain command has been made an alias for the connector graceful-stop explained in this blog post. Enjoy the reading!
Comments
Add new comment