Summary
The Short Version
There are many ways to watch your resources, and Prometheus is one of the more popular.
Due to customer demand, Continuent has embedded the Prometheus exporter functionality into our core products (Manager, Replicator and Connector) starting with version 7.0.0, which is due out shortly.
This blog post covers:
- Getting metrics from each component
- Various metrics available
- Using the bundled command-line tool `
tmonitor` - Customizing the exporter configuration
- How to enable the exporters
- Documentation & Resources
Getting Metrics From Each Component
It is even easier to gather metrics from the enabled Prometheus exporters - just use the curl command or a browser.
By default, the exporters listen on the following ports:
| 8091 | Tungsten Replicator |
| 8092 | Tungsten Manager |
| 8093 | Tungsten Connector |
To manually test any exporter from the command line, use the form of:
`curl “http://127.0.0.1:{PORT}/metrics”`
For example, get all available Tungsten-specific metrics for the Manager:
shell> curl -s http://127.0.0.1:8092/metrics | wc -l
143
shell> curl -s http://127.0.0.1:8092/metrics | grep ^tungsten
tungsten_manager_version{version="Tungsten Clustering 7.0.0 build 1",vendor="Continuent",name="Tungsten Manager",} 1.0
tungsten_manager_service{service="east",datasource="db1-demo.continuent.com",role="slave",state="online",} 1.0
tungsten_manager_policy{policy="MAINTENANCE",} 0.0
tungsten_manager_connections{connections="active",} 0.0
tungsten_manager_connections{connections="total",} 0.0
Please note that there are many metrics returned, many that come as part of Java. Continuent provides the metrics that begin with `tungsten`.
Various Metrics Available
In addition to the Manager metrics shown above, both the Connector and the Replicator have Tungsten-specific metrics too!
Tungsten-specific metrics for the Replicator
shell> curl -s http://127.0.0.1:8091/metrics | wc -l
151
shell> curl -s http://127.0.0.1:8091/metrics | grep ^tungsten
tungsten_replicator_version{version="Tungsten Clustering 7.0.0 build 1",vendor="Continuent",name="Tungsten Replicator",} 1.0
tungsten_replicator_service{service="east",role="slave",state="offline:normal",} 0.0
tungsten_replicator_service{service="east_from_west",role="slave",state="offline:normal",} 0.0
tungsten_replicator_seqno{service="east",seqno="minimum",} -1.0
tungsten_replicator_seqno{service="east",seqno="maximum",} -1.0
tungsten_replicator_seqno{service="east",seqno="current",} -1.0
tungsten_replicator_seqno{service="east_from_west",seqno="minimum",} -1.0
tungsten_replicator_seqno{service="east_from_west",seqno="maximum",} -1.0
tungsten_replicator_seqno{service="east_from_west",seqno="current",} -1.0
tungsten_replicator_latency{service="east",latency="applied",} -1.0
tungsten_replicator_latency{service="east",latency="relative",} -1.0
tungsten_replicator_latency{service="east_from_west",latency="applied",} -1.0
tungsten_replicator_latency{service="east_from_west",latency="relative",} -1.0
Tungsten-specific metrics for the Connector
shell> curl -s http://127.0.0.1:8093/metrics | wc -l
149
shell> curl -s http://127.0.0.1:8093/metrics | grep ^tungsten
tungsten_connector_version{version="Tungsten Clustering 7.0.0 build 1",vendor="Continuent",name="Tungsten Connector",} 1.0
tungsten_connector_connections{dataservice="east",datasource="db1-demo.continuent.com",connections="active",} 0.0
tungsten_connector_connections{dataservice="east",datasource="db1-demo.continuent.com",connections="total",} 0.0
tungsten_connector_connections{dataservice="east",datasource="db2-demo.continuent.com",connections="active",} 0.0
tungsten_connector_connections{dataservice="east",datasource="db2-demo.continuent.com",connections="total",} 0.0
tungsten_connector_connections{dataservice="east",datasource="db3-demo.continuent.com",connections="active",} 0.0
tungsten_connector_connections{dataservice="east",datasource="db3-demo.continuent.com",connections="total",} 0.0
tungsten_connector_connections{dataservice="usa_active_active",datasource="east",connections="active",} 0.0
tungsten_connector_connections{dataservice="usa_active_active",datasource="east",connections="total",} 0.0
Using the Provided `tmonitor` Command-line Tool
The `tmonitor` command provides a simple CLI interface to the exporter subsystem.
Usage: tmonitor [args] {action}
where {action} is either status or test when using the three “internal” or built-in exporters for the Connector, Manager and Replicator.
To do a quick check of all the exporters, run `tmonitor status`, for example:
shell> tmonitor status
Tungsten Connector exporter running ok on port 8093
Tungsten Manager exporter running ok on port 8092
MySQL exporter running ok on port 9104
Node exporter running ok on port 9100
Tungsten Replicator exporter running ok on port 8091
All 5 exporters are running ok (Up: Tungsten Connector, Tungsten Manager, MySQL, Node, Tungsten Replicator)
To limit the quick check to just the Tungsten-specifc exporters, run `tmonitor -i status` to show just the “internal” exporters; for example:
shell> tmonitor -i status
Tungsten Connector exporter running ok on port 8093
Tungsten Manager exporter running ok on port 8092
Tungsten Replicator exporter running ok on port 8091
All 3 internal exporters are running ok (Up: Tungsten Connector, Tungsten Manager, Tungsten Replicator)
To do a full test of all the exporters, run `tmonitor -i test`, which uses curl to verify that the exporter metrics are available; for example:
shell> tmonitor -i test | wc -l
452
shell> tmonitor -i -t test | wc -l
62
shell> tmonitor -i -t test
====================================================================================================
== Metrics for the connector exporter:
====================================================================================================
# HELP tungsten_connector_version The Tungsten Clustering software version number
# TYPE tungsten_connector_version gauge
tungsten_connector_version{version="Tungsten Clustering 7.0.0 build 1",vendor="Continuent",name="Tungsten Connector",} 1.0
# HELP tungsten_connector_connections Connector active/total connections value
# TYPE tungsten_connector_connections gauge
tungsten_connector_connections{dataservice="east",datasource="db1-demo.continuent.com",connections="active",} 0.0
tungsten_connector_connections{dataservice="east",datasource="db1-demo.continuent.com",connections="total",} 0.0
...
To limit the test to just one component, specify one of -C, -M or -R (or more than one!), for example:
shell> tmonitor -M test | wc -l
146
shell> tmonitor -M -t test | wc -l
16
shell> tmonitor -M -t test
====================================================================================================
== Metrics for the manager exporter:
====================================================================================================
# HELP tungsten_manager_version The Tungsten Clustering software version number
# TYPE tungsten_manager_version gauge
tungsten_manager_version{version="Tungsten Clustering 7.0.0 build 1",vendor="Continuent",name="Tungsten Manager",} 1.0
# HELP tungsten_manager_service The service and datasource served by this Tungsten Manager
# TYPE tungsten_manager_service gauge
tungsten_manager_service{service="east",datasource="db1-demo.continuent.com",role="slave",state="online",} 1.0
# HELP tungsten_manager_policy The current Tungsten Manager policy mode
# TYPE tungsten_manager_policy gauge
tungsten_manager_policy{policy="MAINTENANCE",} 0.0
# HELP tungsten_manager_connections Connector active/total connections value
# TYPE tungsten_manager_connections gauge
tungsten_manager_connections{connections="active",} 0.0
tungsten_manager_connections{connections="total",} 0.0
...
Tungsten also supports two “external” exporter binaries which are shipped with the software package: mysqld_exporter, which handles the database metrics and node_exporter, which handles the host metrics.
The `tmonitor` command has additional features designed to ease administration for these two “external” exporters. Those actions include:
start- invoke the process to get it runningstop- terminate a running processrestart- stop and start a running processinstall- enable start at boot for init.d and systemctl-based OS’sremove- disable start at boot for init.d and systemctl-based OS’s
Customizing the Exporter Configuration
If the default ports (8091, 8092 & 8093) are in conflict, it is possible to change them.
Add one line per component to your INI file as needed, then run `tpm update`.
shell> vi /etc/tungsten/tungsten.ini
[defaults]
property=replicator.prometheus.exporter.port=28091
property=manager.prometheus.exporter.port=28092
property=connector.prometheus.exporter.port=28093
…
shell> tpm update -i
How to Disable the Exporters
The exporters are all enabled by default. It is simple to disable the Prometheus exporters - just add one line per component to your INI file, then run `tpm update`.
shell> vi /etc/tungsten/tungsten.ini
[defaults]
property=manager.prometheus.exporter.enabled=false
property=replicator.prometheus.exporter.enabled=false
property=connector.prometheus.exporter.enabled=false
…
shell> tpm update -i
Documentation & Resources
Since this blog post will be published prior to the release of version 7.0.0, I am unable to include links to documentation at this time. When v7.0.0 is published, we will re-publish this blog post with the appropriate documentation links.
Wrap-Up
Prometheus is one of the more popular ways to monitor your resources, and starting with version 7.0.0, Continuent has embedded the Prometheus exporter functionality into our core products (Manager, Replicator and Connector).
This blog post covered how to enable the exporters, getting metrics and what metrics are available, along with cli tools, customizing the config, documentation references and a little bit about Grafana Dashboard for Tungsten Clusters.
In our next post, we will cover displaying the Tungsten Prometheus metrics in Grafana using the free Grafana Dashboard for Tungsten Clusters.
Comments
Add new comment