How to Resolve Systemd Issue with Percona XtraDB Cluster on CentOS 7

By | In MySQL | January 18th, 2017

Under CentOS 7, with the transition from System V to Systemd, Percona XtraDB Cluster was integrated with Systemd. As a result, Percona XtraDB cluster’s operation mode was split into two different scripts.


Systemd brings many advantages over its previous init systems, i.e. sysvinit, and upstart, including but not limited to: socket and D-Bus activation during startup; on-demand startup of services; aggressive parallelization capabilities; and many others. However, it presents an issue with the synchronization of nodes in PXC as discussed below.

PXC Issue with Systemd

PXC SST fails to sync the nodes, thus being unable to join cluster group and giving broken pipe 32 SIG errors on both xtrabackup-v2 and rsync methods. As indicated in the log’s error message on both donor and joining node, the ports are open and communicating well, and SST is working correctly. 

On Donor Node



On Joining Node

Log Entries:


Explanation of the Problem

On Joining Node, during startup, SST is taking longer time to sync, therefore forcing startup script to fail. Unfortunately, the error message in the logs is misleading because it points to the broken pipe on SST streaming method. It can be noted that SST is mostly failing precisely at the same time in approximately 2 hours 28 minutes.


Check Systemd process and startup scripts for MySQL:

mysql systemd startup script (/usr/lib/systemd/system/mysql.service) has references for the timeout settings. Normally, you would not notice similar issues in Linux6.

Look for these variables and lines in /usr/bin/mysql-systemd






; do




During SST transfer, the startup script will sleep for 10 seconds, effectively increasing timeout until it reaches the desired timeout value of 2 hours and 28 minutes. On slower networks, if you have multiple 100’s of gigabytes of data, it might take more time for SST to complete and these settings might not be sufficient. Depending on your dbsize, N/W bandwidth and how long it takes for SST to finish, increase these settings. During testing on a slower network with ~400G data, it took ~9 hours for SST to finish successfully.

This issue is not present in the older versions of Linux. Therefore, it can be deduced that if you are running PXC cluster on CentOS 7 with Systemd, it’s recommended to properly map references for timeout settings from multiple configuration files to resolve the Systemd issue with PXC cluster. Let’s hope the issue will be addressed in future releases.

To learn more, please contact Datavail today. With more than 600 database administrators worldwide, Datavail is the largest database services provider in North America. As a reliable provider of 24×7 managed services for applications, BI/Analytics, and databases, Datavail can support your organization, regardless of the build you’ve selected.

Datavail Script: Terms & Conditions

By using this software script (“Script”), you are agreeing to the following terms and condition, as a legally enforceable contract, with Datavail Corporation (“Datavail”). If you do not agree with these terms, do not download or otherwise use the Script. You (which includes any entity whom you represent or for whom you use the Script) and Datavail agree as follows:

1. CONSIDERATION. As you are aware, you did not pay a fee to Datavail for the license to the Script. Consequently, your consideration for use of the Script is your agreement to these terms, including the various waivers, releases and limitations of your rights and Datavail’s liabilities, as setforth herein.

2. LICENSE. Subject to the terms herein, the Script is provided to you as a non-exclusive, revocable license to use internally and not to transfer, sub-license, copy, or create derivative works from the Script, not to use the Script in a service bureau and not to disclose the Script to any third parties. No title or other ownership of the Script (or intellectual property rights therein) is assigned to you.

3. USE AT YOUR OWN RISK; DISCLAIMER OF WARRANTIES. You agree that your use of the Script and any impacts on your software, databases, systems, networks or other property or services are solely and exclusively at your own risk. Datavail does not make any warranties, and hereby expressly disclaims any and all warranties, implied or express, including without limitation, the following: (1) performance of or results from the Script, (2) compatibility with any other software or hardware, (3) non-infringement or violation of third party’s intellectual property or other property rights, (4) fitness for a particular purpose, or (5) merchantability.


You hereby release Datavail from any claims, causes of action, losses, damages, costs and expenses resulting from your downloading or other use of the Script.

5. AGREEMENT. These terms and conditions constitute your complete and exclusive legal agreement between you and Datavail.

Contact Us
Srinivasa Krishna
Krishna has more than 11 years of experience in the IT industry, including 8+ years as a MySQL database administrator with Datavail Corporation. He specializes in MySQL database administration and technologies like MongoDB, Amazon Redshift and Linux Administration and scripting. He also specializes in database architecture, DBA, HA Solutions, performance tuning, and managing enterprise level database support.

Leave a Reply

Your email address will not be published.
Required fields are marked (*).

3 thoughts on “How to Resolve Systemd Issue with Percona XtraDB Cluster on CentOS 7”
  1. I can rejoin my cluster:
    Can please help me?


    WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | pv -f -i 10 -N joiner -F ‘%N => Rate:%r Avg:%a Elapsed:%t %e Bytes: %b %p’ | gzip -dc | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20180219 09:07:12.489)
    2018-02-19 9:07:15 139810312283904 [Note] WSREP: (37165fdc, ‘tcp://’) connection to peer 37165fdc with addr tcp://xxxx:4567 timed out, no messages seen in PT3S
    2018-02-19 9:07:15 139810312283904 [Note] WSREP: (37165fdc, ‘tcp://’) turning message relay requesting off
    joiner: => Rate:[ 0 B/s] Avg:[ 0 B/s] Elapsed:0:01:20
    WSREP_SST: [ERROR] Removing /tmp/tmp.TA2J14T4EN/xtrabackup_galera_info file due to signal (20180219 09:08:41.691)
    WSREP_SST: [INFO] NOTE: Joiner-Recv-gtid took 89 seconds (20180219 09:08:41.696)
    WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 143 32 143 143 (20180219 09:08:41.699)
    WSREP_SST: [ERROR] Cleanup after exit with status:32 (20180219 09:08:41.701)
    2018-02-19 9:08:41 139810303891200 [ERROR] WSREP: Failed to read ‘ready ‘ from: wsrep_sst_xtrabackup-v2 –role ‘joiner’ –address ‘xxx’ –datadir ‘/var/lib/mysql/’ –parent ‘133906’ –binlog ‘mysql-bin’
    Read: ‘(null)’
    2018-02-19 9:08:41 139810303891200 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 –role ‘joiner’ –address ‘xxx’ –datadir ‘/var/lib/mysql/’ –parent ‘133906’ –binlog ‘mysql-bin’ : 32 (Broken pipe)
    2018-02-19 9:08:41 139816009234176 [ERROR] WSREP: Failed to prepare for ‘xtrabackup-v2’ SST. Unrecoverable.
    2018-02-19 9:08:41 139816009234176 [ERROR] Aborting

    Error in my_thread_global_end(): 1 threads didn’t exit

    innodb log:

    ^Ginnobackupex: Error writing file ‘UNOPENED’ (Errcode: 32 – Broken pipe)
    [05] xtrabackup: Error: failed to copy datafile.
    ^Ginnobackupex: Error writing file ‘UNOPENED’ (Errcode: 32 – Broken pipe)
    xb_stream_write_data() failed.
    compress: write to the destination stream failed.

    Can help me what should I do?

    1. Hi Syamiera,
      It looks like an error with the donor while performing the SST method backup.
      look at those connection timed out messages in the logs.
      backup logs states a broken pipe error, this could be related to many issues w.r.t port issues on both sides, n/w packet latency/drop, how its configured, if any mismatch,etc.
      What versions of PXC and Xtrabackup are you using?
      if the versions and systems matched, did you try to increase the startup_sleep_time mentioned above?
      do you have sample my.cnf to check the config ?
      what options you have in wsrep_provider_options,etc.
      Is it on a Linux server with SELInux?
      I would suggest to check the system logs and donor mysql logs , to see if any more details ?

      Thank you