MySQL Group Replication: Timeout while waiting for the group communication engine to be ready

Question 1

I am running into an issue joining the second instance into a MySQL replication group. It appears to be either a communication or timeout error. In short, I’m seeing this error in the log of the second node:

"Timeout while waiting for the group communication engine to be ready!"
"The group communication engine is not ready for the member to join. Local port: 33061"

1

2

3

"Timeout while waiting for the group communication engine to be ready!"

"The group communication engine is not ready for the member to join. Local port: 33061"

The steps I have taken to set up the group:

1) First I ensured that the proper ports are open between the two servers:

For reference, I’m using these four variables for privacy:

AA.AAA.AA.A = My FIRST server’s PUBLIC IP address
AA.0.0.000 = My FIRST server’s PRIVATE IP address
BBB.BBB.BBB.BBB = My SECOND server’s PUBLIC IP address
BBB.0.0.000 = My SECOND server’s PRIVATE IP address

The FIRST server lives on Amazon AWS, and it has a security group which opens itself to BBB.BBB.BBB.BBB on 3306 and 33061.

The SECOND server lives on a private server and it has a firewall which allows traffic from AA.AAA.AA.A on 3306 and 33061.

From the FIRST server, make sure I can access the second server:

mysql -u repl -p'XXXXXXXXXXXXX' -h BBB.BBB.BBB.BBB
telnet BBB.BBB.BBB.BBB 3306
telnet BBB.BBB.BBB.BBB 33061
telnet BBB.BBB.BBB.BBB 7777 #Times out, as expected

1

2

3

4

5

mysql -u repl -p'XXXXXXXXXXXXX' -h BBB.BBB.BBB.BBB

telnet BBB.BBB.BBB.BBB 3306

telnet BBB.BBB.BBB.BBB 33061

telnet BBB.BBB.BBB.BBB 7777 #Times out, as expected

From the FIRST server, make sure I can make outgoing connections:

curl portquiz.net:3306
curl portquiz.net:33061

1

2

3

curl portquiz.net:3306

curl portquiz.net:33061

From the SECOND server, make sure I an access the second server:

mysql -u repl -p'XXXXXXXXXXXXX' -h AA.AAA.AA.A
telnet AA.AAA.AA.A 3306
telnet AA.AAA.AA.A 33061
telnet AA.AAA.AA.A 7777 #Times out, as expected

1

2

3

4

5

mysql -u repl -p'XXXXXXXXXXXXX' -h AA.AAA.AA.A

telnet AA.AAA.AA.A 3306

telnet AA.AAA.AA.A 33061

telnet AA.AAA.AA.A 7777 #Times out, as expected

From the SECOND server, make sure I can make outgoing connections:

curl portquiz.net:3306
curl portquiz.net:33061

1

2

3

curl portquiz.net:3306

curl portquiz.net:33061

This all seems to be in working order!

2) Next I start up MySQL on each server with the following configs:

My FIRST server’s Config file:

[mysqld_safe]
nice                                    = 0
socket                                  = /var/run/mysqld/mysqld.sock

[mysqld]
basedir                                 = /usr
bind_address                            = 0.0.0.0
binlog_checksum                         = NONE
binlog_format                           = ROW
datadir                                 = /var/lib/mysql
enforce_gtid_consistency                = ON
general_log                             = 1
general_log_file                        = /var/log/mysql/mysql.log
gtid_mode                               = ON
key_buffer_size                         = 8388608
lc_messages_dir                         = /usr/share/mysql
log_bin                                 = binlog
log_error                               = /var/log/mysql/error.log
log_slave_updates                       = ON
long_query_time                         = 60
loose-group_replication_allow_local_disjoint_gtids_join= ON
loose-group_replication_bootstrap_group = OFF
loose-group_replication_enforce_update_everywhere_checks= ON
loose-group_replication_group_name      = ZZZZZZ-ZZZZZZ-ZZZZZZ-ZZZZZZ
loose-group_replication_group_seeds     = AA.0.0.000:33061,BBB.BBB.BBB.BBB:33061
loose-group_replication_ip_whitelist    = AA.0.0.000,BBB.BBB.BBB.BBB,BBB.0.0.000
loose-group_replication_local_address   = AA.0.0.000:33061
loose-group_replication_recovery_use_ssl= 1
loose-group_replication_single_primary_mode= OFF
loose-group_replication_ssl_mode        = REQUIRED
loose-group_replication_start_on_boot   = OFF
master_info_repository                  = TABLE
max_allowed_packet                      = 16M
max_binlog_size                         = 100M
max_connect_errors                      = 100000000
pid-file                                = /var/run/mysqld/mysqld.pid
port                                    = 3306
relay_log                               = first-server-relay-bin
relay_log_info_repository               = TABLE
report_host                             = AA.AAA.AA.A
server_id                               = 1
skip_external_locking
slow_query_log                          = 1
slow_query_log_file                     = /var/log/mysql/slow_query.log
socket                                  = /var/run/mysqld/mysqld.sock
thread_cache_size                       = 8
thread_stack                            = 192K
tmpdir                                  = /tmp
transaction_write_set_extraction        = XXHASH64
user                                    = mysql

[mysqldump]
max_allowed_packet                      = 16M
quick
quote_names

[mysql]
no_auto_rehash

[isamchk]
key_buffer_size                         = 16M

[client]
port                                    = 3306
socket                                  = /var/run/mysqld/mysqld.sock

!includedir /etc/mysql/conf.d/

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

[mysqld_safe]

nice = 0

socket = /var/run/mysqld/mysqld.sock

[mysqld]

basedir = /usr

bind_address = 0.0.0.0

binlog_checksum = NONE

binlog_format = ROW

datadir = /var/lib/mysql

enforce_gtid_consistency = ON

general_log = 1

general_log_file = /var/log/mysql/mysql.log

gtid_mode = ON

key_buffer_size = 8388608

lc_messages_dir = /usr/share/mysql

log_bin = binlog

log_error = /var/log/mysql/error.log

log_slave_updates = ON

long_query_time = 60

loose-group_replication_allow_local_disjoint_gtids_join= ON

loose-group_replication_bootstrap_group = OFF

loose-group_replication_enforce_update_everywhere_checks= ON

loose-group_replication_group_name = ZZZZZZ-ZZZZZZ-ZZZZZZ-ZZZZZZ

loose-group_replication_group_seeds = AA.0.0.000:33061,BBB.BBB.BBB.BBB:33061

loose-group_replication_ip_whitelist = AA.0.0.000,BBB.BBB.BBB.BBB,BBB.0.0.000

loose-group_replication_local_address = AA.0.0.000:33061

loose-group_replication_recovery_use_ssl= 1

loose-group_replication_single_primary_mode= OFF

loose-group_replication_ssl_mode = REQUIRED

loose-group_replication_start_on_boot = OFF

master_info_repository = TABLE

max_allowed_packet = 16M

max_binlog_size = 100M

max_connect_errors = 100000000

pid-file = /var/run/mysqld/mysqld.pid

port = 3306

relay_log = first-server-relay-bin

relay_log_info_repository = TABLE

report_host = AA.AAA.AA.A

server_id = 1

skip_external_locking

slow_query_log = 1

slow_query_log_file = /var/log/mysql/slow_query.log

socket = /var/run/mysqld/mysqld.sock

thread_cache_size = 8

thread_stack = 192K

tmpdir = /tmp

transaction_write_set_extraction = XXHASH64

user = mysql

[mysqldump]

max_allowed_packet = 16M

quick

quote_names

[mysql]

no_auto_rehash

[isamchk]

key_buffer_size = 16M

[client]

port = 3306

socket = /var/run/mysqld/mysqld.sock

!includedir /etc/mysql/conf.d/

This is my SECOND server’s config:

[mysqld_safe]
nice                                    = 0
socket                                  = /var/run/mysqld/mysqld.sock

[mysqld]
basedir                                 = /usr
bind_address                            = 0.0.0.0
binlog_checksum                         = NONE
binlog_format                           = ROW
datadir                                 = /var/lib/mysql
enforce_gtid_consistency                = ON
general_log                             = 1
general_log_file                        = /var/log/mysql/mysql.log
gtid_mode                               = ON
key_buffer_size                         = 8388608
lc_messages_dir                         = /usr/share/mysql
log_bin                                 = binlog
log_error                               = /var/log/mysql/error.log
log_slave_updates                       = ON
long_query_time                         = 60
loose-group_replication_allow_local_disjoint_gtids_join= ON
loose-group_replication_bootstrap_group = OFF
loose-group_replication_enforce_update_everywhere_checks= ON
loose-group_replication_group_name      = ZZZZZZ-ZZZZZZ-ZZZZZZ-ZZZZZZ
loose-group_replication_group_seeds     = BBB.0.0.000:33061,AA.AAA.AA.A:33061
loose-group_replication_ip_whitelist    = BBB.0.0.000,AA.AAA.AA.A,AA.0.0.000
loose-group_replication_local_address   = BBB.0.0.000:33061
loose-group_replication_recovery_use_ssl= 1
loose-group_replication_single_primary_mode= OFF
loose-group_replication_ssl_mode        = REQUIRED
loose-group_replication_start_on_boot   = OFF
master_info_repository                  = TABLE
max_allowed_packet                      = 16M
max_binlog_size                         = 100M
max_connect_errors                      = 100000000
pid-file                                = /var/run/mysqld/mysqld.pid
port                                    = 3306
relay_log                               = second-server-relay-bin
relay_log_info_repository               = TABLE
report_host                             = BB.BBB.BB.B
server_id                               = 2
skip_external_locking
slow_query_log                          = 1
slow_query_log_file                     = /var/log/mysql/slow_query.log
socket                                  = /var/run/mysqld/mysqld.sock
thread_cache_size                       = 8
thread_stack                            = 192K
tmpdir                                  = /tmp
transaction_write_set_extraction        = XXHASH64
user                                    = mysql

[mysqldump]
max_allowed_packet                      = 16M
quick
quote_names

[mysql]
no_auto_rehash

[isamchk]
key_buffer_size                         = 16M

[client]
port                                    = 3306
socket                                  = /var/run/mysqld/mysqld.sock

!includedir /etc/mysql/conf.d/

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

[mysqld_safe]

nice = 0

socket = /var/run/mysqld/mysqld.sock

[mysqld]

basedir = /usr

bind_address = 0.0.0.0

binlog_checksum = NONE

binlog_format = ROW

datadir = /var/lib/mysql

enforce_gtid_consistency = ON

general_log = 1

general_log_file = /var/log/mysql/mysql.log

gtid_mode = ON

key_buffer_size = 8388608

lc_messages_dir = /usr/share/mysql

log_bin = binlog

log_error = /var/log/mysql/error.log

log_slave_updates = ON

long_query_time = 60

loose-group_replication_allow_local_disjoint_gtids_join= ON

loose-group_replication_bootstrap_group = OFF

loose-group_replication_enforce_update_everywhere_checks= ON

loose-group_replication_group_name = ZZZZZZ-ZZZZZZ-ZZZZZZ-ZZZZZZ

loose-group_replication_group_seeds = BBB.0.0.000:33061,AA.AAA.AA.A:33061

loose-group_replication_ip_whitelist = BBB.0.0.000,AA.AAA.AA.A,AA.0.0.000

loose-group_replication_local_address = BBB.0.0.000:33061

loose-group_replication_recovery_use_ssl= 1

loose-group_replication_single_primary_mode= OFF

loose-group_replication_ssl_mode = REQUIRED

loose-group_replication_start_on_boot = OFF

master_info_repository = TABLE

max_allowed_packet = 16M

max_binlog_size = 100M

max_connect_errors = 100000000

pid-file = /var/run/mysqld/mysqld.pid

port = 3306

relay_log = second-server-relay-bin

relay_log_info_repository = TABLE

report_host = BB.BBB.BB.B

server_id = 2

skip_external_locking

slow_query_log = 1

slow_query_log_file = /var/log/mysql/slow_query.log

socket = /var/run/mysqld/mysqld.sock

thread_cache_size = 8

thread_stack = 192K

tmpdir = /tmp

transaction_write_set_extraction = XXHASH64

user = mysql

[mysqldump]

max_allowed_packet = 16M

quick

quote_names

[mysql]

no_auto_rehash

[isamchk]

key_buffer_size = 16M

[client]

port = 3306

socket = /var/run/mysqld/mysqld.sock

!includedir /etc/mysql/conf.d/

Both servers start up and there are no errors in the error log.

3) Now I attempt to bootstrap the groups:

On the FIRST server:

  mysql> SET GLOBAL GROUP_REPLICATION_BOOTSTRAP_GROUP=ON;
Query OK, 0 rows affected (0.00 sec)

mysql> START GROUP_REPLICATION;
Query OK, 0 rows affected (2.11 sec)

mysql> SET GLOBAL GROUP_REPLICATION_BOOTSTRAP_GROUP=OFF;
Query OK, 0 rows affected (0.00 sec)

mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | 111-111-111-111-111111-111111-111111 | AA.AAA.AA.A |        3306 | ONLINE       |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
1 row in set (0.00 sec)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

mysql> SET GLOBAL GROUP_REPLICATION_BOOTSTRAP_GROUP=ON;

Query OK, 0 rows affected (0.00 sec)

mysql> START GROUP_REPLICATION;

Query OK, 0 rows affected (2.11 sec)

mysql> SET GLOBAL GROUP_REPLICATION_BOOTSTRAP_GROUP=OFF;

Query OK, 0 rows affected (0.00 sec)

mysql> SELECT * FROM performance_schema.replication_group_members;

+---------------------------+--------------------------------------+-------------+-------------+--------------+

+---------------------------+--------------------------------------+-------------+-------------+--------------+

+---------------------------+--------------------------------------+-------------+-------------+--------------+

1 row in set (0.00 sec)

This looks good and there are no errors in the log file.

On the SECOND server:

mysql> START GROUP_REPLICATION;
ERROR 3092 (HY000): The server is not configured properly to be an active member of the group. Please see more details on error log.
mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-----------------+-------------+--------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST     | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-----------------+-------------+--------------+
| group_replication_applier | 222-222-222-222-222222-222222-222222 | BBB.BBB.BBB.BBB |        3306 | OFFLINE      |
+---------------------------+--------------------------------------+-----------------+-------------+--------------+
1 row in set (0.00 sec)

1

2

3

4

5

6

7

8

9

10

mysql> START GROUP_REPLICATION;

ERROR 3092 (HY000): The server is not configured properly to be an active member of the group. Please see more details on error log.

mysql> SELECT * FROM performance_schema.replication_group_members;

+---------------------------+--------------------------------------+-----------------+-------------+--------------+

+---------------------------+--------------------------------------+-----------------+-------------+--------------+

+---------------------------+--------------------------------------+-----------------+-------------+--------------+

1 row in set (0.00 sec)

In the log file, I can see the group replication starting up successfully, the second server connecting to itself (BBB.0.0.000:33061) and the FIRST server (AA.AAA.AA.A:33061) — “Success connecting using SSL” — but immediately after I get the following two errors: [GCS] Timeout while waiting for the group communication engine to be ready! and [GCS] The group communication engine is not ready for the member to join. Local port: 33061

2018-04-15T00:15:05.987696Z 5 [Note] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2018-04-15T00:15:05.990639Z 3 [Note] Plugin group_replication reported: 'Group Replication applier module successfully initialized!'
2018-04-15T00:15:05.990714Z 3 [Note] Plugin group_replication reported: 'auto_increment_increment is set to 7'
2018-04-15T00:15:05.990754Z 3 [Note] Plugin group_replication reported: 'auto_increment_offset is set to 1'
2018-04-15T00:15:05.990866Z 8 [Note] Slave SQL thread for channel 'group_replication_applier' initialized, starting replication in log 'FIRST' at position 0, relay log './second-server-relay-bin-group_replication_applier.000012' position: 4
2018-04-15T00:15:05.991232Z 0 [Note] Plugin group_replication reported: 'state 0 action xa_init'
2018-04-15T00:15:06.013709Z 0 [Note] Plugin group_replication reported: 'Successfully bound to 0.0.0.0:33061 (socket=51).'
2018-04-15T00:15:06.013787Z 0 [Note] Plugin group_replication reported: 'Successfully set listen backlog to 32 (socket=51)!'
2018-04-15T00:15:06.013827Z 0 [Note] Plugin group_replication reported: 'Successfully unblocked socket (socket=51)!'
2018-04-15T00:15:06.013876Z 0 [Note] Plugin group_replication reported: 'connecting to BBB.0.0.000 33061'
2018-04-15T00:15:06.013989Z 0 [Note] Plugin group_replication reported: 'client connected to BBB.0.0.000 33061 fd 52'
2018-04-15T00:15:06.014094Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'
2018-04-15T00:15:06.014188Z 0 [Note] Plugin group_replication reported: 'Ready to accept incoming connections on 0.0.0.0:33061 (socket=51)!'
2018-04-15T00:15:06.027381Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'
2018-04-15T00:15:06.027583Z 0 [Note] Plugin group_replication reported: 'connecting to BBB.0.0.000 33061'
2018-04-15T00:15:06.027665Z 0 [Note] Plugin group_replication reported: 'client connected to BBB.0.0.000 33061 fd 62'
2018-04-15T00:15:06.027730Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'
2018-04-15T00:15:06.040561Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'
2018-04-15T00:15:06.040752Z 0 [Note] Plugin group_replication reported: 'connecting to BBB.0.0.000 33061'
2018-04-15T00:15:06.040831Z 0 [Note] Plugin group_replication reported: 'client connected to BBB.0.0.000 33061 fd 66'
2018-04-15T00:15:06.040894Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'
2018-04-15T00:15:06.053675Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'
2018-04-15T00:15:06.053889Z 0 [Note] Plugin group_replication reported: 'connecting to BBB.0.0.000 33061'
2018-04-15T00:15:06.053975Z 0 [Note] Plugin group_replication reported: 'client connected to BBB.0.0.000 33061 fd 70'
2018-04-15T00:15:06.054045Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'
2018-04-15T00:15:06.067043Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'
2018-04-15T00:15:06.067228Z 0 [Note] Plugin group_replication reported: 'connecting to BBB.0.0.000 33061'
2018-04-15T00:15:06.067305Z 0 [Note] Plugin group_replication reported: 'client connected to BBB.0.0.000 33061 fd 74'
2018-04-15T00:15:06.067368Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'
2018-04-15T00:15:06.080143Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'
2018-04-15T00:15:06.080326Z 0 [Note] Plugin group_replication reported: 'connecting to BBB.0.0.000 33061'
2018-04-15T00:15:06.080403Z 0 [Note] Plugin group_replication reported: 'client connected to BBB.0.0.000 33061 fd 78'
2018-04-15T00:15:06.080465Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'
2018-04-15T00:15:06.093262Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'
2018-04-15T00:15:06.093460Z 0 [Note] Plugin group_replication reported: 'connecting to AA.AAA.AA.A 33061'
2018-04-15T00:15:06.125679Z 0 [Note] Plugin group_replication reported: 'client connected to AA.AAA.AA.A 33061 fd 82'
2018-04-15T00:15:06.125774Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'
2018-04-15T00:15:06.204717Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'
2018-04-15T00:15:36.270721Z 0 [ERROR] Plugin group_replication reported: '[GCS] Timeout while waiting for the group communication engine to be ready!'
2018-04-15T00:15:36.270933Z 0 [ERROR] Plugin group_replication reported: '[GCS] The group communication engine is not ready for the member to join. Local port: 33061'
2018-04-15T00:15:36.271093Z 0 [Note] Plugin group_replication reported: 'state 4337 action xa_terminate'
2018-04-15T00:15:36.271158Z 0 [Note] Plugin group_replication reported: 'new state x_start'
2018-04-15T00:15:36.271210Z 0 [Note] Plugin group_replication reported: 'state 4337 action xa_exit'
2018-04-15T00:15:36.271331Z 0 [Note] Plugin group_replication reported: 'Exiting xcom thread'
2018-04-15T00:15:36.271383Z 0 [Note] Plugin group_replication reported: 'new state x_start'
2018-04-15T00:15:36.273406Z 0 [Warning] Plugin group_replication reported: 'read failed'
2018-04-15T00:15:36.281153Z 0 [ERROR] Plugin group_replication reported: '[GCS] The member was unable to join the group. Local port: 33061'
2018-04-15T00:16:05.990871Z 3 [ERROR] Plugin group_replication reported: 'Timeout on wait for view after joining group'
2018-04-15T00:16:05.991036Z 3 [Note] Plugin group_replication reported: 'Requesting to leave the group despite of not being a member'
2018-04-15T00:16:05.991118Z 3 [ERROR] Plugin group_replication reported: '[GCS] The member is leaving a group without being on one.'
2018-04-15T00:16:05.991182Z 0 [Note] Plugin group_replication reported: 'Destroying SSL'
2018-04-15T00:16:05.991242Z 0 [Note] Plugin group_replication reported: 'Success destroying SSL'
2018-04-15T00:16:05.991352Z 3 [Note] Plugin group_replication reported: 'auto_increment_increment is reset to 1'
2018-04-15T00:16:05.991408Z 3 [Note] Plugin group_replication reported: 'auto_increment_offset is reset to 1'
2018-04-15T00:16:05.991618Z 8 [Note] Error reading relay log event for channel 'group_replication_applier': slave SQL thread was killed
2018-04-15T00:16:05.992230Z 5 [Note] Plugin group_replication reported: 'The group replication applier thread was killed'

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

2018-04-15T00:15:05.987696Z 5 [Note] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.

2018-04-15T00:15:05.990639Z 3 [Note] Plugin group_replication reported: 'Group Replication applier module successfully initialized!'

2018-04-15T00:15:05.990714Z 3 [Note] Plugin group_replication reported: 'auto_increment_increment is set to 7'

2018-04-15T00:15:05.990754Z 3 [Note] Plugin group_replication reported: 'auto_increment_offset is set to 1'

2018-04-15T00:15:05.990866Z 8 [Note] Slave SQL thread for channel 'group_replication_applier' initialized, starting replication in log 'FIRST' at position 0, relay log './second-server-relay-bin-group_replication_applier.000012' position: 4

2018-04-15T00:15:05.991232Z 0 [Note] Plugin group_replication reported: 'state 0 action xa_init'

2018-04-15T00:15:06.013709Z 0 [Note] Plugin group_replication reported: 'Successfully bound to 0.0.0.0:33061 (socket=51).'

2018-04-15T00:15:06.013787Z 0 [Note] Plugin group_replication reported: 'Successfully set listen backlog to 32 (socket=51)!'

2018-04-15T00:15:06.013827Z 0 [Note] Plugin group_replication reported: 'Successfully unblocked socket (socket=51)!'

2018-04-15T00:15:06.013876Z 0 [Note] Plugin group_replication reported: 'connecting to BBB.0.0.000 33061'

2018-04-15T00:15:06.013989Z 0 [Note] Plugin group_replication reported: 'client connected to BBB.0.0.000 33061 fd 52'

2018-04-15T00:15:06.014094Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'

2018-04-15T00:15:06.014188Z 0 [Note] Plugin group_replication reported: 'Ready to accept incoming connections on 0.0.0.0:33061 (socket=51)!'

2018-04-15T00:15:06.027381Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'

2018-04-15T00:15:06.027583Z 0 [Note] Plugin group_replication reported: 'connecting to BBB.0.0.000 33061'

2018-04-15T00:15:06.027665Z 0 [Note] Plugin group_replication reported: 'client connected to BBB.0.0.000 33061 fd 62'

2018-04-15T00:15:06.027730Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'

2018-04-15T00:15:06.040561Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'

2018-04-15T00:15:06.040752Z 0 [Note] Plugin group_replication reported: 'connecting to BBB.0.0.000 33061'

2018-04-15T00:15:06.040831Z 0 [Note] Plugin group_replication reported: 'client connected to BBB.0.0.000 33061 fd 66'

2018-04-15T00:15:06.040894Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'

2018-04-15T00:15:06.053675Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'

2018-04-15T00:15:06.053889Z 0 [Note] Plugin group_replication reported: 'connecting to BBB.0.0.000 33061'

2018-04-15T00:15:06.053975Z 0 [Note] Plugin group_replication reported: 'client connected to BBB.0.0.000 33061 fd 70'

2018-04-15T00:15:06.054045Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'

2018-04-15T00:15:06.067043Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'

2018-04-15T00:15:06.067228Z 0 [Note] Plugin group_replication reported: 'connecting to BBB.0.0.000 33061'

2018-04-15T00:15:06.067305Z 0 [Note] Plugin group_replication reported: 'client connected to BBB.0.0.000 33061 fd 74'

2018-04-15T00:15:06.067368Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'

2018-04-15T00:15:06.080143Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'

2018-04-15T00:15:06.080326Z 0 [Note] Plugin group_replication reported: 'connecting to BBB.0.0.000 33061'

2018-04-15T00:15:06.080403Z 0 [Note] Plugin group_replication reported: 'client connected to BBB.0.0.000 33061 fd 78'

2018-04-15T00:15:06.080465Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'

2018-04-15T00:15:06.093262Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'

2018-04-15T00:15:06.093460Z 0 [Note] Plugin group_replication reported: 'connecting to AA.AAA.AA.A 33061'

2018-04-15T00:15:06.125679Z 0 [Note] Plugin group_replication reported: 'client connected to AA.AAA.AA.A 33061 fd 82'

2018-04-15T00:15:06.125774Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'

2018-04-15T00:15:06.204717Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'

2018-04-15T00:15:36.270721Z 0 [ERROR] Plugin group_replication reported: '[GCS] Timeout while waiting for the group communication engine to be ready!'

2018-04-15T00:15:36.270933Z 0 [ERROR] Plugin group_replication reported: '[GCS] The group communication engine is not ready for the member to join. Local port: 33061'

2018-04-15T00:15:36.271093Z 0 [Note] Plugin group_replication reported: 'state 4337 action xa_terminate'

2018-04-15T00:15:36.271158Z 0 [Note] Plugin group_replication reported: 'new state x_start'

2018-04-15T00:15:36.271210Z 0 [Note] Plugin group_replication reported: 'state 4337 action xa_exit'

2018-04-15T00:15:36.271331Z 0 [Note] Plugin group_replication reported: 'Exiting xcom thread'

2018-04-15T00:15:36.271383Z 0 [Note] Plugin group_replication reported: 'new state x_start'

2018-04-15T00:15:36.273406Z 0 [Warning] Plugin group_replication reported: 'read failed'

2018-04-15T00:15:36.281153Z 0 [ERROR] Plugin group_replication reported: '[GCS] The member was unable to join the group. Local port: 33061'

2018-04-15T00:16:05.990871Z 3 [ERROR] Plugin group_replication reported: 'Timeout on wait for view after joining group'

2018-04-15T00:16:05.991036Z 3 [Note] Plugin group_replication reported: 'Requesting to leave the group despite of not being a member'

2018-04-15T00:16:05.991118Z 3 [ERROR] Plugin group_replication reported: '[GCS] The member is leaving a group without being on one.'

2018-04-15T00:16:05.991182Z 0 [Note] Plugin group_replication reported: 'Destroying SSL'

2018-04-15T00:16:05.991242Z 0 [Note] Plugin group_replication reported: 'Success destroying SSL'

2018-04-15T00:16:05.991352Z 3 [Note] Plugin group_replication reported: 'auto_increment_increment is reset to 1'

2018-04-15T00:16:05.991408Z 3 [Note] Plugin group_replication reported: 'auto_increment_offset is reset to 1'

2018-04-15T00:16:05.991618Z 8 [Note] Error reading relay log event for channel 'group_replication_applier': slave SQL thread was killed

2018-04-15T00:16:05.992230Z 5 [Note] Plugin group_replication reported: 'The group replication applier thread was killed'

When I go back to the FIRST server, there is no movement in the logs, they are exactly the same as they were previously.

I can’t find any useful information about these two errors: “Timeout while waiting for the group communication engine to be ready!” and “The group communication engine is not ready for the member to join. Local port: 33061”

Has anyone seen this before and know what it indicates?

Question 2

It turns out that this was a networking issue. Instead of using a mix of internal and public IPs, I needed to use only the public IPs. Additionally, I needed to run an ifconfig command to bind the servers’ public IPs.

On each server, I ran “ifconfig” to identify the name of the network endpoint. Then I ran “ifcongig ntwk 00.00.00.00”:

 @FirstServer> ifconfig
 en1       Link encap:Ethernet  HWaddr 00:00:00:00:00:00  
           inet addr:00.00.00.00  Bcast:00.00.00.255  Mask:255.255.255.0
           inet6 addr: 00::00:00:00:00/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:718094 errors:0 dropped:104 overruns:0 frame:0
           TX packets:708888 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000 
           RX bytes:666094518 (666.0 MB)  TX bytes:493309981 (493.3 MB)

 lo        Link encap:Local Loopback  
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet6 addr: ::1/128 Scope:Host
           UP LOOPBACK RUNNING  MTU:65536  Metric:1
           RX packets:46526 errors:0 dropped:0 overruns:0 frame:0
           TX packets:46526 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1 
           RX bytes:13353180 (13.3 MB)  TX bytes:13353180 (13.3 MB)


@FirstServer> sudo ifconfig en1:1 AA.AAA.AA.A

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

@FirstServer> ifconfig

en1 Link encap:Ethernet HWaddr 00:00:00:00:00:00

inet addr:00.00.00.00 Bcast:00.00.00.255 Mask:255.255.255.0

inet6 addr: 00::00:00:00:00/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:718094 errors:0 dropped:104 overruns:0 frame:0

TX packets:708888 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:666094518 (666.0 MB) TX bytes:493309981 (493.3 MB)

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:46526 errors:0 dropped:0 overruns:0 frame:0

TX packets:46526 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1

RX bytes:13353180 (13.3 MB) TX bytes:13353180 (13.3 MB)

@FirstServer> sudo ifconfig en1:1 AA.AAA.AA.A

Second Server:

 @SecondServer> ifconfig
 MyNets    Link encap:Ethernet  HWaddr 00:00:00:00:00:00  
           inet addr:00.00.00.00  Bcast:00.00.00.255  Mask:255.255.255.0
           inet6 addr: 00::00:00:00:00/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:718094 errors:0 dropped:104 overruns:0 frame:0
           TX packets:708888 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000 
           RX bytes:666094518 (666.0 MB)  TX bytes:493309981 (493.3 MB)

 lo        Link encap:Local Loopback  
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet6 addr: ::1/128 Scope:Host
           UP LOOPBACK RUNNING  MTU:65536  Metric:1
           RX packets:46526 errors:0 dropped:0 overruns:0 frame:0
           TX packets:46526 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1 
           RX bytes:13353180 (13.3 MB)  TX bytes:13353180 (13.3 MB)


@SecondServer> sudo ifconfig MyNets:1 BBB.BBB.BBB.BBB

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

@SecondServer> ifconfig

MyNets Link encap:Ethernet HWaddr 00:00:00:00:00:00

inet addr:00.00.00.00 Bcast:00.00.00.255 Mask:255.255.255.0

inet6 addr: 00::00:00:00:00/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:718094 errors:0 dropped:104 overruns:0 frame:0

TX packets:708888 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:666094518 (666.0 MB) TX bytes:493309981 (493.3 MB)

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:46526 errors:0 dropped:0 overruns:0 frame:0

TX packets:46526 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1

RX bytes:13353180 (13.3 MB) TX bytes:13353180 (13.3 MB)

@SecondServer> sudo ifconfig MyNets:1 BBB.BBB.BBB.BBB

After binding to the public IP addresses, I updated the config files. The FIRST server config file is:

[mysqld_safe]
nice                                    = 0
socket                                  = /var/run/mysqld/mysqld.sock

[mysqld]
basedir                                 = /usr
bind_address                            = 0.0.0.0
binlog_checksum                         = NONE
binlog_format                           = ROW
datadir                                 = /var/lib/mysql
enforce_gtid_consistency                = ON
general_log                             = 1
general_log_file                        = /var/log/mysql/mysql.log
gtid_mode                               = ON
key_buffer_size                         = 8388608
lc_messages_dir                         = /usr/share/mysql
log_bin                                 = binlog
log_error                               = /var/log/mysql/error.log
log_slave_updates                       = ON
long_query_time                         = 60
loose-group_replication_allow_local_disjoint_gtids_join= ON
loose-group_replication_bootstrap_group = OFF
loose-group_replication_enforce_update_everywhere_checks= ON
loose-group_replication_group_name      = ZZZZZZ-ZZZZZZ-ZZZZZZ-ZZZZZZ
loose-group_replication_group_seeds     = AA.AAA.AA.A:33061,BBB.BBB.BBB.BBB:33061
loose-group_replication_ip_whitelist    = AA.AAA.AA.A,BBB.BBB.BBB.BBB
loose-group_replication_local_address   = AA.AAA.AA.A:33061
loose-group_replication_recovery_use_ssl= 1
loose-group_replication_single_primary_mode= OFF
loose-group_replication_ssl_mode        = REQUIRED
loose-group_replication_start_on_boot   = OFF
master_info_repository                  = TABLE
max_allowed_packet                      = 16M
max_binlog_size                         = 100M
max_connect_errors                      = 100000000
pid-file                                = /var/run/mysqld/mysqld.pid
port                                    = 3306
relay_log                               = first-server-relay-bin
relay_log_info_repository               = TABLE
report_host                             = AA.AAA.AA.A
server_id                               = 1
skip_external_locking
slow_query_log                          = 1
slow_query_log_file                     = /var/log/mysql/slow_query.log
socket                                  = /var/run/mysqld/mysqld.sock
thread_cache_size                       = 8
thread_stack                            = 192K
tmpdir                                  = /tmp
transaction_write_set_extraction        = XXHASH64
user                                    = mysql

[mysqldump]
max_allowed_packet                      = 16M
quick
quote_names

[mysql]
no_auto_rehash

[isamchk]
key_buffer_size                         = 16M

[client]
port                                    = 3306
socket                                  = /var/run/mysqld/mysqld.sock

!includedir /etc/mysql/conf.d/

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

[mysqld_safe]

nice = 0

socket = /var/run/mysqld/mysqld.sock

[mysqld]

basedir = /usr

bind_address = 0.0.0.0

binlog_checksum = NONE

binlog_format = ROW

datadir = /var/lib/mysql

enforce_gtid_consistency = ON

general_log = 1

general_log_file = /var/log/mysql/mysql.log

gtid_mode = ON

key_buffer_size = 8388608

lc_messages_dir = /usr/share/mysql

log_bin = binlog

log_error = /var/log/mysql/error.log

log_slave_updates = ON

long_query_time = 60

loose-group_replication_allow_local_disjoint_gtids_join= ON

loose-group_replication_bootstrap_group = OFF

loose-group_replication_enforce_update_everywhere_checks= ON

loose-group_replication_group_name = ZZZZZZ-ZZZZZZ-ZZZZZZ-ZZZZZZ

loose-group_replication_group_seeds = AA.AAA.AA.A:33061,BBB.BBB.BBB.BBB:33061

loose-group_replication_ip_whitelist = AA.AAA.AA.A,BBB.BBB.BBB.BBB

loose-group_replication_local_address = AA.AAA.AA.A:33061

loose-group_replication_recovery_use_ssl= 1

loose-group_replication_single_primary_mode= OFF

loose-group_replication_ssl_mode = REQUIRED

loose-group_replication_start_on_boot = OFF

master_info_repository = TABLE

max_allowed_packet = 16M

max_binlog_size = 100M

max_connect_errors = 100000000

pid-file = /var/run/mysqld/mysqld.pid

port = 3306

relay_log = first-server-relay-bin

relay_log_info_repository = TABLE

report_host = AA.AAA.AA.A

server_id = 1

skip_external_locking

slow_query_log = 1

slow_query_log_file = /var/log/mysql/slow_query.log

socket = /var/run/mysqld/mysqld.sock

thread_cache_size = 8

thread_stack = 192K

tmpdir = /tmp

transaction_write_set_extraction = XXHASH64

user = mysql

[mysqldump]

max_allowed_packet = 16M

quick

quote_names

[mysql]

no_auto_rehash

[isamchk]

key_buffer_size = 16M

[client]

port = 3306

socket = /var/run/mysqld/mysqld.sock

!includedir /etc/mysql/conf.d/

And the SECOND server config file is:

[mysqld_safe]
nice                                    = 0
socket                                  = /var/run/mysqld/mysqld.sock

[mysqld]
basedir                                 = /usr
bind_address                            = 0.0.0.0
binlog_checksum                         = NONE
binlog_format                           = ROW
datadir                                 = /var/lib/mysql
enforce_gtid_consistency                = ON
general_log                             = 1
general_log_file                        = /var/log/mysql/mysql.log
gtid_mode                               = ON
key_buffer_size                         = 8388608
lc_messages_dir                         = /usr/share/mysql
log_bin                                 = binlog
log_error                               = /var/log/mysql/error.log
log_slave_updates                       = ON
long_query_time                         = 60
loose-group_replication_allow_local_disjoint_gtids_join= ON
loose-group_replication_bootstrap_group = OFF
loose-group_replication_enforce_update_everywhere_checks= ON
loose-group_replication_group_name      = ZZZZZZ-ZZZZZZ-ZZZZZZ-ZZZZZZ
loose-group_replication_group_seeds     = BB.BBB.BB.B:33061,AA.AAA.AA.A:33061
loose-group_replication_ip_whitelist    = BB.BBB.BB.B,AA.AAA.AA.A
loose-group_replication_local_address   = BB.BBB.BB.B:33061
loose-group_replication_recovery_use_ssl= 1
loose-group_replication_single_primary_mode= OFF
loose-group_replication_ssl_mode        = REQUIRED
loose-group_replication_start_on_boot   = OFF
master_info_repository                  = TABLE
max_allowed_packet                      = 16M
max_binlog_size                         = 100M
max_connect_errors                      = 100000000
pid-file                                = /var/run/mysqld/mysqld.pid
port                                    = 3306
relay_log                               = second-server-relay-bin
relay_log_info_repository               = TABLE
report_host                             = BB.BBB.BB.B
server_id                               = 2
skip_external_locking
slow_query_log                          = 1
slow_query_log_file                     = /var/log/mysql/slow_query.log
socket                                  = /var/run/mysqld/mysqld.sock
thread_cache_size                       = 8
thread_stack                            = 192K
tmpdir                                  = /tmp
transaction_write_set_extraction        = XXHASH64
user                                    = mysql

[mysqldump]
max_allowed_packet                      = 16M
quick
quote_names

[mysql]
no_auto_rehash

[isamchk]
key_buffer_size                         = 16M

[client]
port                                    = 3306
socket                                  = /var/run/mysqld/mysqld.sock

!includedir /etc/mysql/conf.d/

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

[mysqld_safe]

nice = 0

socket = /var/run/mysqld/mysqld.sock

[mysqld]

basedir = /usr

bind_address = 0.0.0.0

binlog_checksum = NONE

binlog_format = ROW

datadir = /var/lib/mysql

enforce_gtid_consistency = ON

general_log = 1

general_log_file = /var/log/mysql/mysql.log

gtid_mode = ON

key_buffer_size = 8388608

lc_messages_dir = /usr/share/mysql

log_bin = binlog

log_error = /var/log/mysql/error.log

log_slave_updates = ON

long_query_time = 60

loose-group_replication_allow_local_disjoint_gtids_join= ON

loose-group_replication_bootstrap_group = OFF

loose-group_replication_enforce_update_everywhere_checks= ON

loose-group_replication_group_name = ZZZZZZ-ZZZZZZ-ZZZZZZ-ZZZZZZ

loose-group_replication_group_seeds = BB.BBB.BB.B:33061,AA.AAA.AA.A:33061

loose-group_replication_ip_whitelist = BB.BBB.BB.B,AA.AAA.AA.A

loose-group_replication_local_address = BB.BBB.BB.B:33061

loose-group_replication_recovery_use_ssl= 1

loose-group_replication_single_primary_mode= OFF

loose-group_replication_ssl_mode = REQUIRED

loose-group_replication_start_on_boot = OFF

master_info_repository = TABLE

max_allowed_packet = 16M

max_binlog_size = 100M

max_connect_errors = 100000000

pid-file = /var/run/mysqld/mysqld.pid

port = 3306

relay_log = second-server-relay-bin

relay_log_info_repository = TABLE

report_host = BB.BBB.BB.B

server_id = 2

skip_external_locking

slow_query_log = 1

slow_query_log_file = /var/log/mysql/slow_query.log

socket = /var/run/mysqld/mysqld.sock

thread_cache_size = 8

thread_stack = 192K

tmpdir = /tmp

transaction_write_set_extraction = XXHASH64

user = mysql

[mysqldump]

max_allowed_packet = 16M

quick

quote_names

[mysql]

no_auto_rehash

[isamchk]

key_buffer_size = 16M

[client]

port = 3306

socket = /var/run/mysqld/mysqld.sock

!includedir /etc/mysql/conf.d/

Finally, on the first server I restarted mysql, logged in, and ran the following commands:

mysql> SET GLOBAL GROUP_REPLICATION_BOOTSTRAP_GROUP=ON;
Query OK, 0 rows affected (0.00 sec)

mysql> START GROUP_REPLICATION;
Query OK, 0 rows affected (2.11 sec)

mysql> SET GLOBAL GROUP_REPLICATION_BOOTSTRAP_GROUP=OFF;
Query OK, 0 rows affected (0.00 sec)

mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | 111-111-111-111-111111-111111-111111 | AA.AAA.AA.A |        3306 | ONLINE       |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
1 row in set (0.00 sec)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

mysql> SET GLOBAL GROUP_REPLICATION_BOOTSTRAP_GROUP=ON;

Query OK, 0 rows affected (0.00 sec)

mysql> START GROUP_REPLICATION;

Query OK, 0 rows affected (2.11 sec)

mysql> SET GLOBAL GROUP_REPLICATION_BOOTSTRAP_GROUP=OFF;

Query OK, 0 rows affected (0.00 sec)

mysql> SELECT * FROM performance_schema.replication_group_members;

+---------------------------+--------------------------------------+-------------+-------------+--------------+

+---------------------------+--------------------------------------+-------------+-------------+--------------+

+---------------------------+--------------------------------------+-------------+-------------+--------------+

1 row in set (0.00 sec)

On the second server I restarted mysql, logged in, and ran the following commands:

mysql> START GROUP_REPLICATION;
Query OK, 0 rows affected (2.11 sec)

mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | 111-111-111-111-111111-111111-111111 | AA.AAA.AA.A |        3306 | ONLINE       |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | 111-111-111-111-111111-111111-111111 | BBB.BBB.BBB.BBB |    3306 | ONLINE       |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
1 row in set (0.00 sec)

1

2

3

4

5

6

7

8

9

10

11

12

13

mysql> START GROUP_REPLICATION;

Query OK, 0 rows affected (2.11 sec)

mysql> SELECT * FROM performance_schema.replication_group_members;

+---------------------------+--------------------------------------+-------------+-------------+--------------+

+---------------------------+--------------------------------------+-------------+-------------+--------------+

+---------------------------+--------------------------------------+-------------+-------------+--------------+

+---------------------------+--------------------------------------+-------------+-------------+--------------+

1 row in set (0.00 sec)

Now they are communicating and connected!