Cassandra集群管理-節點異常重啟

Cassandra集群管理-節點異常重啟

登陸一臺集群節點，直接重啟服務器(172.20.101.166)，設置了 cassandra 開機啟動。

注意:

本文檔只是體系文檔中的一部分，前面文檔信息詳見：
測試準備+下線正常節點：https://blog.51cto.com/michaelkang/2419518
節點異常重啟：https://blog.51cto.com/michaelkang/2419524
添加新節點：https://blog.51cto.com/michaelkang/2419521
刪除異常節點：https://blog.51cto.com/michaelkang/2419525

場景：

節點被異常重啟，對集群引發的反應。

cassandra.log 基本沒有輸出

tailf /var/log/cassandra/cassandra.log 

system.log

有明顯日志報 172.20.101.166 DOWN ?。?！

172.20.101.165 節點：

[root@kubm-03 lib]# tailf /var/log/cassandra/system.log 
INFO  [GossipStage:1] 2019-07-11 18:19:23,372 Gossiper.java:1026 - InetAddress /172.20.101.166 is now DOWN

查看異常節點

[root@kubm-01 ~]# nodetool describecluster
Cluster Information:
        Name: pttest
        Snitch: org.apache.cassandra.locator.GossipingPropertyFileSnitch
        DynamicEndPointSnitch: enabled
        Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
        Schema versions:
                cfce5a85-19c8-327a-ab19-e1faae2358f7: [172.20.101.164, 172.20.101.165, 172.20.101.167, 172.20.101.160, 172.20.101.157]

                UNREACHABLE: [172.20.101.166]

debug.log

大量報無法連接 172.20.101.166

172.20.101.164 節點：

tailf /var/log/cassandra/debug.log

DEBUG [GossipStage:1] 2019-07-11 18:19:23,374 OutboundTcpConnection.java:205 - Enqueuing socket close for /172.20.101.166
DEBUG [MessagingService-Outgoing-/172.20.101.166-Small] 2019-07-11 18:19:23,374 OutboundTcpConnection.java:411 - Socket to /172.20.101.166 closed
DEBUG [GossipStage:1] 2019-07-11 18:19:23,374 OutboundTcpConnection.java:205 - Enqueuing socket close for /172.20.101.166
DEBUG [MessagingService-Outgoing-/172.20.101.166-Gossip] 2019-07-11 18:19:23,374 OutboundTcpConnection.java:411 - Socket to /172.20.101.166 closed
DEBUG [GossipStage:1] 2019-07-11 18:19:23,374 FailureDetector.java:313 - Forcing conviction of /172.20.101.166
DEBUG [MessagingService-Outgoing-/172.20.101.166-Gossip] 2019-07-11 18:19:24,740 OutboundTcpConnection.java:425 - Attempting to connect to /172.20.101.166
INFO  [HANDSHAKE-/172.20.101.166] 2019-07-11 18:19:24,741 OutboundTcpConnection.java:561 - Handshaking version with /172.20.101.166
DEBUG [MessagingService-Outgoing-/172.20.101.166-Gossip] 2019-07-11 18:19:24,742 OutboundTcpConnection.java:533 - Done connecting to /172.20.101.166

驗證查詢

系統啟動后，服務自然啟動，能正常加入集群。

cassandra@cqlsh> SELECT * from kevin_test.t_users; 

 user_id | emails                          | first_name | last_name
---------+---------------------------------+------------+-----------
       6 | {'k6-6@gmail.com', 'k6@pt.com'} |     kevin6 |      kang
       7 | {'k7-7@gmail.com', 'k7@pt.com'} |     kevin7 |      kang
       9 | {'k9-9@gmail.com', 'k9@pt.com'} |     kevin9 |      kang
       4 | {'k4-4@gmail.com', 'k4@pt.com'} |     kevin4 |      kang
       3 | {'k3-3@gmail.com', 'k3@pt.com'} |     kevin3 |      kang
       5 | {'k5-5@gmail.com', 'k5@pt.com'} |     kevin5 |      kang
       0 | {'k0-0@gmail.com', 'k0@pt.com'} |     kevin0 |      kang
       8 | {'k8-8@gmail.com', 'k8@pt.com'} |     kevin8 |      kang
       2 | {'k2-2@gmail.com', 'k2@pt.com'} |     kevin2 |      kang
       1 | {'k1-1@gmail.com', 'k1@pt.com'} |     kevin1 |      kang

測試結果：

反復重啟節點，查詢表內容正常。