今天就跟大家聊聊有關kafka中怎么增加topic的備份數量，可能很多人都不太了解，為了讓大家更加了解，小編給大家總結了以下內容，希望大家根據這篇文章可以有所收獲。
建立topic的時候，可以通過指定參數 --replication-factor 設置備份數量。但是，一旦完成建立topic，則無法通過kafka-topic.sh 或者 命令修改replica數量。
實際上，我們可以考慮一種 “另類” 的辦法：可以利用 kafka-reassign-partitions.sh 命令對所有分區進行重新分布，在做分區重新分布的時候，通過增加每個分區的replica備份數量來達到目的。
本文將介紹如何利用 kafka-reassign-partitions.sh 命令增加topic的備份數量。
注意：以下命令使用到的topic名稱、zookeeper的ip和port，需要讀者替換成為實際集群的參數。
（假設kafka集群有4個broker，id分別為：1001,1002,1003,1004）
[root@tbds bin]# ./kafka-topics.sh --zookeeper 172.16.32.13:2181 --topic ranger_audits --describe Topic:ranger_audits PartitionCount:10 ReplicationFactor:1 Configs: Topic: ranger_audits Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: ranger_audits Partition: 1 Leader: 1002 Replicas: 1002 Isr: 1002 Topic: ranger_audits Partition: 2 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: ranger_audits Partition: 3 Leader: 1002 Replicas: 1002 Isr: 1002 Topic: ranger_audits Partition: 4 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: ranger_audits Partition: 5 Leader: 1002 Replicas: 1002 Isr: 1002 Topic: ranger_audits Partition: 6 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: ranger_audits Partition: 7 Leader: 1002 Replicas: 1002 Isr: 1002 Topic: ranger_audits Partition: 8 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: ranger_audits Partition: 9 Leader: 1002 Replicas: 1002 Isr: 1002
可以看出，ranger_audits 這個topic有10個分區，每個分區只有一個feplica備份，分布在1001和1002兩臺broker上面。
下面我們需要將ranger_audits 的每個分區數據都增加到2個replica備份，且分布到4個broker上面。
（注意：盡量保持topic的原有每個分區的主備份不變化。因此，配置文件的每個分區的第一個broker保持不變。）
[root@tbds bin]# vim ../config/increase-replication-factor.json
{"version":1,
"partitions":[
{"topic":"ranger_audits","partition":0,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":1,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":2,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":3,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":4,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":5,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":6,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":7,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":8,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":9,"replicas":[1002,1004]}
]}上面的配置文件說明，我們將topic的每個分區都增加了一個replica，且保持每個分區原有的主備份所在broker不變化，將每個分區新增的replica備份數據放到到1003和1004兩個broker上面。
[root@tbds bin]# ./kafka-reassign-partitions.sh -zookeeper 172.16.32.13:2181 --reassignment-json-file ../config/increase-replication-factor.json --execute
Current partition replica assignment
{"version":1,"partitions":[{"topic":"ranger_audits","partition":3,"replicas":[1002]},{"topic":"ranger_audits","partition":9,"replicas":[1002]},{"topic":"ranger_audits","partition":8,"replicas":[1001]},{"topic":"ranger_audits","partition":1,"replicas":[1002]},{"topic":"ranger_audits","partition":4,"replicas":[1001]},{"topic":"ranger_audits","partition":2,"replicas":[1001]},{"topic":"ranger_audits","partition":5,"replicas":[1002]},{"topic":"ranger_audits","partition":0,"replicas":[1001]},{"topic":"ranger_audits","partition":6,"replicas":[1001]},{"topic":"ranger_audits","partition":7,"replicas":[1002]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions
{"version":1,"partitions":[{"topic":"ranger_audits","partition":0,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":8,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":5,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":2,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":9,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":1,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":3,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":4,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":7,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":6,"replicas":[1001,1003]}]}[root@tbds bin]# ./kafka-reassign-partitions.sh -zookeeper 172.16.32.13:2181 --reassignment-json-file ../config/increase-replication-factor.json --verify Status of partition reassignment: Reassignment of partition [ranger_audits,0] completed successfully Reassignment of partition [ranger_audits,8] completed successfully Reassignment of partition [ranger_audits,5] completed successfully Reassignment of partition [ranger_audits,2] completed successfully Reassignment of partition [ranger_audits,9] completed successfully Reassignment of partition [ranger_audits,1] completed successfully Reassignment of partition [ranger_audits,3] completed successfully Reassignment of partition [ranger_audits,4] completed successfully Reassignment of partition [ranger_audits,7] completed successfully Reassignment of partition [ranger_audits,6] completed successfully
上面顯示增加分區操作成功
[root@tbds bin]# ./kafka-topics.sh --zookeeper 172.16.32.13:2181 --topic ranger_audits --describe Topic:ranger_audits PartitionCount:10 ReplicationFactor:2 Configs: Topic: ranger_audits Partition: 0 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003 Topic: ranger_audits Partition: 1 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004 Topic: ranger_audits Partition: 2 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003 Topic: ranger_audits Partition: 3 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004 Topic: ranger_audits Partition: 4 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003 Topic: ranger_audits Partition: 5 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004 Topic: ranger_audits Partition: 6 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003 Topic: ranger_audits Partition: 7 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004 Topic: ranger_audits Partition: 8 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003 Topic: ranger_audits Partition: 9 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004
從上面可以看出，備份數量增加成功
利用上述介紹的辦法，除了可以用來增加topic的備份數量之外，還能夠處理以下幾個場景：
**1、對topic的所有分區數據進行整體遷移。**怎么理解呢？假如集群有N個broker，后來新擴容M個broker。由于新擴容的broker磁盤都是空的，原有的broker磁盤占用都很滿。那么我們可以利用上述方法，將存儲在原有N個broker的某些topic整體搬遷到新擴容的M個broker，進而實現kafka集群的整體數據均衡。
具體使用方法就是：通過編寫2.2章節的配置文件，將topic的所有分區都配置到新的M個broker上面去，再執行excute，即可完成topic的所有分區數據整體遷移到新擴容的M個broker節點。
***2、broker壞掉的情況。***導致某些topic的某些分區的replica數量減少，可以利用kafka-reassign-partitions.sh增加replica；
***3、kafka 某些broker磁盤占用很滿，某些磁盤占用又很少。***可以利用kafka-reassign-partitions.sh遷移某些topic的分區數據到磁盤占用少的broker，實現數據均衡；
***4、kafka集群擴容。***需要把原來broker的topic數據整體遷移到新的broker，合理利用新擴容的broker，實現負載均衡。
看完上述內容，你們對kafka中怎么增加topic的備份數量有進一步的了解嗎？如果還想了解更多知識或者相關內容，請關注億速云行業資訊頻道，感謝大家的支持。
免責聲明：本站發布的內容（圖片、視頻和文字）以原創、轉載和分享為主，文章觀點不代表本網站立場，如果涉及侵權請聯系站長郵箱：is@yisu.com進行舉報，并提供相關證據，一經查實，將立刻刪除涉嫌侵權內容。
