在CentOS上實現HDFS高可用性(High Availability, HA)主要涉及配置主備NameNode、使用ZooKeeper進行狀態監控和故障切換,以及配置JournalNode來共享編輯日志以實現數據同步。以下是詳細的步驟和配置說明:
安裝和配置ZooKeeper
下載并解壓ZooKeeper:
wget https://downloads.apache.org/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz
tar -xzf apache-zookeeper-3.8.0-bin.tar.gz
cd apache-zookeeper-3.8.0
配置ZooKeeper:
編輯 conf/zoo.cfg
文件,設置數據目錄和客戶端連接端口:
dataDir /var/lib/zookeeper
clientPort 2181
在每個NameNode節點上創建 myid
文件:
echo 1 > /var/lib/zookeeper/myid
啟動ZooKeeper服務:
./bin/zkServer.sh start
配置Hadoop
修改 hdfs-site.xml
文件:
<configuration>
<property>
name dfs.nameservices / name value mycluster / value /
</property>
<property>
name dfs.ha.namenodes.mycluster / name value nn1,nn2 / value /
</property>
<property>
name dfs.namenode.rpc-address.mycluster.nn1 / name value namenode1:8020 / value /
</property>
<property>
name dfs.namenode.rpc-address.mycluster.nn2 / name value namenode2:8020 / value /
</property>
<property>
name dfs.namenode.http-address.mycluster.nn1 / name value namenode1:50070 / value /
</property>
<property>
name dfs.namenode.http-address.mycluster.nn2 / name value namenode2:50070 / value /
</property>
<property>
name dfs.namenode.shared.edits.dir / name value qjournal://journalnode1:8485;journalnode2:8485;journalnode3:8485/mycluster / value /
</property>
<property>
name dfs.client.failover.proxy.provider.mycluster / name value org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider / value /
</property>
<property>
name dfs.ha.fencing.methods / name value sshfence / value /
</property>
<property>
name dfs.ha.fencing.ssh.private-key-files / name value /root/.ssh/id_rsa / value /
</property>
</configuration>
修改 core-site.xml
文件:
<configuration>
<property>
name fs.defaultFS / name value hdfs://mycluster / value /
</property>
</configuration>
格式化和啟動NameNode
在Active NameNode上格式化NameNode:
hdfs namenode -format
啟動NameNode:
sbin/hadoop-daemon.sh start namenode
將Active NameNode的元數據復制到Standby NameNode:
hdfs namenode -bootstrapStandby
啟動Standby NameNode:
sbin/hadoop-daemon.sh start namenode
啟動JournalNode
sbin/hadoop-daemon.sh start journalnode
驗證高可用性
http://namenode1:50070
查看NameNode狀態。通過以上步驟,您可以在CentOS上配置一個高可用的HDFS集群,確保在發生故障時能夠快速切換并繼續提供服務。