在CentOS上配置HDFS高可用性(HA)涉及多個步驟,包括安裝和配置Hadoop集群、設置ZooKeeper、配置HDFS屬性等。以下是一個基本的指南,幫助你在CentOS上實現HDFS高可用性。
首先,確保所有節點上安裝了Java環境。
sudo yum install java-1.8.0-openjdk-devel
編輯/etc/profile
文件,添加Hadoop路徑:
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
source /etc/profile
下載Hadoop安裝包并解壓到指定目錄:
wget https://downloads.apache.org/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz
tar -xzf hadoop-3.3.0.tar.gz -C /usr/local/
編輯/usr/local/hadoop/etc/hadoop/core-site.xml
文件,添加以下內容:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://mycluster</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>hadoop1:2181,hadoop2:2181,hadoop3:2181</value>
</property>
</configuration>
編輯/usr/local/hadoop/etc/hadoop/hdfs-site.xml
文件,添加以下內容:
<configuration>
<property>
<name>dfs.nameservices</name>
<value>mycluster</value>
</property>
<property>
<name>dfs.ha.namenodes.mycluster</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn1</name>
<value>hadoop1:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn2</name>
<value>hadoop2:8020</value>
</property>
<property>
<name>dfs.namenode.http-address.mycluster.nn1</name>
<value>hadoop1:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.mycluster.nn2</name>
<value>hadoop2:50070</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://hadoop1:8485;hadoop2:8485;hadoop3:8485/mycluster</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.mycluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
</configuration>
下載并解壓ZooKeeper:
wget https://downloads.apache.org/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
tar -xzf zookeeper-3.4.14.tar.gz -C /usr/local/
編輯/usr/local/zookeeper-3.4.14/conf/zoo.cfg
文件,添加以下內容:
dataDir=/usr/local/zookeeper-3.4.14/data
clientPort=2181
server.1.name=hadoop1
server.1.port=2888
server.2.name=hadoop2
server.2.port=2889
server.3.name=hadoop3
server.3.port=2890
在每個ZooKeeper節點上創建myid
文件:
echo 1 > /usr/local/zookeeper-3.4.14/data/myid
echo 2 > /usr/local/zookeeper-3.4.14/data/myid
echo 3 > /usr/local/zookeeper-3.4.14/data/myid
啟動ZooKeeper服務:
/usr/local/zookeeper-3.4.14/bin/zkServer.sh start
在主NameNode節點上格式化:
hdfs namenode -format
啟動HDFS和YARN服務:
/usr/local/hadoop/sbin/start-dfs.sh
/usr/local/hadoop/sbin/start-yarn.sh
使用以下命令驗證HDFS是否正常運行:
hdfs dfs -ls /
通過以上步驟,你可以在CentOS上配置一個高可用的HDFS集群。根據具體需求,你可能需要進一步調整和優化配置。