溫馨提示×

溫馨提示×

您好，登錄后才能下訂單哦！

密碼登錄×
登錄注冊×
獲取短信驗證碼
其他方式登錄
點擊 登錄注冊 即表示同意《億速云用戶服務條款》
登 錄 注冊有禮
云服務器 香港服務器 高防服務器
最新更新 網站標簽 地圖導航
產品

hadoop-006完全分布式問題有哪些

發布時間：2021-12-08 09:58:33 來源：億速云 閱讀：334 作者：小新 欄目：云計算

小編給大家分享一下hadoop-006完全分布式問題有哪些，希望大家閱讀完這篇文章之后都有所收獲，下面讓我們一起去探討吧！

一、運行mapreduce 任務，但是在yarn里看不到任務。

bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar  wordcount /test /out1

我配置的resourceManager 為 http://192.168.31.136:8088

原因是在 mapred-site.xml 沒有配置

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

二、配置好問題一后，運行mr任務提示The auxService:mapreduce_shuffle does not exist

原因是在yarn-site.xml 沒有配置yarn.nodemanager.aux-services 節點

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

報錯如下

16/11/29 23:10:45 INFO mapreduce.Job: Task Id : attempt_1480432102879_0001_m_000000_2, Status : FAILED
Container launch failed for container_e02_1480432102879_0001_01_000004 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

-----------------------------------------------------------------------------------------------------------------

配置文件記錄

已提前搭建好三個節點的zookeeper 小集群，配置可以實現 HDFS?。龋梁汀。伲粒遥巍。龋?/p>

一、hadoop-env.sh 修改了

export JAVA_HOME=/usr/lib/jvm/jdk8/jdk1.8.0_111

二、yarn-env.sh 修改了

export JAVA_HOME=/usr/lib/jvm/jdk8/jdk1.8.0_111

三、core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<!-- 指定nameservice的名稱為mycluster -->
    <property>
      <name>fs.defaultFS</name>
      <value>hdfs://mycluster</value>
    </property>
    <!-- 編輯日志文件存儲的路徑 -->
    <property>
      <name>dfs.journalnode.edits.dir</name>
      <value>/home/jxlgzwh/hadoop-2.7.2/data/jn</value>
    </property>
    
    <!-- 配置緩存文件的目錄 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/jxlgzwh/hadoop-2.7.2/data/tmp</value>
    </property>
    <property>
       <name>ha.zookeeper.quorum</name>
       <value>master:2181,slave01:2181,slave02:2181</value>
     </property>
</configuration>

 

四、hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<!-- 指定hdfs的nameservice的名稱為mycluster -->
    <property>
      <name>dfs.nameservices</name>
      <value>mycluster</value>
    </property>
    <!-- 指定mycluster的兩個namenode的名稱，分別是nn1，nn2 -->
    <property>
      <name>dfs.ha.namenodes.mycluster</name>
      <value>nn1,nn2</value>
    </property>
    
    <!-- 配置nn1，nn2的rpc通信 端口    -->
    <property>
      <name>dfs.namenode.rpc-address.mycluster.nn1</name>
      <value>master:8020</value>
    </property>
    <property>
      <name>dfs.namenode.rpc-address.mycluster.nn2</name>
      <value>slave01:8020</value>
    </property>
    
    <!-- 配置nn1，nn2的http訪問端口 -->
    <property>
      <name>dfs.namenode.http-address.mycluster.nn1</name>
      <value>master:50070</value>
    </property>
    <property>
      <name>dfs.namenode.http-address.mycluster.nn2</name>
      <value>slave01:50070</value>
    </property>
    <!-- 指定namenode的元數據存儲在journalnode中的路徑 -->
    <property>
      <name>dfs.namenode.shared.edits.dir</name>
      <value>qjournal://master:8485;slave01:8485;slave02:8485/mycluster</value>
    </property>
    <!-- 配置失敗自動切換的方式 -->
    <property>
      <name>dfs.client.failover.proxy.provider.mycluster</name>
      <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    <!-- 配置隔離機制 -->
     <property>
      <name>dfs.ha.fencing.methods</name>
      <value>sshfence</value>
    </property>
    <!-- 指定秘鑰的位置 -->
    <property>
      <name>dfs.ha.fencing.ssh.private-key-files</name>
      <value>/home/jxlgzwh/.ssh/id_dsa</value>
    </property>
    <!-- 數據備份的個數 -->
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
        <!--關閉權限驗證 -->
    <property>
        <name>dfs.permissions.enabled</name>
        <value>false</value>
    </property>
    
    <property>
       <name>dfs.ha.automatic-failover.enabled</name>
       <value>true</value>
     </property>
    
</configuration>

 

五、mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
</configuration>

 

六、slaves

192.168.31.136
192.168.31.130
192.168.31.229

 

七、yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<property>
  <name>yarn.resourcemanager.ha.enabled</name>
  <value>true</value>
</property>
<property>
  <name>yarn.resourcemanager.cluster-id</name>
  <value>cluster1</value>
</property>
<property>
  <name>yarn.resourcemanager.ha.rm-ids</name>
  <value>rm1,rm2</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname.rm1</name>
  <value>master</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname.rm2</name>
  <value>slave01</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address.rm1</name>
  <value>master:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address.rm2</name>
  <value>slave01:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.zk-address</name>
  <value>master:2181,slave01:2181,slave02:2181</value>
</property>

<!--  yarn restart-->
<!-- 開啟resourcemanager restart -->
<property>
  <name>yarn.resourcemanager.recovery.enabled</name>
  <value>true</value>
</property>
<!-- 配置zookeeper的存儲位置 -->
<property>
  <name>yarn.resourcemanager.zk-state-store.parent-path</name>
  <value>/rmstore</value>
</property>

<!-- 配置存儲到zookeeper中 -->
<property>
  <name>yarn.resourcemanager.store.class</name>
  <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>
<!-- 開啟nodemanager restart -->
<property>
  <name>yarn.nodemanager.recovery.enabled</name>
  <value>true</value>
</property>
<!-- 指定nodemanager 存儲端口 -->
<property>
  <name>yarn.nodemanager.address</name>
  <value>0.0.0.0:45454</value>
</property>
<!-- 開啟nodemanager 存儲目錄-->
<property>
  <name>yarn.nodemanager.recovery.dir</name>
  <value>/home/jxlgzwh/hadoop-2.7.2/data/tmp/yarn-nm-recovery</value>
</property>
<!-- Site specific YARN configuration properties -->
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
</configuration>

八、配置 /etc/hosts 文件 并配置 ssh免密碼登錄

192.168.31.136 master.com master
192.168.31.130 slave01
192.168.31.229 slave02

看完了這篇文章，相信你對“hadoop-006完全分布式問題有哪些”有了一定的了解，如果想了解更多相關知識，歡迎關注億速云行業資訊頻道，感謝各位的閱讀！

向AI問一下細節
推薦閱讀：
  1. Hadoop完全分布式配置
  2. Hadoop完全分布式部署

免責聲明：本站發布的內容（圖片、視頻和文字）以原創、轉載和分享為主，文章觀點不代表本網站立場，如果涉及侵權請聯系站長郵箱：is@yisu.com進行舉報，并提供相關證據，一經查實，將立刻刪除涉嫌侵權內容。

hadoop

猜你喜歡

最新資訊
相關推薦

相關標簽

hadoophdfs hadoop集群 hadoop安裝 hadoop集群搭建 hadoop生態圈 hadoop安裝配置 hadoop序列化 hadoop hbase dkhadoop hadoop yarn hadoop3.x hadoop2.6.0 hadoop應用 hadoop2 apache hadoop wcf itunes uniq handler yii
AI
助
手
產品服務
地區劃分
專題活動
幫助支持
關于我們
售后咨詢
7*24小時在線電話：400-100-2938
7*24小時在線 QQ：800811969
關注億速云

億速云公眾號

手機網站二維碼

Copyright ? Yisu Cloud Ltd. All Rights Reserved. 2018 版權所有

廣州億速云計算有限公司粵ICP備17096448號-1 粵公網安備 44010402001142號增值電信業務經營許可證編號：B1-20181529

亚洲午夜精品一区二区_中文无码日韩欧免_久久香蕉精品视频_欧美主播一区二区三区美女