centos下hadoop2.6.0集群搭建詳細過程

一 、centos集群環境配置 1.創建一個namenode節點，5個datanode節點

主機名IPnamenodezsw192.168.129.158datanode1zsw192.168.129.159datanode2zsw192.168.129.160datanode3zsw192.168.129.161datanode4zsw192.168.129.162datanode5zsw192.168.129.1632.關閉防火牆，設置selinux為disabled #service iptables stop

#chkconfig iptables off

設置selinux為disabled,如下圖所示 #vim /etc/selinux/config

重啟後生效，建議全部配置完成後再全部重啟。

3.配置hosts文件

將各節點對應的主機名與IP地址記錄在hosts文件中

vi /etc/hosts

4.配置java環境

本文所用java安裝包為jdk-7u25-linux-x64.tar.gz(百度上各種版本的都有)

解壓到/opt目錄下

修改環境變量：

#vi /etc/profile

在文件末尾添加以下內容

export JAVA_HOME=/opt/jdk1.7.0_25

export JAR_HOME=/opt/jdk1.7.0_25/jre

export ClASSPATH=$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$JAR_HOME/lib

export PATH=$JAVA_HOME/bin:$PATH

保存後退出，執行

#source \etc\profile

查看java版本:

#java -version

顯示上圖結果說明配置成功

注意：使用source \etc\profile命令盡在本終端有效，重啟後才全部生效

提示：namenode節點和datanode節點上述配置都相同 二、設置各節點間SSH無密碼通信

在一個節點namenodezsw上操作

1.生成密鑰對

#ssh-keygen -t rsa

按三次回車，結果如下圖

2.將各個節點生成的公有密鑰添加到authorized_keys

#cat .ssh/id_rsa.pub > .ssh/authorized_keys

#ssh 192.168.129.159 ssh-keygen -t rsa

#ssh 192.168.129.159 cat ~/.ssh/id_rsa.pub >>.ssh/authorized_keys

#ssh 192.168.129.160 ssh-keygen -t rsa

#ssh 192.168.129.160 cat ~/.ssh/id_rsa.pub >>.ssh/authorized_keys

#ssh 192.168.129.161 ssh-keygen -t rsa

#ssh 192.168.129.161 cat ~/.ssh/id_rsa.pub >>.ssh/authorized_keys

#ssh 192.168.129.162 ssh-keygen -t rsa

#ssh 192.168.129.162 cat ~/.ssh/id_rsa.pub >>.ssh/authorized_keys

#ssh 192.168.129.163 ssh-keygen -t rsa

#ssh 192.168.129.163 cat ~/.ssh/id_rsa.pub >>.ssh/authorized_keys

3.將authorized_keys文件傳到各個節點

#scp /root/.ssh/authorized_keys [email protected]:/root/.ssh/authorized_keys

#scp /root/.ssh/authorized_keys [email protected]:/root/.ssh/authorized_keys

#scp /root/.ssh/authorized_keys [email protected]:/root/.ssh/authorized_keys

#scp /root/.ssh/authorized_keys [email protected]:/root/.ssh/authorized_keys

#scp /root/.ssh/authorized_keys [email protected]:/root/.ssh/authorized_keys

4.測試

#ssh 192.168.129.159 date

#ssh datanode1zsw date

1個namenode節點，5個datanode節點以此類推

兩次ssh命令：

第一遍都需要輸入yes，然後顯示時間

第二遍則直接顯示時間，說明配置成功

說明：每個節點上都要ssh命令其他的節點主機名和IP一下，檢測是否可以

三、hadoop集群搭建

1.去hadoop官網或者百度下載hadoop-2.6.0-tar.gz，然後解壓到namenode的/opt目錄下

2.修改配置文件

#cd /opt/hadoop-2.6.0/etc/hadoop

①修改 hadoop-env.sh和yarn-env.sh

export JAVA_HOME=/opt/jdk1.7.0_25

②修改core-site.xml

[code]<configuration>

         <property>

               <name>hadoop.tmp.dir</name>

                <value>/opt/hadoop-2.6.0/tmp</value>

               <description>Abase for other temporarydirectories.</description>

       </property>

       <property>

               <name>fs.defaultFS</name>

               <value>hdfs://namenodezsw:9000</value>

        </property>

       <property>

               <name>io.file.buffer.size</name>

               <value>4096</value>

       </property>

</configuration>

[code]configuration>

   <property>

         <name>dfs.namenode.name.dir</name>

         <value>file:///opt/hadoop-2.6.0/dfs/name</value>

   </property>

       <property>

        <name>dfs.datanode.data.dir</name>

        <value>file:///opt/hadoop-2.6.0/dfs/data</value>

   </property>

   <property>

        <name>dfs.replication</name>

        <value>2</value>

   </property>

   <property>

       <name>dfs.nameservices</name>

       <value>h1</value>

   </property>

   <property>

       <name>dfs.namenode.secondary.http-address</name>

       <value>namenodezsw:50090</value>

   </property>

    <property>

       <name>dfs.webhdfs.enabled</name>

       <value>true</value>

   </property>

</configuration>

cp mapred-site.xml.template mapred-site.xml

[code]<configuration>

   <property>

       <name>mapreduce.framework.name</name>

       <value>yarn</value>

       <final>true</final>

   </property>

   <property>

       <name>mapreduce.jobtracker.http.address</name>

       <value>namenodezsw:50030</value>

   </property>

   <property>

       <name>mapreduce.jobhistory.address</name>

       <value>namenodezsw:10020</value>

   </property>

   <property>

       <name>mapreduce.jobhistory.webapp.address</name>

       <value>namenodezsw:19888</value>

   </property>

   <property>

       <name>mapred.job.tracker</name>

        <value>http://namenodezsw:9001</value>

   </property>

</configuration>

[code]<configuration> 

<!-- Site specific YARN configuration properties-->

   <property>

       <name>yarn.resourcemanager.hostname</name>

       <value>namenodezsw</value>

    </property>

   <property>

       <name>yarn.nodemanager.aux-services</name>

       <value>mapreduce_shuffle</value>

   </property>

   <property>

       <name>yarn.resourcemanager.address</name>

       <value>namenodezsw:8032</value>

   </property>

   <property>

       <name>yarn.resourcemanager.scheduler.address</name>

        <value>namenodezsw:8030</value>

   </property>

   <property>

       <name>yarn.resourcemanager.resource-tracker.address</name>

       <value>namenodezsw:8031</value>

   </property>

   <property>

       <name>yarn.resourcemanager.admin.address</name>

       <value>namenodezsw:8033</value>

   </property>

   <property>

       <name>yarn.resourcemanager.webapp.address</name>

       <value>namenodezsw:8088</value>

   </property>

</configuration>

⑦配置環境變量

# vi /etc/profile

export JAVA_HOME=/opt/jdk1.7.0_25

export HADOOP_HOME=/opt/hadoop-2.6.0

export JAR_HOME=/opt/jdk1.7.0_25/jre

export CLASSPATH=$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$JAR_HOME/lib

export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

環境變量配置同樣source /etc/profile命令本終端有效，重啟後都有效

至此，修改完成

3.啟動hadoop

使用scp -r /opt/hadoop-2.6.0/ hostname:/opt/命令將hadoop文件依次拷貝到5個datanode節點上。

在namenode上操作：

首先格式化

#hadoop namenode -format

啟動所有服務

#start-all.sh

4.驗證

namenode上：

#jps

datanode上：

#jps

網頁管理端:

也可以使用 hadoop dfsadmin -report命令查看總體情況

至此集群搭建全部完成