hadoop on yarn搭建

1、下载
https://hadoop.apache.org/releases.html
jdk: https://bell-sw.com/pages/downloads/#jdk-8-lts

2、网络配置 (/etc/hosts)

192.168.0.110 master
192.168.0.140 slave1

3、ssh免密配置

# 在master主机执行
ssh-keygen -t rsa
ssh-copy-id master
ssh-copy-id worker1

4、环境变量配置 (/etc/profile)

export JAVA_HOME=/usr/local/jdk8
export HADOOP_HOME=/usr/local/hadoop34
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native:$LD_LIBRARY_PATH
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

5、关闭主机防火墙

systemctl disable --now firewalld.service

6、hadoop配置
一. 共同配置（Master 和 Slave 都需要）
core-site.xml:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/hadoop/tmp</value>
    </property>
</configuration>

hdfs-site.xml:

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>  <!-- 因为只有1个slave -->
    </property>
</configuration>

hadoop-env.sh:

export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

export JAVA_HOME=/usr/local/jdk8

二. Master 节点配置
yarn-site.xml:

<configuration>
    <!-- ResourceManager 配置 -->
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    
    <!-- 其他可选配置 -->
    <property>
        <name>yarn.resourcemanager.scheduler.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    </property>
</configuration>

mapred-site.xml:

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

workers 文件:

slave1

三. Slave 节点配置
yarn-site.xml:

<configuration>
    <!-- 指向 Master 节点的 ResourceManager -->
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
</configuration>

mapred-site.xml (与 Master 相同):

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

7、启动
在 Master 上启动 HDFS

start-dfs.sh

在 Master 上启动 YARN:

start-yarn.sh

8、访问Web界面
HDFS: http://master:9870
YARN: http://master:8088

补充:
修改从主机的hostname

hostnamectl set-hostname slave1