第1章Hadoop集群搭建 Ver1.6-20230921

Imagemap

hide

第1章Hadoop集群搭建
Ver1.6-20230921

hide

VMware上安装Hadoop3.x

hide

安装及配置虚拟机

hide

wmware安装与Linux安装

leaf

安装WMWare Workstation10，注册

hide

配置wmware的虚拟网络

leaf

编辑->虚拟网络编辑器

hide

添加网络->选择Nat模式->NAT设置

leaf

网关:192.168.137.2

leaf

选中:将主机虚拟适配器连接到此网络
选中:使用本地DHCP服务将IP地址分配给虚拟机

leaf

子网IP:192.168.137.0,子网掩码:255.255.255.0

hide

在WMWare上创建虚拟机

hide

创建master主机

leaf

选择默认的:典型

leaf

下一步，选择安装程序光盘映像文件->浏览->前面下载的文件:
CentOS-7-x86_64-Minimal-2009.iso

leaf

下一步，选中Linux操作系统，CentOS64位

leaf

下一步，设置虚拟机名称:CentOS_Hadoop3.x_master,选择位置

leaf

下一步，最大磁盘48G，将虚拟磁盘存储为单个文件

leaf

完成设置，开始安装

hide

远程连接虚拟机

hide

leaf

crt工具安装

leaf

crt工具的配置与连接虚拟机

leaf

以下linux操作如无说明均在crt里操作

hide

hide

修改显示主机名

leaf

hostnamectl set-hostname master

hide

修改网络主机名

hide

vi /etc/sysconfig/network

leaf

HOSTNAME=master

leaf

systemctl restart network

hide

leaf

说明:替换下面这些IP和名字后在命令行执行这些echo命令

leaf

echo '' > /etc/hosts
echo '192.168.137.100 master master.centos.com' >> /etc/hosts
echo '192.168.137.101 slave1 slave1.centos.com' >> /etc/hosts
echo '192.168.137.102 slave2 slave2.centos.com' >> /etc/hosts
echo '192.168.137.103 slave3 slave3.centos.com' >> /etc/hosts

hide

leaf

yum -y install ntp

hide

虚拟机安装Java

hide

leaf

yum -y install wget

leaf

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak

leaf

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

hide

hide

rpm -qa|grep jdk
rpm -e --nodeps 刚查出来的jdk

leaf

[root@c23 ~]# rpm -qa|grep jdk
copy-jdk-configs-3.3-10.el7_5.noarch
java-1.8.0-openjdk-1.8.0.342.b07-1.el7_9.x86_64
java-1.8.0-openjdk-headless-1.8.0.342.b07-1.el7_9.x86_64
java-1.8.0-openjdk-devel-1.8.0.342.b07-1.el7_9.x86_64
[root@c23 ~]# rpm -e --nodeps java-1.8.0-openjdk-devel-1.8.0.342.b07-1.el7_9.x86_64
rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.342.b07-1.el7_9.x86_64
rpm -e --nodeps java-1.8.0-openjdk-1.8.0.342.b07-1.el7_9.x86_64
rpm -e --nodeps copy-jdk-configs-3.3-10.el7_5.noarch[root@c23 ~]# rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.342.b07-1.el7_9.x86_64

[root@c23 ~]# rpm -e --nodeps java-1.8.0-openjdk-1.8.0.342.b07-1.el7_9.x86_64
[root@c23 ~]# rpm -e --nodeps copy-jdk-configs-3.3-10.el7_5.noarch
[root@c23 ~]# rpm -qa|grep jdk
[root@c23 ~]#

leaf

运行结果如图

User Link

hide

leaf

sudo yum install -y java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64 --skip-broken

hide

hide

leaf

预期结果:
[root@c31 yum.repos.d]# java -version
openjdk version "1.8.0_322"
OpenJDK Runtime Environment (build 1.8.0_322-b06)
OpenJDK 64-Bit Server VM (build 25.322-b06, mixed mode)
[root@c31 yum.repos.d]#

hide

hide

leaf

预期结果最后一行可能包含如下路径:
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre

hide

leaf

sudo yum install -y java-1.8.0-openjdk-devel-1.8.0.322.b06-1.el7_9.x86_64

hide

虚拟机安装Hadoop集群

hide

配置主机master

hide

下载安装hadoop3.3.1

leaf

mkdir /root/hadoop
cd /root/hadoop
wget https://mirrors.aliyun.com/apache/hadoop/core/hadoop-3.3.1/hadoop-3.3.1.tar.gz
tar -zxf hadoop-3.3.1.tar.gz -C /usr/local

hide

配置主机hadoop3.3.1

hide

在/usr/local/hadoop-3.3.1/etc/hadoop目录下修改配置

leaf

cd /usr/local/hadoop-3.3.1/etc/hadoop

hide

vi ./core-site.xml

leaf

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9864</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop-3.3.1/tmp</value>
</property>
<property>
   <name>hadoop.native.lib</name>
   <value>false</value>
   <description>Should native hadoop libraries, if present, be used.
   </description>
</property>
</configuration>

hide

vi ./hadoop-env.sh

leaf

说明:在文件上方空白行处添加如下添加如下两行,注意Java_home路径参考java -verbose命令

leaf

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre

leaf

export HADOOP_HOME=/usr/local/hadoop-3.3.1

hide

vi ./hdfs-site.xml

leaf

<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///data/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///data/hadoop/hdfs/data</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:9868</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>

hide

vi ./mapred-site.xml

leaf

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
</configuration>

hide

vi ./yarn-site.xml

leaf

<configuration>


<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>${yarn.resourcemanager.hostname}:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>${yarn.resourcemanager.hostname}:8030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address</name>
<value>${yarn.resourcemanager.hostname}:8090</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>${yarn.resourcemanager.hostname}:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>${yarn.resourcemanager.hostname}:8033</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/data/hadoop/yarn/local</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/data/tmp/logs</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://master:19888/jobhistory/logs/</value>
<description>URL for job history server</description>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>128</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>2048</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>512</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>1024</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>2</value>
</property>
</configuration>

hide

vi ./yarn-env.sh

leaf

说明:在文件上方空白行处添加如下：

leaf

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre

hide

leaf

说明:删除已有的localhost这行，然后添加本集群中所有slave从机名，如下：

leaf

slave1
slave2
slave3

hide

vi ./hadoop-env.sh

leaf

说明:在刚才添加的export行后添加如下5行内容:

leaf

HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
HADOOP_SHELL_EXECNAME=root

hide

修改linux系统配置文件profile并使生效

hide

vi /etc/profile

leaf

说明:在文件最下方export行后添加如下：

leaf

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre
export HADOOP_HOME=/usr/local/hadoop-3.3.1
export PATH=$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$PATH:$JAVA_HOME/bin

leaf

source /etc/profile

hide

验证hadoop相关配置是否生效

hide

leaf

说明：如果显示如下内容说明hadoop配置生效
[root@c31 ~]# hadoop version
Hadoop 3.3.1
Source code repository https://github.com/apache/hadoop.git -r a3b9c37a397ad4188041dd80621bdeefc46885f2
Compiled by ubuntu on 2021-06-15T05:13Z
Compiled with protoc 3.7.1
From source with checksum 88a4ddb2299aca054416d6b7f81ca55
This command was run using /usr/local/hadoop-3.3.1/share/hadoop/common/hadoop-common-3.3.1.jar

hide

克隆从机slave

hide

从master克隆出slave

hide

主机master关机

leaf

leaf

wmware左侧树图->右击主机master->管理->克隆->虚拟机当前状态->创建完整克隆,选择存放目录

hide

hide

测试写速度

hide

cd /opt/hadoop/share/hadoop/mapreduce/
hadoop jar ./hadoop-mapreduce-client-jobclient-3.2.1-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 10MB

leaf

User Link

leaf

User Link

hide

hide

查看运行结果

leaf

2023-07-21 07:05:27,971 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
2023-07-21 07:05:27,972 INFO fs.TestDFSIO: Date & time: Fri Jul 21 07:05:27 GMT 2023
2023-07-21 07:05:27,973 INFO fs.TestDFSIO: Number of files: 10
2023-07-21 07:05:27,974 INFO fs.TestDFSIO: Total MBytes processed: 100
2023-07-21 07:05:27,975 INFO fs.TestDFSIO: Throughput mb/sec: 1.67
2023-07-21 07:05:27,976 INFO fs.TestDFSIO: Average IO rate mb/sec: 2.08
2023-07-21 07:05:27,977 INFO fs.TestDFSIO: IO rate std deviation: 0.94
2023-07-21 07:05:27,978 INFO fs.TestDFSIO: Test exec time sec: 202.62

hide

查看日志文件

leaf

cat TestDFSIO_results.log
----- TestDFSIO ----- : write
Date & time: Fri Jul 21 07:05:27 GMT 2023
Number of files: 10
Total MBytes processed: 100
Throughput mb/sec: 1.67
Average IO rate mb/sec: 2.08
IO rate std deviation: 0.94
Test exec time sec: 202.62

hide

测试读速度

hide

cd /opt/hadoop/share/hadoop/mapreduce/
rm -rf TestDFSIO_results.log
hadoop jar ./hadoop-mapreduce-client-jobclient-3.2.1-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 10MB

leaf

User Link

hide

查看运行结果

leaf

2023-07-21 07:17:52,772 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
2023-07-21 07:17:52,773 INFO fs.TestDFSIO: Date & time: Fri Jul 21 07:17:52 GMT 2023
2023-07-21 07:17:52,774 INFO fs.TestDFSIO: Number of files: 10
2023-07-21 07:17:52,774 INFO fs.TestDFSIO: Total MBytes processed: 100
2023-07-21 07:17:52,775 INFO fs.TestDFSIO: Throughput mb/sec: 8.33
2023-07-21 07:17:52,775 INFO fs.TestDFSIO: Average IO rate mb/sec: 9.16
2023-07-21 07:17:52,776 INFO fs.TestDFSIO: IO rate std deviation: 2.9
2023-07-21 07:17:52,779 INFO fs.TestDFSIO: Test exec time sec: 156.42

hide

leaf

JPS无法查看已经启动的Java进程信息

hide

Failed to replace a bad datanode

hide

leaf

java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try.

hide

leaf

因为，我的集群共有3个DataNode节点，而我设置的默认副本数是3个。在执行写入到HDFS的操作时，当我的一台Datanode写入失败时，它要保持副本数为3,它就会去寻找一个可用的DataNode节点来写入，可是我的流水线上就只有3 台DataNode节点，所以导致会报错Failed to replace a bad datanode

hide

查看自己已经存在的副本数

leaf

Hadoop配置文件中的hdfs-site.xml

leaf

<property>
<name>dfs.replication</name>
<value>3</value>
</property>

hide

leaf

在hdfs-site.xml 文件中添加下面几行代码

leaf

<property>
<name>dfs.client.block.write.replace-datanode-on-failure.policy</name>
<value>NEVER</value>
</property>

leaf

User Link

hide

NoClassDefFoundError:
org/sparkproject/guava/base/Preconditions

leaf

原因,hadoop的common目录下包版本冲突

leaf

处理,将common新加的包移到../bak目录下，然后一个个加回来测试

leaf

经测试，发现是由spark-streaming-kafka-0-10-assembly_2.12引起的hadoop启不来

leaf

处理：改使用spark-streaming-kafka-0-10_2.12

hide

ava.lang.ClassNotFoundException:
org.apache.spark.kafka010.KafkaConfigUpdater

leaf

参考：https://blog.csdn.net/dkl12/article/details/118637084#%E6%9C%80%E7%BB%88%E6%B7%BB%E5%8A%A0%E7%9A%84jar%E5%8C%85

hide

处理：添加

leaf

commons-pool2-2.11.1.jar

leaf

spark-token-provider-kafka-0-10_2.12-3.1.3.jar

leaf

jedis-3.0.1.jar

hide

NoClassDefFoundError:
com/google/common/util/concurrent/ThreadFactoryBuilder

hide

处理：添加

leaf

guava-27.0-jre.jar

leaf

find / -name guava*
cp /opt/hadoop-3.2.1/share/hadoop/common/lib/guava-27.0-jre.jar /usr/local/spark/jars/

hide

hide

Byte、KB、MB、GB、
TB、PB、EB、ZB、
YB、NB、DB、CB、
XB

leaf

1B (Byte字节)；
1KB（Kilobyte） = 2^10 B = 1024 B；
1MB（Megabyte） = 2^10 KB = 1024 KB = 2^20 B；
1GB（Gigabyte） = 2^10 MB = 1024 MB = 2^30 B；
1TB（Terabyte） = 2^10 GB = 1024 GB = 2^40 B；
1PB（Petabyte） = 2^10 TB = 1024 TB = 2^50 B；
1EB（Exabyte） = 2^10 PB = 1024 PB = 2^60 B；
1ZB（Zettabyte） = 2^10 EB = 1024 EB = 2^70 B；
1YB（YottaByte） = 2^10 ZB = 1024 ZB = 2^80 B；
1BB（Brontobyte） = 2^10 YB = 1024 YB = 2^90 B；
1NB（NonaByte） = 2^10 BB = 1024 BB = 2^100 B；
1DB（DoggaByte） = 2^10 NB = 1024 NB = 2^110 B；
1CB (Corydonbyte) = 2^10 DB = 1024 DB = 2^120 B；
1XB (Xerobyte) = 2^10 CB = 1024 CB = 2^130 B；

hide

退出安全模式

hide

查看hdfs safe mode

leaf

hadoop dfsadmin -safemode get

hide

退出hdfs safe mode

leaf

hadoop dfsadmin -safemode leave

hide

降低hdfs上权限

hide

给根目录加权限

leaf

hdfs dfs -chmod -R 777 /

hide

Liux时区设置

leaf

使用cp,替换当前时区为上海时区，如有提示，输入yes进行确认
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime