Hadoop 生态系统

Imagemap
Hadoop 生态系统Apache HadoopHadoop独立模式Docker版本安装hadoopdocker run -i -t -p 50070:50070 -p 9000: ...访问hadoopNodeManagerhttp://192.168.31.11:8042/nodeYarn管理应用http://192.168.31.11:8088/clusterNameNode管理站点http://192.168.31.11:50070/DataNodehttp://192.168.31.11:50075/secondarynamenodehttp://192.168.31.11:50090/进入容器docker ps查看到该容器信息容器ID: c825d4428a07使用attach进入容器docker attach c825d4428a07或使用exec进入容器docker exec -it c825d4428a07 /bin/bash 退出容器ctrl+c 退出容器并关闭容器ctrl+p+q 退出不关闭容器运行容器进入容器后使用exit退出就关闭了容器 docker ps -all 查看容器还在 docker start -ia c825d4428a07测试应用docker exec -it c825d4428a07 /bin/bash cd /usr/local/hadoop-2.6.0/share/hadoop ...../../../bin/hadoop jar ./hadoop-mapredu ...Hadoop集群模式Docker版本安装Hadoopdocker run -it -p 8088:8088 -p 9820:9820 ...docker run -it -p 18088:8088 -p 19820:98 ...docker run -it -p 28088:8088 -p 29820:98 ...ctrl+p+q 退出不关闭容器配置集群需要配置内容共需要配置/opt/hadoop/hadoop-3.1.0/etc/hadoop ...修改hostnamemaster机hostnamectl set-hostname masterslave机hostnamectl set-hostname slave1hostnamectl set-hostname slave2配置hostsVi /etc/hosts172.17.0.2 master 172.17.0.3 s ...复制到master和slavescp /etc/hosts master:/etc/hostsscp /etc/hosts slave1:/etc/hostsscp /etc/hosts slave2:/etc/hosts配置免密登陆在master上生成keySsh-keygen复制key到从机ssh-copy-id root@slave1ssh-copy-id root@slave2配置系统环境变量vim ~/.bashrcJAVA_HOME=/usr/lib/jvm/java-1.8.0-openjd ...使配置生效source ~/.bashrc进行验证hadoop version如果显示版本则说明配置正确复制到slavescp ~/.bashrc slave1:~/.bashrcscp ~/.bashrc slave2:~/.bashrc配置workersvim $HADOOP_HOME/etc/hadoop/workers删除原内容 添加 slave1 slave2创建dfs目录配置hadoop-envVi etc/hadoop/hadoop-env.sh添加 export JAVA_HOME=/usr/lib/jvm/java-1. ...配置yarn-env.shvi etc/hadoop/yarn-env.sh添加 export JAVA_HOME=/usr/lib/jvm/java-1. ...配置Core-siteVi etc/hadoop/core-site.xml配置hdfs-siteVi etc/hadoop/hdfs-site.xml配置mapred-sitevi etc/hadoop/mapred-site.xml配置yarn-sitevi etc/hadoop/yarm-site.xml<configuration> <property> < ...yarn.resourcemanager.hostname 指定 yarn 的 ...yarn.nodemanager.aux-services reducer 获取 ...yarn.nodemanager.vmem-check-enabled 忽略虚拟 ...复制到其它节点scp -rp /usr/local/hadoop slave1:/usr/lo ...scp -rp /usr/local/hadoop slave2:/usr/lo ...格式化在 master 上进行即可hadoop namenode -format注意，如果需要重新格式化 NameNode需要先将原来 NameNode 和 DataNode 下的文件全部删除rm -rf $HADOOP_HOME/dfs/data/*rm -rf $HADOOP_HOME/dfs/name/*启停服务start-all.shstop-all.sh验证服务master 上执行 jpsslave上执行jps测试应用hadoop jar hadoop-mapreduce-examples-3.2 ...hadoop jar hadoop-mapreduce-examples-3.2 ...Hadoop集群模式虚拟机版本Apache Flume™官网https://flume.apache.org/下载wget https://downloads.apache.org/flume/ ...参考https://blog.csdn.net/weixin_38231448/ar ...安装安装JDK 1.8+ 配置环境变量安装Flume[root@CentOSA ~]# tar -zxf apache-flume ...配置Agent配置Apache Ambari官网https://ambari.apache.org/
hide
Hadoop 生态系统
hide
Apache Hadoop
hide
Hadoop独立模式Docker版本
hide
安装hadoop
leaf
docker run -i -t -p 50070:50070 -p 9000:9000 -p 8088:8088 -p 8040:8040 -p 8042:8042 -p 49707:49707 -p 50010:50010 -p 50075:50075 -p 50090:50090 sequenceiq/hadoop-docker:2.6.0 /etc/bootstrap.sh -bash
hide
访问hadoop
hide
NodeManager
leaf
http://192.168.31.11:8042/node
hide
Yarn管理应用
leaf
http://192.168.31.11:8088/cluster
hide
NameNode管理站点
leaf
http://192.168.31.11:50070/
hide
DataNode
leaf
http://192.168.31.11:50075/
hide
secondarynamenode
leaf
http://192.168.31.11:50090/
hide
进入容器
hide
docker ps查看到该容器信息
leaf
容器ID: c825d4428a07
hide
使用attach进入容器
leaf
docker attach c825d4428a07
hide
或使用exec进入容器
leaf
docker exec -it c825d4428a07 /bin/bash
hide
退出容器
leaf
ctrl+c 退出容器并关闭容器
leaf
ctrl+p+q 退出不关闭容器
hide
运行容器
leaf
进入容器后使用exit退出就关闭了容器
leaf
docker ps -all 查看容器还在
leaf
docker start -ia c825d4428a07
hide
测试应用
leaf
docker exec -it c825d4428a07 /bin/bash
leaf
cd /usr/local/hadoop-2.6.0/share/hadoop/mapreduce
leaf
../../../bin/hadoop jar ./hadoop-mapreduce-examples-2.6.0.jar pi 10 10
hide
Hadoop集群模式Docker版本
hide
安装Hadoop
leaf
docker run -it -p 8088:8088 -p 9820:9820 -p 9864-9870:9864-9870 --ip 172.17.0.10 --name hadoop_master -v D:\docker\public:/home/public --privileged=true goodyym/hadoop_alone:v100 /usr/sbin/init
leaf
docker run -it -p 18088:8088 -p 19820:9820 -p 19864-19870:9864-9870 --ip 172.17.0.11 --name hadoop_slave1 -v D:\docker\public:/home/public --privileged=true goodyym/hadoop_alone:v100 /usr/sbin/init
leaf
docker run -it -p 28088:8088 -p 29820:9820 -p 29864-29870:9864-9870 --ip 172.17.0.12 --name hadoop_slave2 -v D:\docker\public:/home/public --privileged=true goodyym/hadoop_alone:v100 /usr/sbin/init
leaf
ctrl+p+q 退出不关闭容器
hide
配置集群
hide
需要配置内容
leaf
共需要配置/opt/hadoop/hadoop-3.1.0/etc/hadoop/下的六个个文件，分别是 hadoop-env.sh、core-site.xml、hdfs-site.xml、yarn-site.xml、mapred-site.xml、workers
hide
修改hostname
hide
master机
leaf
hostnamectl set-hostname master
hide
slave机
leaf
hostnamectl set-hostname slave1
leaf
hostnamectl set-hostname slave2
hide
配置hosts
Arrow Link
hide
Vi /etc/hosts
leaf
172.17.0.2 master
172.17.0.3 slave1
172.17.0.4 slave2
hide
复制到master和slave
leaf
scp /etc/hosts master:/etc/hosts
leaf
scp /etc/hosts slave1:/etc/hosts
leaf
scp /etc/hosts slave2:/etc/hosts
hide
配置免密登陆
hide
在master上生成key
leaf
Ssh-keygen
hide
复制key到从机
leaf
ssh-copy-id root@slave1 User Link
leaf
ssh-copy-id root@slave2 User Link
hide
配置系统环境变量
hide
vim ~/.bashrc
leaf
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/jre
(ext: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre)
export JAVA_HOME
export PATH=$JAVA_HOME/bin:$PATH
export HADOOP_HOME=/usr/local/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
hide
使配置生效
leaf
source ~/.bashrc
hide
进行验证
hide
hadoop version
leaf
如果显示版本则说明配置正确
hide
复制到slave
leaf
scp ~/.bashrc slave1:~/.bashrc
leaf
scp ~/.bashrc slave2:~/.bashrc
hide
配置workers
hide
vim $HADOOP_HOME/etc/hadoop/workers
leaf
删除原内容
添加
slave1
slave2
hide
创建dfs目录
leaf
cd /usr/local/hadoop
mkdir dfs
cd dfs
mkdir name data tmp
/usr/hadoop/dfs/name - 存储namenode文件
/usr/hadoop/dfs/data - 存储数据
/usr/hadoop/dfs/tmp - 存储临时文件
cd /usr/local/hadoop
mkdir tmp
mkdir name data
hide
配置hadoop-env
hide
Vi etc/hadoop/hadoop-env.sh
leaf
添加
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/jre
(or: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre)
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
hide
配置yarn-env.sh
hide
vi etc/hadoop/yarn-env.sh
leaf
添加
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/jre
(or: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre)

export YARN_NODEMANAGER_USER=root
export YARN_RESOURCEMANAGER_USER=root
hide
配置Core-site
hide
Vi etc/hadoop/core-site.xml
hide
<configuration></configuration>中加入
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9820</value> --3.0使用9820
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
</property>
leaf
tmp 文件夹可能需要自行创建
leaf
fs.defaultFS 默认端口为 8020(2.x),9820(3.x)
leaf
fs.defaultFS 指定HDFS运行时nameNode地址
leaf
fs.defaultFS 在 2.X 已经替代 fs.default.name
hide
配置hdfs-site
hide
Vi etc/hadoop/hdfs-site.xml
leaf
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:9868</value>
</property>
<property>
<name>dfs.datanode.address</name>
<value>master:9866</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/hadoop/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
hide
配置mapred-site
hide
vi etc/hadoop/mapred-site.xml
leaf
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>

<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value>
</property>

</configuration>
hide
配置yarn-site
hide
vi etc/hadoop/yarm-site.xml
hide
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
</configuration>
leaf
yarn.resourcemanager.hostname
指定 yarn 的 resourcemanager 的地址
leaf
yarn.nodemanager.aux-services
reducer 获取数据的方式
leaf
yarn.nodemanager.vmem-check-enabled
忽略虚拟内存的检查
如果安装在虚拟机上，这个配置很有用，配上去之后后续操作不容易出问题。
如果是在实体机上，并且内存够多，可以将这个配置去掉
hide
复制到其它节点
leaf
scp -rp /usr/local/hadoop slave1:/usr/local/
leaf
scp -rp /usr/local/hadoop slave2:/usr/local/
hide
格式化
leaf
在 master 上进行即可
leaf
hadoop namenode -format
hide
注意，如果需要重新格式化 NameNode
hide
需要先将原来 NameNode 和 DataNode 下的文件全部删除
leaf
rm -rf $HADOOP_HOME/dfs/data/*
leaf
rm -rf $HADOOP_HOME/dfs/name/*
hide
启停服务
leaf
start-all.sh
leaf
stop-all.sh
hide
验证服务
leaf
master 上执行 jps
leaf
slave上执行jps
hide
测试应用
leaf
hadoop jar hadoop-mapreduce-examples-3.2.2.jar pi 10 10
leaf
hadoop jar hadoop-mapreduce-examples-3.2.2.jar wordcount input output
leaf
Hadoop集群模式虚拟机版本
hide
Apache Flume™
hide
官网
leaf
https://flume.apache.org/ User Link
hide
下载
leaf
wget https://downloads.apache.org/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz
hide
参考
leaf
https://blog.csdn.net/weixin_38231448/article/details/89973878
hide
安装
leaf
安装JDK 1.8+ 配置环境变量
hide
安装Flume
leaf
[root@CentOSA ~]# tar -zxf apache-flume-1.9.0-bin.tar.gz -C /usr/
[root@CentOSA ~]# cd /usr/apache-flume-1.9.0-bin/
[root@CentOSA apache-flume-1.9.0-bin]# ./bin/flume-ng version
Flume 1.9.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: d4fcab4f501d41597bc616921329a4339f73585e
Compiled by fszabo on Mon Dec 17 20:45:25 CET 2018
From source with checksum 35db629a3bda49d23e9b3690c80737f9
hide
配置
leaf
Agent配置
hide
Apache Ambari
hide
官网
leaf
https://ambari.apache.org/ User Link