一、环境配置
由于集群至少需要三台服务器，我就拿上次做的MongoDB Master, Slave, Arbiter环境来做Hadoop集群。服务器还是ibmcloud 免费提供的。其中Arbiter在这里做的也是slave的角色。
配置三台机器的Master hosts文件如下:
$cat/etc/hosts127.0.0.1localhostDatabase-Masterlocalhost.localdomainlocalhost4localhost4.localdomain4::1localhostlocalhost.localdomainlocalhost6localhost6.localdomain6192.168.0.28Database-Mastermaster192.168.0.29Database-Slaveslave192.168.0.30Database-Arbiterarbiter
Master机器有安装ansible，其他所需要的软件包地址：
/hadoop/common/hadoop-2.6.4/hadoop-2.6.4.tar.gz
http://mirrors./apache/zookeeper/zookeeper-3.4.8/zookeeper-3.4.8.tar.gz
/hbase/1.2.0/hbase-1.2.0-bin.tar.gz
/otn-pub/java/jdk/8u73-b02/jdk-8u73-linux-x64.tar.gz
java我解压缩到/usr/java/目录下，然后编辑环境变量.zshrc
exportJAVA_HOME=/usr/java/jdk1.8.0_73exportPATH=$JAVA_HOME/bin:$PATHexportCLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tool.jar
然后重新加载,使变量生效, source .zshrc.
然后需要集群见无密码登录，此前做MongoDB实验的时候已经设置过，再次不在赘述。
二、Hadoop的安装和配置
1.首先将刚才下载的hadoop-2.6.4.tar.gz文件解压到/home/ibmcloud/hadoop，然后编辑etc/hadoop/core-site.xml
<configuration><property><name>fs.default.name</name><value>hdfs://master:9000</value></property></configuration>
2.添加JAVA_HOME变量到hadoop-env.sh
exportJAVA_HOME=/usr/java/jdk1.8.0_73
3.hdfs-site.xml
<configuration><property><name>dfs.name.dir</name><value>/home/ibmcloud/hadoop/name</value></property><property><name>dfs.data.dir</name><value>/home/ibmcloud/hadoop/data</value></property><property><name>dfs.replication</name><value>3</value></property></configuration>
4.将mapred-site.xml.template 改名mapred-site.xml
<configuration><property><name>mapred.job.tracker</name><value>master:9001</value></property></configuration>
5.add master and slave
echo"master">~/hadoop/etc/hadoop/masterecho-e"slave\narbiter">~/hadoop/etc/hadoop/slaves
6.copy hadoop folder to slave and arbiter
ansibleall-mcopy-a"src=hadoopdest=~'
7.启动hadoop集群
第一次执行，需要格式化namenode,以后启动不需要执行此步骤。
hadoop/bin/hadoopnamenode-format
然后启动hadoop
hadoop/sbin/start-all.sh
启动完成后，如果没有什么错误，执行jps查询一下当前进程，NameNode是Hadoop Master进程，SecondaryNameNode，ResourceManager是Hadoop进程。
$jps23076NameNode20788ResourceManager23302SecondaryNameNode27559Jps
三、ZooKeeper集群安装
1.解压缩zookeeper-3.4.8.tar.gz并重命名zookeeper, 进入zookeeper/conf目录,cpzoo_sample.cfg zoo.cfg 并编辑
$egrep-v'^$|^#'zoo.cfgtickTime=2000initLimit=10syncLimit=5dataDir=/home/ibmcloud/zookeeper/dataclientPort=2181server.1=192.168.0.28:2888:3888server.2=192.168.0.29:2888:3888server.3=192.168.0.30:2888:3888
2.新建并编辑myid文件
mkdir/home/zookeeper/dataecho"1">/home/zookeeper/data/myid
3.然后同步zookeeper到其他两个节点,然后在其他节点需要修改myid为相应的数字。
ansibleall-mcopy-a"src=zookeeperdest=~'
4.启动zookeeper，查看启动信息
-03-1506:43:00,421[myid:1]-INFO[CommitProcessor:1:ZooKeeperServer@645]-Establishedsession0x15378e1050a0005withnegotiatedtimeout40000forclient/192.168.0.28:57372-03-1506:43:01,755[myid:1]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192]-Acceptedsocketconnectionfrom/192.168.0.28:57379-03-1506:43:01,757[myid:1]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@900]-Clientattemptingtoestablishnewsessionat/192.168.0.28:57379-03-1506:43:01,760[myid:1]-INFO[CommitProcessor:1:ZooKeeperServer@645]-Establishedsession0x15378e1050a0006withnegotiatedtimeout40000forclient/192.168.0.28:57379-03-1506:43:02,211[myid:1]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192]-Acceptedsocketconnectionfrom/192.168.0.28:57383-03-1506:43:02,215[myid:1]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@900]-Clientattemptingtoestablishnewsessionat/192.168.0.28:57383-03-1506:43:02,217[myid:1]-INFO[CommitProcessor:1:ZooKeeperServer@645]-Establishedsession0x15378e1050a0007withnegotiatedtimeout40000forclient/192.168.0.28:57383-03-1506:46:57,531[myid:1]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008]-Closedsocketconnectionforclient/192.168.0.28:57379whichhadsessionid0x15378e1050a0006-03-1506:46:57,544[myid:1]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008]-Closedsocketconnectionforclient/192.168.0.28:57383whichhadsessionid0x15378e1050a0007-03-1506:46:57,555[myid:1]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008]-Closedsocketconnectionforclient/192.168.0.28:57372whichhadsessionid0x15378e1050a0005-03-1506:47:10,171[myid:1]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192]-Acceptedsocketconnectionfrom/192.168.0.30:60866-03-1506:47:10,184[myid:1]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@900]-Clientattemptingtoestablishnewsessionat/192.168.0.30:60866-03-1506:47:10,186[myid:1]-INFO[CommitProcessor:1:ZooKeeperServer@645]-Establishedsession0x15378e1050a0008withnegotiatedtimeout40000forclient/192.168.0.30:60866-03-1506:47:10,625[myid:1]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192]-Acceptedsocketconnectionfrom/192.168.0.28:58169-03-1506:47:10,626[myid:1]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@900]-Clientattemptingtoestablishnewsessionat/192.168.0.28:58169-03-1506:47:10,629[myid:1]-INFO[CommitProcessor:1:ZooKeeperServer@645]-Establishedsession0x15378e1050a0009withnegotiatedtimeout40000forclient/192.168.0.28:58169-03-1506:47:11,199[myid:1]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192]-Acceptedsocketconnectionfrom/192.168.0.30:60867-03-1506:47:11,200[myid:1]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@900]-Clientattemptingtoestablishnewsessionat/192.168.0.30:60867-03-1506:47:11,204[myid:1]-INFO[CommitProcessor:1:ZooKeeperServer@645]-Establishedsession0x15378e1050a000awithnegotiatedtimeout40000forclient/192.168.0.30:60867
来自Slave的信息：
-03-1506:43:02,667[myid:2]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192]-Acceptedsocketconnectionfrom/192.168.0.28:58604-03-1506:43:02,667[myid:2]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@900]-Clientattemptingtoestablishnewsessionat/192.168.0.28:58604-03-1506:43:02,670[myid:2]-INFO[CommitProcessor:2:ZooKeeperServer@645]-Establishedsession0x25378e0edf00006withnegotiatedtimeout40000forclient/192.168.0.28:58604-03-1506:46:55,407[myid:2]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192]-Acceptedsocketconnectionfrom/192.168.0.28:59328-03-1506:46:55,410[myid:2]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@900]-Clientattemptingtoestablishnewsessionat/192.168.0.28:59328-03-1506:46:55,415[myid:2]-INFO[CommitProcessor:2:ZooKeeperServer@645]-Establishedsession0x25378e0edf00007withnegotiatedtimeout40000forclient/192.168.0.28:59328-03-1506:46:57,242[myid:2]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008]-Closedsocketconnectionforclient/192.168.0.28:59328whichhadsessionid0x25378e0edf00007-03-1506:46:57,928[myid:2]-WARN[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357]-caughtendofstreamexceptionEndOfStreamException:Unabletoreadadditionaldatafromclientsessionid0x25378e0edf00006,likelyclienthasclosedsocketatorg.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230)atorg.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)atjava.lang.Thread.run(Thread.java:745)-03-1506:46:57,929[myid:2]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008]-Closedsocketconnectionforclient/192.168.0.28:58604whichhadsessionid0x25378e0edf00006-03-1506:47:08,780[myid:2]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192]-Acceptedsocketconnectionfrom/192.168.0.28:59377-03-1506:47:08,786[myid:2]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@900]-Clientattemptingtoestablishnewsessionat/192.168.0.28:59377-03-1506:47:08,789[myid:2]-INFO[CommitProcessor:2:ZooKeeperServer@645]-Establishedsession0x25378e0edf00008withnegotiatedtimeout40000forclient/192.168.0.28:59377-03-1506:49:57,202[myid:2]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192]-Acceptedsocketconnectionfrom/192.168.0.28:59911-03-1506:49:57,212[myid:2]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@900]-Clientattemptingtoestablishnewsessionat/192.168.0.28:59911-03-1506:49:57,215[myid:2]-INFO[CommitProcessor:2:ZooKeeperServer@645]-Establishedsession0x25378e0edf00009withnegotiatedtimeout40000forclient/192.168.0.28:59911-03-1506:52:15,489[myid:2]-WARN[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357]-caughtendofstreamexceptionEndOfStreamException:Unabletoreadadditionaldatafromclientsessionid0x25378e0edf00009,likelyclienthasclosedsocketatorg.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230)atorg.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)atjava.lang.Thread.run(Thread.java:745)-03-1506:52:15,490[myid:2]-INFO[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008]-Closedsocketconnectionforclient/192.168.0.28:59911whichhadsessionid0x25378e0edf00009
5.再次查看jps, 此时会看到zookeeper进程QuorumPeerMain
$jps23076NameNode20788ResourceManager30821Jps23302SecondaryNameNode30538QuorumPeerMain
四、HBase集群的安装和配置
1. 解压缩hbase-1.2.0-bin.tar.gz并重命名为hbase, 编辑/hbase/conf/hbase-env.sh
$egrep-v'^$|^#'hbase-env.shexportJAVA_HOME=/usr/java/jdk1.8.0_73exportHBASE_CLASSPATH=/home/ibmcloud/hadoop/etc/hadoopexportHBASE_OPTS="-XX:+UseConcMarkSweepGC"exportHBASE_MASTER_OPTS="$HBASE_MASTER_OPTS-XX:PermSize=128m-XX:MaxPermSize=128m"exportHBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS-XX:PermSize=128m-XX:MaxPermSize=128m"exportHBASE_MANAGES_ZK=false
2.编辑hbase-site.xml
<configuration><property><name>hbase.rootdir</name><value>hdfs://master:9000/hbase</value></property><property><name>hbase.master</name><value>master</value></property><property><name>hbase.cluster.distributed</name><value>true</value></property><property><name>hbase.zookeeper.property.clientPort</name><value>2181</value></property><property><name>hbase.zookeeper.quorum</name><value>master,slave,arbiter</value></property><property><name>zookeeper.session.timeout</name><value>60000000</value></property><property><name>dfs.support.append</name><value>true</value></property></configuration>
3.添加Slave, Arbiter 到regionservers
4.分发hbase到其他两个节点
ansibleall-mcopy-a"src=hbasedest=~"
五、启动集群
1. 启动zookeeper
zookeeper/bin/zkServer.shstart
2.启动Hadoop
$hadoop/sbin/start-all.shThisscriptisDeprecated.Insteadusestart-dfs.shandstart-yarn.sh16/03/1507:33:09WARNutil.NativeCodeLoader:Unabletoloadnative-hadooplibraryforyourplatform...usingbuiltin-javaclasseswhereapplicableStartingnamenodeson[master]master:namenoderunningasprocess23076.Stopitfirst.arbiter:datanoderunningasprocess2111.Stopitfirst.slave:datanoderunningasprocess1×××.Stopitfirst.Startingsecondarynamenodes[0.0.0.0]0.0.0.0:secondarynamenoderunningasprocess23302.Stopitfirst.16/03/1507:33:16WARNutil.NativeCodeLoader:Unabletoloadnative-hadooplibraryforyourplatform...usingbuiltin-javaclasseswhereapplicablestartingyarndaemonsresourcemanagerrunningasprocess20788.Stopitfirst.arbiter:startingnodemanager,loggingto/home/ibmcloud/hadoop/logs/yarn-ibmcloud-nodemanager-Database-Arbiter.outslave:startingnodemanager,loggingto/home/ibmcloud/hadoop/logs/yarn-ibmcloud-nodemanager-Database-Slave.out
3.启动hbase
$hbase/bin/start-hbase.shmasterrunningasprocess10144.Stopitfirst.arbiter:regionserverrunningasprocess3515.Stopitfirst.slave:startingregionserver,loggingto/home/ibmcloud/hbase/bin/../logs/hbase-ibmcloud-regionserver-Database-Slave.outslave:JavaHotSpot(TM)64-BitServerVMwarning:ignoringoptionPermSize=128m;supportwasremovedin8.0slave:JavaHotSpot(TM)64-BitServerVMwarning:ignoringoptionMaxPermSize=128m;supportwasremovedin8.0
查询各个节点的集群进程情况
Master：
#ibmcloudatDatabase-Masterin~[7:33:44]$jps10144HMaster23076NameNode20788ResourceManager20773Jps23302SecondaryNameNode30538QuorumPeerMain
Slave:
#ibmcloudatDatabase-Slavein~/hbase/bin[6:47:55]$jps1×××DataNode26794Jps16397QuorumPeerMain26526HRegionServer
Arbiter:
#ibmcloudatDatabase-Arbiterin~/hbase/bin[6:46:34]$jpsQuorumPeerMain3515HRegionServer3628Jps2111DataNode
进程都已经开启，进入habse shell环境,
#ibmcloudatDatabase-Masterin~[7:34:03]$hbase/bin/hbaseshell-03-1507:35:04,687WARN[main]util.NativeCodeLoader:Unabletoloadnative-hadooplibraryforyourplatform...usingbuiltin-javaclasseswhereapplicableHBaseShell;enter'help<RETURN>'forlistofsupportedcommands.Type"exit<RETURN>"toleavetheHBaseShellVersion1.2.0,r25b281972df2f5b15c426c8963cbf77dd853a5ad,ThuFeb1823:01:49CSThbase(main):001:0>hbase(main):002:0*statusERROR:org.apache.hadoop.hbase.PleaseHoldException:Masterisinitializing
提示master还是initialzing, 我的虚拟机1.5G内存，单核，10G硬盘，跑着MongoDB, PHP, Nginx, 加上Hadoop集群，肯定消化不良了。如图：
查看链接发现Hadoop监听网卡是内网的，加上端口转发，打开公网地址查看一下Hadoop运行状态
sudoiptables-tnat-IPREROUTING-d129.41.153.232-ptcp--dport50070-jDNAT--to192.168.0.28:50070
然后打开浏览器输入http://129.41.153.232:50070/dfshealth.html#tab-overview，如图,hadoop状态
YARN状态：
Hbase状态：
其中遇到的问题又hbase启动不起来，一直报permission denied,后来发现Slave, Arbiter bin目录下的脚本没有给执行权限，然后logs下日志文件的权限不对。
参考文章:
http://songlee24.github.io//07/20/hadoop-hbase-zookeeper-distributed-mode/
/questions/21166542/hbase-does-not-run-after-start-hbase-sh-permission-denied
