南山草舍

0%

Ambari集成HUE

发表于 更新于 分类于

Hue介绍

Hue 是运营和开发Hadoop应用的图形化用户界面。Hue程序被整合到一个类似桌面的环境，以web程序的形式发布，对于单独的用户来说不需要额外的安装。

依赖环境配置

配置Maven3.5.4

1
2
3
4
5
6
7
cd /usr/local/src/
wget http://apache.mirrors.lucidnetworks.net/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz 
tar xf apache-maven-3.5.4-bin.tar.gz -C /usr/local/
vim  /etc/profile
export MAVEN_HOME=/usr/local/apache-maven-3.5.4
export PATH=$PATH:$MAVEN_HOME/bin
source /etc/profile

python 版本2.6.6+

1
2
python -V
Python 2.7.5

其他依赖工具和包

1
yum -y install ant asciidoc krb5-devel cyrus-sasl-gssapi cyrus-sasl-devel libxml2-devel libxslt-devel libtidy mysql mysql-devel openldap-devel python-devel python-simplejson python-setuptools sqlite-devel gcc gcc-c++ rsync saslwrapper-devel pycrypto gmp-devel libyaml-devel cyrus-sasl-plain cyrus-sasl-devel cyrus-sasl-gssapi libssl-devel libffi-devel libsasl2-dev libsasl2-modules-gssapi-mit libkrb5-dev

python模块依赖安装

1
2
3
4
pip install --upgrade pip
pip install ipdb
pip install psycopg2
pip install psycopg2-binary

编译安装

1
2
3
4
5
6
7
8
9
git clone https://github.com/cloudera/hue.git
make apps（这一步可能遇到很多问题，比如某个包无法下载等，多执行几次就可以通过了）
make locales（编译多语言版本）
#注：若只编译简体中文版，需修改$HUE/desktop/core/src/desktop/settings.py文件中：LANGUAGE_CODE = 'zh_CN'，并删除LANGUAGES中的其它语言项。
make install  #（make install PREFIX=/usr/hdp/2.3.4.0-3485/hue HADOOP_HOME=/usr/hdp/current/hadoop-client）
$HUE/build/env/bin/hue test all  #测试
ln -s /usr/local/hue/desktop/libs/hadoop/java-lib/hue-plugins-3.9.0-SNAPSHOT.jar /opt/hadoop/hadoop-2.6.0/lib
useradd hue
chown -R hue:hue /usr/local/hue #必须新建hue用户，且hue存在home目录，并赋hue目录权限，否则启动报错

修改/usr/local/hue/desktop/conf/pseudo-distributed.ini配置

1
2
3
4
5
6
[desktop]
secret_key=qwertyuiopasdfghjklzxcvbnm1234567890   #key随便写一串就行
http_host=bigdata05.islocal.cc
http_port=8888
time_zone=Asia/Shanghai
app_blacklist = search,indexer,security   #黑名单：如果不安装oozie的话，jobsub和oozie需同时加入黑名单才能禁用oozie,因为jobsub依赖oozie，否则启动会报错。

启动hue

$HUE/build/env/bin/supervisor & #在生产环境中使用这个命令
$HUE/build/env/bin/hue runcpserver & #用于development模式
打开web页面：http://bigdata05.islocal.cc:8888 #第一次输入的用户名密码为超级管理员的用户密码。
启动访问没有问题，即可将/usr/local/hue打成tar包，使用ambari进行集成安装

Ambari 集成安装HUE

制作hue tar包

1
2
3
4
5
6
7
8
9
10
11
12
13
tar cvzf hue-3.11.0.tgz 、/usr/local/hue/
cp hue-3.11.0.tgz /var/www/html/hdp/HDP/centos7/2.6.5.0-292/hue/hue-3.11.0.tgz

cat /etc/yum.repos.d/HDP.repo
[HDP-2.6-repo-1]
name=HDP-2.6-repo-1
baseurl=http://192.168.6.157/hdp/HDP/centos7/2.6.5.0-292/
path=/
enabled=1
gpgcheck=0

vim /usr/lib/ambari-agent/lib/resource_management/core/sudo.py +142
fp.write(content.encode("utf-8"))

版本说明

Github中提供了两个版本的Ambari-Hue：v1.0.0和v2.0.0
release-1.0.0

  • Ambari: 2.1.0~2.2.2
  • Hue: 3.9.0

release-2.0.0

  • Ambari: 2.4.0+
  • Hue: 3.10.0+

下载ambari-hue到Ambari组件服务目录

1
2
3
4
5
6
7
VERSION=`hdp-select status hadoop-client | sed 's/hadoop-client - \([0-9]\.[0-9]\).*/\1/'`
rm -rf /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/HUE  
# 我们这里使用hue 3.11版本，手动下载v2.0.0
wget https://github.com/EsharEditor/ambari-hue-service/archive/v2.0.0.tar.gz
mkdir -p /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/HUE
tar xf v2.0.0.tar.gz -C /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/HUE
rm -fr /usr/hdp/current/hadoop-client/lib/hue-plugins-3.11.0-SNAPSHOT.jar

重启ambari-server

1
ambari-server restart

通过Ambari安装Hue

On bottom left -> Actions -> Add service -> check Hue server -> Next -> Next -> Change any config you like (e.g. install dir, port) -> Next -> Deploy
数据库使用默认的SqlLite即可，改成postgresql未安装成功！

Q&A

Q1：编译报错信息：error: ffi.h: No such file or directory (HUE 3.10.0编译时报此错误)

A1：解决办法：yum -y install libssl-devel libffi-devel

Q2：启动运行报错：UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xef in position 166: ordinal not in range(128)

A2:
问题分析：编码问题,Python的str默认是ascii编码，和unicode编码冲突
解决办法：在出现问题的python文件或者页面添加如下3行信息：

1
2
3
import sys
reload(sys)
sys.setdefaultencoding('utf-8')

Q3: Hue提交Oozie任务，任务一直处于运行状态，Log日志报错信息如下：org.apache.oozie.action.ActionExecutorException: JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses

A3: 解决办法：在HDFS是ha模式下，需要配置hue-hadoop中配置信息为：logical_name= hdfs://nncluster

Q4: NoteBook中提交spark shell任务：

报错：The Spark session could not be created in the cluster: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:558) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at com.sun.proxy.$Proxy28.mkdirs(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3018) … 28 more
A4: 解决办法：在hadoop的core-site.xml文件中添加spark用户的授权信息：

1
2
3
4
5
6
7
8
<property>
  <name>hadoop.proxyuser.spark.groups</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.spark.hosts</name>
  <value>*</value>
</property>

Q5: Hue中确认配置好hive以及在hive中添加了相关配置后，发现依旧无法连接Hiveserver2,无Kerberos认证

Hue中的报错信息： Could not start SASL: Error in sasl_client_start (-4) SASL(-4): no mechanism available: No worthy mechs foundHiveserver2日志中的报错信息： ERROR [HiveServer2-Handler-Pool: Thread-31]: server.TThreadPoolServer (TThreadPoolServer.java:run(296)) - Error occurred during processing of message. java.lang.RuntimeException: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream
A5: 问题分析：Hue连接Hiveserver2使用的是Python，在Centos操作系统上丢失了一些依赖：cyrus-sasl-plain cyrus-sasl-devel cyrus-sasl-gssapi
解决办法：yum -y install cyrus-sasl-plain cyrus-sasl-devel cyrus-sasl-gssapi
设置以下配置像为NOSASL或None

1
2
3
4
5
6
7
<property>
  <name>hive.server2.authentication</name>
  <value>NOSASL</value>
  <!--或者设置为
    <value>None</value>
  -->
</property>

Q6 报错信息：django.core.exceptions.ImproperlyConfigured: psycopg2_version 2.5.4 or newer is required; you have 2.5.1 (dt dec pq3 ext)

A6: 解决方法

1
2
3
pip install --upgrade pip
pip upgrade psycopg2
pip install psycopg2-binary

Q7报错信息：RuntimeError: Model class beeswax.models.QueryHistory doesn’t declare an explicit app_label and isn’t in an application in INSTALLED_APPS.

A7: 解决方法，修改配置文件/usr/local/hue/desktop/core/src/desktop/settings.py

1
2
3
4
INSTALLED_APPS = [
    'beeswax',
    'oozie',
]

Q8 web也没访问报错信息：Invalid HTTP_HOST header:’bigdata05.islocal.cc:8888’.You may need to add u’bigdata05.islocal.cc’ to ALLOWED_HOSTS.

A8： 解决方法，修改配置文件/usr/local/hue/desktop/core/src/desktop/settings.py

1
ALLOWED_HOSTS = ['*']

参考文档

https://eshareditor.github.io/2016/09/14/Hue-编译安装/
https://eshareditor.github.io/2016/10/11/Ambari-Hue-Service/
SDK文档：http://cloudera.github.io/hue/docs-3.7.0/sdk/sdk.html#introduction-and-overview
Hue配置信息说明(中文):2018/7/26 http://www.cloudera.com/content/www/zh-CN/documentation/enterprise/5-3-x/topics/cdh_ig_hue_config.html
(英文)http://archive.cloudera.com/cdh5/cdh/5/hue/user-guide/index.html