在Centos7上进行Hive 3.1.2集群安装

注意：本文安装的Hive3.1.2不是stable版本

1. 依赖组件安装 组件参考文章Java8centos7同时安装java8和openJdk11、windows同时安装java8和openJdk11mysqlcentos7安装mysql8.0.25版本hadoopCentos7上Hadoop 3.3.1的高可用HA安装过程 2. 下载并解压

[root@hive1 ~]#
[root@hive1 ~]# wget --no-check-certificate https://dlcdn.apache.org/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz
[root@hive1 ~]#
[root@hive1 ~]# tar -zxvf apache-hive-3.1.2-bin.tar.gz
[root@hive1 ~]# cd apache-hive-3.1.2-bin

3. 配置conf 3.1 配置Hive运行环境: hive-env.sh

复制hive-env.sh

[root@hive1 apache-hive-3.1.2-bin]# cp conf/hive-env.sh.template conf/hive-env.sh

修改内容如下：

# 指定Hadoop目录
HADOOP_HOME=/root/hadoop-3.3.1
# 指定Hive配置文件目录
export HIVE_CONF_DIR=/root/apache-hive-3.1.2-bin/conf
# 指定Hive依赖包目录
export HIVE_AUX_JARS_PATH=/root/apache-hive-3.1.2-bin/lib

添加内容如下：

export JAVA_HOME=/root/jdk1.8.0_201

3.2 配置Hive相关参数: hive-site.xml

[root@hive1 apache-hive-3.1.2-bin]# touch conf/hive-site.xml

文件内容如下：

    
        hive.metastore.warehouse.dir
        hdfs://nnha/user/hive/warehouse
        配置Hive数据储存在HDFS上的目录
    

    
        hive.exec.scratchdir
        hdfs://nnha/tmp/hive
        配置Hive在HDFS上的临时目录
    

    
        hive.metastore.local
        false
        指定Hive是否开启本地模式, 此处表示远程模式
    

    
        hive.metastore.uris
        thrift://hive1:9083
        指定Metastore服务地址
    

    
        hive.metastore.event.db.notification.api.auth
        false
        指定Hive的metastore是否开启database权限认证
    

    
        hive.server2.active.passive.ha.enable
        true
        指定Hive是否开启ActivePassiveHA高可用
    

    
        javax.jdo.option.ConnectionURL
        jdbc:mysql://hive3:3306/hive?createDatabaseIfNotExist=true&useSSL=false&allowPublicKeyRetrieval=true
        配置JDBC连接地址, allowPublicKeyRetrieval指定是否允许客户端从服务器自动请求公钥
    


    
        javax.jdo.option.ConnectionDriverName
        com.mysql.cj.jdbc.Driver
        配置JDBC驱动
    

    
        javax.jdo.option.ConnectionUserName
        root
        配置连接MySQL的用户名
    

    
        javax.jdo.option.ConnectionPassword
        Root_123
        配置连接MySQL的密码
    

    
        hive.cli.print.header
        true
        配置在命令行界面HiveCLI中显示表的列名
    

    
        hive.cli.print.current.db
        true
        配置在命令行界面HiveCLI中显示当前数据库名称
    

4. 下载mysql-connector-java-8.0.25.jar到lib目录

[root@hive1 apache-hive-3.1.2-bin]# 
[root@hive1 apache-hive-3.1.2-bin]# wget -P /root/apache-hive-3.1.2-bin/lib https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.25/mysql-connector-java-8.0.25.jar

5. 配置环境变量

向/root/.bashrc添加如下内容

export HIVE_HOME=/root/apache-hive-3.1.2-bin
export PATH=$PATH:$HIVE_HOME/bin

执行source /root/.bashrc使环境变量生效

6. 初始化Mysql元数据

初始化Mysql元数据

[root@hive1 ~]# 
[root@hive1 ~]# schematool -initSchema -dbType mysql
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/apache-hive-3.1.2-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:	 jdbc:mysql://hive3:3306/hive?createDatabaseIfNotExist=true&useSSL=false
Metastore Connection Driver :	 com.mysql.cj.jdbc.Driver
Metastore connection User:	 root
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.mysql.sql
......省略部分......
Initialization script completed
schemaTool completed
[root@hive1 ~]# 

7. 解决数据库、表、分区、索引的注释中文乱码

登录Hive的Mysql元数据库，将元数据表字段的latin1字符类型，改成utf8类型

修改数据库注释

alter table DBS modify column `DESC` varchar(4000) character set utf8 default null;

修改表注释和表字段注释

alter table TABLE_PARAMS modify column PARAM_VALUE mediumtext character set utf8;
alter table COLUMNS_V2 modify column COMMENT varchar(256) character set utf8 default null;

修改分区字段注释

alter table PARTITION_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8 default null;
alter table PARTITION_KEYS modify column PKEY_COMMENT varchar(4000) character set utf8 default null;

修改索引注释

alter table INDEX_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8 default null;

注意：对于已经创建的Hive表，还是会出现中文注释乱码。

8. 启动metastore和HiveServer2

HiveServer2的作用：允许远程不同编程语言的客户端，向Hive提交查询请求。其核心是基于Thrift，由Thrift负责Hive的查询服务

[root@hive1 ~]# 
[root@hive1 ~]# mkdir apache-hive-3.1.2-bin/logs
[root@hive1 ~]# 
[root@hive1 ~]# nohup hive --service metastore >> /root/apache-hive-3.1.2-bin/logs/metastore.log 2>&1 &
[1] 13700
[root@hive1 ~]# nohup hive --service hiveserver2 >> /root/apache-hive-3.1.2-bin/logs/hiveserver2.log 2>&1 &
[2] 13817
[root@hive1 ~]# 

如果出现问题，可以查看/tmp/root/hive.log日志

9. beeline连接Hive进行测试

用户名和密码为Centos服务器的用户名和密码

[root@hive1 ~]# beeline -u jdbc:hive2://hive1:10000 -n root -p root123
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/apache-hive-3.1.2-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/apache-hive-3.1.2-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://hive1:10000
Connected to: Apache Hive (version 3.1.2)
Driver: Hive JDBC (version 3.1.2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 3.1.2 by Apache Hive
0: jdbc:hive2://hive1:10000>
0: jdbc:hive2://hive1:10000> show databases;
INFO  : Compiling command(queryId=root_20220208144518_4eeac460-1ae4-469d-9ce2-81bce6071fc9): show databases
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null)
INFO  : Completed compiling command(queryId=root_20220208144518_4eeac460-1ae4-469d-9ce2-81bce6071fc9); Time taken: 1.455 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=root_20220208144518_4eeac460-1ae4-469d-9ce2-81bce6071fc9): show databases
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=root_20220208144518_4eeac460-1ae4-469d-9ce2-81bce6071fc9); Time taken: 0.112 seconds
INFO  : OK
INFO  : Concurrency mode is disabled, not creating a lock manager
+----------------+
| database_name  |
+----------------+
| default        |
+----------------+
1 row selected (2.414 seconds)
0: jdbc:hive2://hive1:10000> 
0: jdbc:hive2://hive1:10000> !exit
Closing: 0: jdbc:hive2://hive1:10000
[root@hive1 ~]#