- GitHub – cas-packone/ambari-impala-service：使HDP上的Impala启用
- (261条消息) HDP上安装impala_weixin_33834137的博客-CSDN博客l’l
- (261条消息) Impala配置与错误解决方案_Dino系我的博客-CSDN博客
- 1.下载 CDH6.3.0 centos7的安装包 ，上传 到 服务器 的 /var/www/html文件夹
- 2.配置 本地yum源
- 3.repolist yum clean all yum makecache
- 4.修改ambari-impala-server安装包里的文件
- 修改 impala-catalog.py文件里新增yum源 的 步骤，因为 已经下载了本地 CDH5.14.0 centos7的 文件且 已设置好 本地 Yum源 ，可以实现 本地离线安装
- 5.将修改好的ambari-impala集成压缩包上传到服务器 /var/lib/ambari-server/resources/stacks/HDP/3.1/services
- 6.修改ambari配置参数 core-site.xml
- <property>
- <name>dfs.client.read.shortcircuit</name>
- <value>true</value>
- </property>
- <property>
- <name>dfs.client.read.shortcircuit.skip.checksum</name>
- <value>false</value>
- </property>
- <property>
- <name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
- <value>true</value>
- </property>
- hdfs-site.xml
- <property>
- <name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
- <value>true</value>
- </property>
- <property>
- <name>dfs.block.local-path-access.user</name>
- <value>impala</value>
- </property>
- <property>
- <name>dfs.client.file-block-storage-locations.timeout.millis</name>
- <value>60000</value>
- </property>
- 重启ambari-server ,新增impala服务
- 新增 impala服务报错
- 报错一
- ambari新增hbase遇到Cannot create /var/run/ambari-server/stack-recommendations
- 解决方案
- 方法一：修改目录的主用户
- chown -R ambari /var/run/ambari-server
- 报错二
- Error: Error: Unable to run the custom hook script [‘/usr/bin/python’, ‘/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py’, ‘ANY’, ‘/var/lib/ambari-agent/data/command-2155.json’, ‘/var/lib/ambari-agent/cache/stack-hooks/before-ANY’, ‘/var/lib/ambari-agent/data/structured-out-2155.json’, ‘INFO’, ‘/var/lib/ambari-agent/tmp’, ‘PROTOCOL_TLSv1_2’, ”]
- 解决方案
- #执行成功的语句
- python /var/lib/ambari-server/resources/scripts/configs.py -u admin -p admin -n YChdp_test -l data-01-test -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true （测试环境）
- python /var/lib/ambari-server/resources/scripts/configs.py -u admin -p admin -n Ychdp_pro -l data-01-pro -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true （生产环境）
- python /var/lib/ambari-server/resources/scripts/configs.py -u admin -p admin -n Ychdp_dev -l data-01-dev -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true （预发布环境）
- 再次新增impala服务成功
- 服务 启动后一会儿就挂掉了，需要修改jar包重定向
- 报错一
- 新增 impala服务报错
- 7.修改jar包重定向软链接
- 如下包需要scp给hadoop002 和 hadoop003
- scp hbase-client-1.2.6.jar data-02-test:/usr/lib/impala/lib/
- scp hbase-common-1.3.0.jar data-02-test:/usr/lib/impala/lib/
- scp hive-metastore-1.2.0.jar data-02-test:/usr/lib/impala/lib/
- scp hadoop-aws-2.7.2.jar data-02-test:/usr/lib/impala/lib/
- scp ant-1.9.1.jar data-02-test:/usr/lib/impala/lib/
- scp hive-orc-2.2.0.jar data-02-test:/usr/lib/impala/lib/
- scp hive-service-rpc-2.1.1-cdh6.3.0.jar data-02-test:/usr/lib/impala/lib/
- scp hive-storage-api-2.1.1-cdh6.3.0.jar data-02-test:/usr/lib/impala/lib/
- scp sentry-binding-hive-common-2.1.0.jar data-02-test:/usr/lib/impala/lib/
- scp sentry-core-model-solr-2.1.0.jar data-02-test:/usr/lib/impala/lib/
- scp sentry-policy-engine-2.1.0.jar data-02-test:/usr/lib/impala/lib/
- scp sentry-service-api-2.1.0.jar data-02-test:/usr/lib/impala/lib/
- scp sentry-service-client-2.1.0.jar data-02-test:/usr/lib/impala/lib/
- scp sentry-shaded-miscellaneous-2.1.0-cdh6.3.3.jar data-02-test:/usr/lib/impala/lib/
- scp hive-hcatalog-core-2.1.1-cdh6.3.0.jar data-02-test:/usr/lib/impala/lib/
- scp hive-hcatalog-pig-adapter-2.1.1-cdh6.3.0.jar data-02-test:/usr/lib/impala/lib/
- scp hive-hcatalog-server-extensions-2.1.1-cdh6.3.0.jar data-02-test:/usr/lib/impala/lib/
- scp hive-hcatalog-streaming-2.1.1-cdh6.3.0.jar data-02-test:/usr/lib/impala/lib/
- scp commons-lang-2.6.jar data-02-test:/usr/lib/impala/lib/
- scp hadoop-azure-datalake-3.0.0-cdh6.3.0.jar data-02-test:/usr/lib/impala/lib/
- scp guava-14.0.1.jar data-02-test:/usr/lib/impala/lib/
- 修改jar包重定向软链接
- ln -sf /usr/lib/avro/avro.jar /usr/lib/impala/lib/avro.jar
- ln -sf /usr/lib/impala/lib/commons-lang-2.6.jar /usr/lib/impala/lib/commons-lang.jar
- ln -sf /usr/lib/impala/lib/guava-14.0.1.jar /usr/lib/impala/lib/guava.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop/hadoop-annotations.jar /usr/lib/impala/lib/hadoop-annotations.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop-mapreduce/hadoop-archives.jar /usr/lib/impala/lib/hadoop-archives.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop/hadoop-auth.jar /usr/lib/impala/lib/hadoop-auth.jar
- ln -sf /usr/lib/impala/lib/hadoop-aws-2.7.2.jar /usr/lib/impala/lib/hadoop-aws.jar
- ln -sf /usr/lib/impala/lib/hadoop-azure-datalake-3.0.0-cdh6.3.0.jar /usr/lib/impala/lib/hadoop-azure-datalake.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop/hadoop-azure.jar /usr/lib/impala/lib/hadoop-azure.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop/hadoop-common.jar /usr/lib/impala/lib/hadoop-common.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop-mapreduce/hadoop-distcp.jar /usr/lib/impala/lib/hadoop-distcp.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop-hdfs/hadoop-hdfs-client.jar /usr/lib/impala/lib/hadoop-hdfs-client.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop-hdfs/hadoop-hdfs.jar /usr/lib/impala/lib/hadoop-hdfs.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop-mapreduce/hadoop-mapreduce-client-common.jar /usr/lib/impala/lib/hadoop-mapreduce-client-common.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop-mapreduce/hadoop-mapreduce-client-core.jar /usr/lib/impala/lib/hadoop-mapreduce-client-core.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar /usr/lib/impala/lib/hadoop-mapreduce-client-jobclient.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar /usr/lib/impala/lib/hadoop-mapreduce-client-shuffle.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop-yarn/hadoop-yarn-api.jar /usr/lib/impala/lib/hadoop-yarn-api.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop-yarn/hadoop-yarn-client.jar /usr/lib/impala/lib/hadoop-yarn-client.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop-yarn/hadoop-yarn-common.jar /usr/lib/impala/lib/hadoop-yarn-common.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar /usr/lib/impala/lib/hadoop-yarn-server-applicationhistoryservice.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop-yarn/hadoop-yarn-server-common.jar /usr/lib/impala/lib/hadoop-yarn-server-common.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop-yarn/hadoop-yarn-server-nodemanager.jar /usr/lib/impala/lib/hadoop-yarn-server-nodemanager.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar /usr/lib/impala/lib/hadoop-yarn-server-resourcemanager.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop-yarn/hadoop-yarn-server-web-proxy.jar /usr/lib/impala/lib/hadoop-yarn-server-web-proxy.jar
- ln -sf /usr/hdp/3.1.4.0-315/hbase/lib/hbase-annotations.jar /usr/lib/impala/lib/hbase-annotations.jar
- ln -sf /usr/lib/impala/lib/hbase-client-1.2.6.jar /usr/lib/impala/lib/hbase-client.jar
- ln -sf /usr/lib/impala/lib/hbase-common-1.3.0.jar /usr/lib/impala/lib/hbase-common.jar
- ln -sf /usr/hdp/3.1.4.0-315/hbase/lib/hbase-hadoop2-compat.jar /usr/lib/impala/lib/hbase-hadoop2-compat.jar
- ln -sf /usr/hdp/3.1.4.0-315/hbase/lib/hbase-hadoop2-compat-2.0.2.3.1.4.0-315-tests.jar /usr/lib/impala/lib/hbase-hadoop2-compat-tests.jar
- ln -sf /usr/hdp/3.1.4.0-315/hbase/lib/hbase-hadoop-compat.jar /usr/lib/impala/lib/hbase-hadoop-compat.jar
- ln -sf /usr/hdp/3.1.4.0-315/hbase/lib/hbase-http.jar /usr/lib/impala/lib/hbase-http.jar
- ln -sf /usr/hdp/3.1.4.0-315/hbase/lib/hbase-mapreduce.jar /usr/lib/impala/lib/hbase-mapreduce.jar
- ln -sf /usr/hdp/3.1.4.0-315/hbase/lib/hbase-metrics-api.jar /usr/lib/impala/lib/hbase-metrics-api.jar
- ln -sf /usr/hdp/3.1.4.0-315/hbase/lib/hbase-metrics.jar /usr/lib/impala/lib/hbase-metrics.jar
- ln -sf /usr/hdp/3.1.4.0-315/hbase/lib/hbase-procedure.jar /usr/lib/impala/lib/hbase-procedure.jar
- ln -sf /usr/hdp/3.1.4.0-315/hbase/lib/hbase-protocol.jar /usr/lib/impala/lib/hbase-protocol.jar
- ln -sf /usr/hdp/3.1.4.0-315/hbase/lib/hbase-protocol-shaded.jar /usr/lib/impala/lib/hbase-protocol-shaded.jar
- ln -sf /usr/hdp/3.1.4.0-315/hbase/lib/hbase-replication.jar /usr/lib/impala/lib/hbase-replication.jar
- ln -sf /usr/hdp/3.1.4.0-315/hbase/lib/hbase-server.jar /usr/lib/impala/lib/hbase-server.jar
- ln -sf /usr/hdp/3.1.4.0-315/hbase/lib/hbase-shaded-miscellaneous-2.2.0.jar /usr/lib/impala/lib/hbase-shaded-miscellaneous.jar
- ln -sf /usr/hdp/3.1.4.0-315/hbase/lib/hbase-shaded-netty-2.2.0.jar /usr/lib/impala/lib/hbase-shaded-netty.jar
- ln -sf /usr/hdp/3.1.4.0-315/hbase/lib/hbase-shaded-protobuf-2.2.0.jar /usr/lib/impala/lib/hbase-shaded-protobuf.jar
- ln -sf /usr/hdp/3.1.4.0-315/hbase/lib/hbase-zookeeper.jar /usr/lib/impala/lib/hbase-zookeeper.jar
- ln -sf /usr/hdp/3.1.4.0-315/hive/lib/ant-1.9.1.jar /usr/lib/impala/lib/hive-ant.jar
- ln -sf /usr/hdp/3.1.4.0-315/hive/lib/hive-beeline.jar /usr/lib/impala/lib/hive-beeline.jar
- ln -sf /usr/hdp/3.1.4.0-315/hive/lib/hive-classification.jar /usr/lib/impala/lib/hive-classification.jar
- ln -sf /usr/hdp/3.1.4.0-315/hive/lib/hive-cli.jar /usr/lib/impala/lib/hive-cli.jar
- ln -sf /usr/hdp/3.1.4.0-315/hive/lib/hive-common.jar /usr/lib/impala/lib/hive-common.jar
- ln -sf /usr/hdp/3.1.4.0-315/hive/lib/hive-exec.jar /usr/lib/impala/lib/hive-exec.jar
- ln -sf /usr/hdp/3.1.4.0-315/hive/lib/hive-hbase-handler.jar /usr/lib/impala/lib/hive-hbase-handler.jar
- ln -sf /usr/lib/impala/lib/hive-hcatalog-core-2.1.1-cdh6.3.0.jar /usr/lib/impala/lib/hive-hcatalog-core.jar
- ln -sf /usr/lib/impala/lib/hive-hcatalog-pig-adapter-2.1.1-cdh6.3.0.jar /usr/lib/impala/lib/hive-hcatalog-pig-adapter.jar
- ln -sf /usr/lib/impala/lib/hive-hcatalog-server-extensions-2.1.1-cdh6.3.0.jar /usr/lib/impala/lib/hive-hcatalog-server-extensions.jar
- ln -sf /usr/lib/impala/lib/hive-hcatalog-streaming-2.1.1-cdh6.3.0.jar /usr/lib/impala/lib/hive-hcatalog-streaming.jar
- ln -sf /usr/hdp/3.1.4.0-315/hive/lib/hive-llap-client.jar /usr/lib/impala/lib/hive-llap-client.jar
- ln -sf /usr/hdp/3.1.4.0-315/hive/lib/hive-llap-common.jar /usr/lib/impala/lib/hive-llap-common.jar
- ln -sf /usr/hdp/3.1.4.0-315/hive/lib/hive-llap-tez.jar /usr/lib/impala/lib/hive-llap-tez.jar
- ln -sf /usr/lib/impala/lib/hive-metastore-2.3.9.jar /usr/lib/impala/lib/hive-metastore.jar
- ln -sf /usr/lib/impala/lib/hive-orc-2.2.0.jar /usr/lib/impala/lib/hive-orc.jar
- ln -sf /usr/hdp/3.1.4.0-315/hive/lib/hive-serde.jar /usr/lib/impala/lib/hive-serde.jar
- ln -sf /usr/hdp/3.1.4.0-315/hive/lib/hive-service.jar /usr/lib/impala/lib/hive-service.jar
- ln -sf /usr/lib/impala/lib/hive-service-rpc-2.1.1-cdh6.3.0.jar /usr/lib/impala/lib/hive-service-rpc.jar
- ln -sf /usr/hdp/3.1.4.0-315/hive/lib/hive-shims-common.jar /usr/lib/impala/lib/hive-shims-common.jar
- ln -sf /usr/hdp/3.1.4.0-315/hive/lib/hive-shims.jar /usr/lib/impala/lib/hive-shims.jar
- ln -sf /usr/hdp/3.1.4.0-315/hive/lib/hive-shims-scheduler.jar /usr/lib/impala/lib/hive-shims-scheduler.jar
- ln -sf /usr/lib/impala/lib/hive-storage-api-2.1.1-cdh6.3.0.jar /usr/lib/impala/lib/hive-storage-api.jar
- ln -sf /usr/lib/kudu/kudu-client.jar /usr/lib/impala/lib/kudu-client.jar
- ln -sf /usr/hdp/3.1.4.0-315/hadoop/lib/native/libhadoop.so /usr/lib/impala/lib/libhadoop.so
- ln -sf /usr/hdp/3.1.4.0-315/hadoop/lib/native/libhadoop.so.1.0.0 /usr/lib/impala/lib/libhadoop.so.1.0.0
- ln -sf /usr/hdp/3.1.4.0-315/usr/lib/libhdfs.so /usr/lib/impala/lib/libhdfs.so
- ln -sf /usr/hdp/3.1.4.0-315/usr/lib/libhdfs.so.0.0.0 /usr/lib/impala/lib/libhdfs.so.0.0.0
- ln -sf /usr/lib/parquet/parquet-hadoop-bundle.jar /usr/lib/impala/lib/parquet-hadoop-bundle.jar
- ln -sf /usr/lib/impala/lib/sentry-binding-hive-common-2.1.0.jar /usr/lib/impala/lib/sentry-binding-hive-common.jar
- ln -sf /usr/lib/sentry/lib/sentry-binding-hive-conf.jar /usr/lib/impala/lib/sentry-binding-hive-conf.jar
- ln -sf /usr/lib/sentry/lib/sentry-binding-hive-follower.jar /usr/lib/impala/lib/sentry-binding-hive-follower.jar
- ln -sf /usr/lib/sentry/lib/sentry-binding-hive.jar /usr/lib/impala/lib/sentry-binding-hive.jar
- ln -sf /usr/lib/sentry/lib/sentry-core-common.jar /usr/lib/impala/lib/sentry-core-common.jar
- ln -sf /usr/lib/sentry/lib/sentry-core-model-db.jar /usr/lib/impala/lib/sentry-core-model-db.jar
- ln -sf /usr/lib/sentry/lib/sentry-core-model-indexer.jar /usr/lib/impala/lib/sentry-core-model-indexer.jar
- ln -sf /usr/lib/sentry/lib/sentry-core-model-kafka.jar /usr/lib/impala/lib/sentry-core-model-kafka.jar
- ln -sf /usr/lib/impala/lib/sentry-core-model-solr-2.1.0.jar /usr/lib/impala/lib/sentry-core-model-solr.jar
- ln -sf /usr/lib/sentry/lib/sentry-hdfs-common.jar /usr/lib/impala/lib/sentry-hdfs-common.jar
- ln -sf /usr/lib/sentry/lib/sentry-policy-common.jar /usr/lib/impala/lib/sentry-policy-common.jar
- ln -sf /usr/lib/impala/lib/sentry-policy-engine-2.1.0.jar /usr/lib/impala/lib/sentry-policy-engine.jar
- ln -sf /usr/lib/sentry/lib/sentry-provider-cache.jar /usr/lib/impala/lib/sentry-provider-cache.jar
- ln -sf /usr/lib/sentry/lib/sentry-provider-common.jar /usr/lib/impala/lib/sentry-provider-common.jar
- ln -sf /usr/lib/sentry/lib/sentry-provider-db.jar /usr/lib/impala/lib/sentry-provider-db.jar
- ln -sf /usr/lib/sentry/lib/sentry-provider-file.jar /usr/lib/impala/lib/sentry-provider-file.jar
- ln -sf /usr/lib/impala/lib/sentry-service-api-2.1.0.jar /usr/lib/impala/lib/sentry-service-api.jar
- ln -sf /usr/lib/impala/lib/sentry-service-client-2.1.0.jar /usr/lib/impala/lib/sentry-service-client.jar
- ln -sf /usr/lib/impala/lib/sentry-shaded-miscellaneous-2.1.0-cdh6.3.3.jar /usr/lib/impala/lib/sentry-shaded-miscellaneous.jar
- ln -sf /usr/hdp/3.1.4.0-315/zookeeper/zookeeper.jar /usr/lib/impala/lib/zookeeper.jar
- 执行拷贝
- cp /etc/hive/conf/hive-site.xml /etc/impala/conf
- cp /etc/hadoop/conf/*.xml /etc/impala/conf
- 权限配置（所有机器都应当进行）
- 添加附属组命令
- usermod -G hive,hdfs,hadoop impala
- sudo -u hdfs hadoop fs -mkdir /user/impala
- sudo -u hdfs hadoop fs -chown impala /user/impala
- 设置scoket path
- 在每个节点上创建/var/run/hadoop-hdfs
- mkdir -p /var/run/hadoop-hdfs
- 注意：该文件夹可能已经存在，应当确认用impala是否有权限进行读写
- 如果已经存在，将用户impala加入该文件所属的组，并修改该文件组的权限即：chmod 777 /var/run/hadoop-hdfs
- data-01-test之外的服务器无法启动impala-daemon 原因在于/etc/default/bigtop-utils 文件内Unable to find Java. JAVA_HOME should be set in /etc/default/bigtop-utils 需要设置java_HOME
- vim /etc/default/bigtop-utils
- export JAVA_HOME=/usr/local/java/jdk1.8.0_241
- 发现云创测试环境 的 表oms_sale_order建表语句中有 insert_only限制，尝试重新在impala建表试试
- 建表语句中无insert only限定时，可以正常使用impala访问数据
- catalog的 启动 失败，修改vim /etc/default/impala 将MYSQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar 这一行有效 化即可
- vim /etc/defult/impala （每台机配置如下impala标准配置即可解决从节点无法启动impalad的问题） scp impala data-02-dev:/etc/default/ 分发到集群的各节点也可以
- IMPALA_CATALOG_SERVICE_HOST=data-01-test
- IMPALA_STATE_STORE_HOST=data-01-test
- IMPALA_STATE_STORE_PORT=24000
- IMPALA_BACKEND_PORT=22000
- IMPALA_LOG_DIR=/var/log/impala
- IMPALA_SCRATCH_DIR=/tmp/impala_scratch
- MEM_LIMIT=20gb
- IMPALA_CATALOG_ARGS=” -log_dir=${IMPALA_LOG_DIR} -state_store_host=${IMPALA_STATE_STORE_HOST} -state_store_port=${IMPALA_STATE_STORE_PORT}”
- IMPALA_STATE_STORE_ARGS=” -log_dir=${IMPALA_LOG_DIR} -state_store_host=${IMPALA_STATE_STORE_HOST} -state_store_port=${IMPALA_STATE_STORE_PORT}”
- IMPALA_SERVER_ARGS=” \
- -log_dir=${IMPALA_LOG_DIR} \
- –scratch_dirs=${IMPALA_SCRATCH_DIR} \
- -mem_limit=${MEM_LIMIT} \
- -catalog_service_host=${IMPALA_CATALOG_SERVICE_HOST} \
- -state_store_host=${IMPALA_STATE_STORE_HOST} \
- -state_store_port=${IMPALA_STATE_STORE_PORT} \
- -use_statestore \
- -be_port=${IMPALA_BACKEND_PORT} \
- “
- ENABLE_CORE_DUMPS=true
- LIBHDFS_OPTS=-Djava.library.path=/usr/lib/impala/lib
- MYSQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar
- # IMPALA_BIN=/usr/lib/impala/sbin
- # IMPALA_HOME=/usr/lib/impala
- # HIVE_HOME=/usr/lib/hive
- # HBASE_HOME=/usr/lib/hbase
- # IMPALA_CONF_DIR=/etc/impala/conf
- # HADOOP_CONF_DIR=/etc/impala/conf
- # HIVE_CONF_DIR=/etc/impala/conf
- # HBASE_CONF_DIR=/etc/impala/conf
作者 admin
张宴银，大数据开发工程师