- insert overwrite table cc_drp.drp_storage_history
- partition (dt=substr(regexp_replace(cast(now() as string),’-‘,”),1,8))
- SELECT
- c.name as drp_store_name,–店仓名称
- cast(c.code as string) as drp_store_code,–店仓编号
- b.no_sku as drp_product_code,–商品条码
- d.name as drp_product_name,–款号
- a.qty as drp_product_qty,–账面数量
- a.creationdate as creation_date,–创建时间
- a.modifieddate as modified_date,–修改时间
- f.value1_code color,–颜色
- f.value2_code size–尺寸
- FROM cc_drp.fa_storage a –728345
- inner join cc_drp.m_product_alias_coy b on trim(cast(a.m_productalias_id as string))=trim(cast(b.id as string)) –728345
- inner join (select * from cc_drp.c_store where ISACTIVE=’Y’ and M_DIM1_ID=1 ) c on trim(cast(a.c_store_id as string))=trim(cast(c.id as string)) –728345
- inner join (select * from cc_drp.m_product where ISACTIVE=’Y’ and M_DIM1_ID=1 ) d on trim(cast(a.m_product_id as string))=trim(cast(d.id as string))
- inner join cc_drp.m_attributesetinstance f on b.m_attributesetinstance_id=f.id
- 问题应当出在hive-common这个包上面
- /usr/lib/impala/lib中的hive-common jar包情况
- lrwxrwxrwx 1 root root 45 Jul 11 16:27 hive-common.jar -> /usr/hdp/3.1.4.0-315/hive/lib/hive-common.jar
- 在/usr/hdp/3.1.4.0-315/hive/lib/中找到 hive -common发现如下信息
- -rw-r–r– 1 root root 500519 Aug 23 2019 hive-common-3.1.0.3.1.4.0-315.jar
- lrwxrwxrwx 1 root root 33 Jun 16 15:21 hive-common.jar -> hive-common-3.1.0.3.1.4.0-315.jar
- 将hive-common-3.1.0.3.1.4.0-315.jar 下载下来 进行解压缩发现存在 org.apache.hadoop.hive.common.HiveStatsUtils 类，
- 在/usr/hdp/3.1.4.0-315/hive/lib/中找到 hive -common发现如下信息
- 查看是否存在 getFileStatusRecurse 方法
- 根据classnotfound.com.cn查到的 org.apache.hadoop.hive.common.HiveStatsUtils 缺失类所在的jar包信息
- 目前情况：
- 1.impala 执行insert overwrite 操作写入数据到分区表，报错 。但是数据写进去了，且查询/var/log/impala找不到报错日志
- 2.写入的 数据直接invalidate metastore 然后 查询失败，无数据
- 3.msck repair table cc_drp.drp_storage_history; 刷新hive的元数据
- 4.再次查询，成功
- NoSuchMethodError: org.apache.hadoop.hive.metastore.MetaStoreUtils.updatePartitionStatsFast(Lorg/apache/hadoop/hive/metastore/api/Partition;Lorg/apache/hadoop/hive/metastore/Warehouse;Lorg/apache/ha
- ln -sf /usr/hdp/3.1.4.0-315/hive/lib/hive-metastore.jar /usr/lib/impala/lib/hive-metastore.jar
- ln -sf /usr/lib/impala/lib/swift-hive-metastore-2.0.jar /usr/lib/impala/lib/swift-hive-metastore.jar
作者 admin
张宴银，大数据开发工程师