且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

在scala/spark代码中不允许在配置单元中添加列

更新时间:2023-11-18 22:21:40

在这里已经多次说过,但只是要重申-Spark不是Hive界面,也不是为实现语言的完全Hive兼容性而设计的(Spark目标SQL标准,Hive使用类似于SQL的自定义查询语言)或功能(Spark是ETL解决方案,Hive是数据仓库解决方案).

It has been said multiple times here, but just to reiterate - Spark is not Hive interface and is not designed for full Hive compatibility in terms of language (Spark targets SQL standard, Hive uses custom SQL-like query language) or capabilities (Spark is ETL solution, Hive is a Data Warehousing solution).

这两个数据布局之间也不完全兼容.

Even data layouts are not fully compatible between these two.

具有Hive支持的Spark是可访问Hive Metastore的Spark,而不是行为类似于Hive的Spark.

Spark with Hive support is Spark with access to Hive metastore, not Spark that behaves like Hive.

如果您需要访问Hive的全部功能,请使用本机客户端或本机(而非Spark)JDBC连接直接连接到Hive,然后从那里进行交互.

If you need to access full set of Hive's features connect to Hive directly with native client or native (not Spark) JDBC connection, and use interact with it from there.