且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

如何将多个语句传递到Spark SQL HiveContext

更新时间:2023-11-18 14:44:28

感谢@SamsonScharfrichter的回答.

这将起作用:

hiveContext.sql("set spark.sql.parquet.compression.codec=SNAPPY")
hiveContext.sql("create table MY_TABLE stored as parquet as select * from ANOTHER_TABLE")
val rs = hiveContext.sql("select * from MY_TABLE limit 5")

请注意,在这种特殊情况下,我们需要使用 spark.sql.parquet.compression.codec 代替 parquet.compression 键.

For example I have few Hive HQL statements which I want to pass into Spark SQL:

set parquet.compression=SNAPPY;
create table MY_TABLE stored as parquet as select * from ANOTHER_TABLE;
select * from MY_TABLE limit 5;

Following doesn't work:

hiveContext.sql("set parquet.compression=SNAPPY; create table MY_TABLE stored as parquet as select * from ANOTHER_TABLE; select * from MY_TABLE limit 5;")

How to pass the statements into Spark SQL?

Thank you to @SamsonScharfrichter for the answer.

This will work:

hiveContext.sql("set spark.sql.parquet.compression.codec=SNAPPY")
hiveContext.sql("create table MY_TABLE stored as parquet as select * from ANOTHER_TABLE")
val rs = hiveContext.sql("select * from MY_TABLE limit 5")

Please note that in this particular case instead of parquet.compression key we need to use spark.sql.parquet.compression.codec