How to Create Sparksession with Hive Support (Fails with "Hive Classes Are Not Found")

How to create SparkSession with Hive support (fails with Hive classes are not found)?

Add following dependency to your maven project.

<dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-hive_2.11</artifactId>
        <version>2.0.0</version>
</dependency>

Unable to instantiate SparkSession with Hive support error when trying to process hive table with spark

You should remove provided for your spark-hive dependency:

"org.apache.spark" %% "spark-hive" % "2.3.2" % "provided" 

change to

"org.apache.spark" %% "spark-hive" % "2.3.2"

Spark with Hive, Unable to instantiate SparkSession with Hive support because Hive classes are not found

<dependency> <!-- Spark dependency -->
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-hive_2.11</artifactId>
    <version>2.4.4</version>
    <scope>compile</scope>
</dependency>

I dont know why compile is the scope it should be runtime. Since You are using maven shade plugin you can package uber jar(with target/original-kafka-consumer-0.1-SNAPSHOT.jar) with all dependencies in one umbrella/archive and it will be in the classpath so that nothing is missed try this.

Also hive-site.xml should be in classpath. then there is no need to seperately configure metastoreuris. in programatic way.

Further reading

Exception while running hive support on Spark: Unable to instantiate SparkSession with Hive support because Hive classes are not found

try adding

libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.4.5"

enableHiveSupport throws error in java spark code

If you are running in IDE, I recommend to use .master("local") in you SparkSession object.

Next important point is that the version of spark-hive should match with spark-core and spark-sql versions. for safety you can define dependency as

<properties>
    <spark.version>2.0.0</spark.version>
</properties>
<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-hive_2.11</artifactId>
        <version>${spark.version}</version>
    </dependency>

---

Related Topics