Spark:Failure: ''Union'' Expected But '(' Found

Spark Error : java.lang.RuntimeException: [5.52] failure: ``union'' expected but `('

There is a missing coma after « timestamp_hour ».

SELECT
       product,
       timestamp_hour,<— HERE
        sum(case when action = 'purchase' then 1 else 0 end) as purchase_count,

Also you should remove all « | » characters from you query

PySpark Error When running SQL Query

Because you are using \ in the first one and that's being passed as odd syntax to spark. If you want to write multi-line SQL statements, use triple quotes:

results5 = spark.sql("""SELECT
  appl_stock.Open
 ,appl_stock.Close
FROM appl_stock
WHERE appl_stock.Close < 500""")

Apache Zeppelin with Java error: ';' expected but 'class' found

I suspect does Zeppelin support Spark JAVA API?

It does not. You can fin a full list of avialable interpreters on the Zeppelin website (https://zeppelin.apache.org/docs/latest/manual/interpreters.html) and there is no Java interpreter there.

Spark - error when selecting a column from a struct in a nested array

Since you have no problem with the element_at() function, I supposed you are using the spark 2.4+, then you can try Spark SQL built-in functions: transform [1][2] + flatten:

>>> mydf1.selectExpr('flatten(transform(home.array_a.array_b, x -> x.a)) as array_field_inside_array').show()
+------------------------+
|array_field_inside_array|
+------------------------+
|                  [1, 3]|
+------------------------+

Where we use transform() function to retrieve only the values of field a of each array element of home.array_a.array_b and transform them to the array [[1], [3]]. then flatten that array into [1, 3]. If you need the result to be [[1, 3]], then just add array() function

array(flatten(transform(home.array_a.array_b, x -> x.a)))

---

Related Topics