Typesafe config file for spark applications
It is best practice to have a config file for your Scala spark applications which may contain pipeline configurations. There are many options available, but Typesafe is one of the popular configuration library for JVM based languages. we are going to use use SBT build tool to manage dependencies.
In…
Case When statement in SQL
In SQL world, very often we write case when statement to deal with conditions. Spark also provides “when function” to deal with multiple conditions.
In this article, will talk about following:
- when
- when otherwise
- when with multiple conditions
Let’s get started !
Let’s consider an example, Below is a spark…
Spark applications must have a SparkSession. which acts as an entry point for an applications. It was added in park 2.0 before this Spark Context was the entry point of any spark application. It allows you to control spark applications through a driver process called the SparkSession.
Let’s get started…
We all get this error very often while working with Scala & spark. There could be several reason for this error. But if you are sure that you have installed all required plugins or dependencies with correct version. And still getting this error.Then this article is for you.
Let’s get…
