It is best practice to have a config file for your Scala spark applications which may contain pipeline configurations. There are many options available, but Typesafe is one of the popular configuration library for JVM based languages. we are going to use use SBT build tool to manage dependencies.

In…

--

--

In SQL world, very often we write case when statement to deal with conditions. Spark also provides “when function” to deal with multiple conditions.

In this article, will talk about following:

  1. when
  2. when otherwise
  3. when with multiple conditions

Let’s get started !

Let’s consider an example, Below is a spark…

--

--

Spark applications must have a SparkSession. which acts as an entry point for an applications. It was added in park 2.0 before this Spark Context was the entry point of any spark application. It allows you to control spark applications through a driver process called the SparkSession.

Let’s get started…

--

--

Parmanand kumar

Parmanand kumar

Data Engineering | Machine Learning | Front-end | NIT Trichy