How to Read a Config file in spark scala (SBT)
It is best practice to have a config file for your Scala spark applications which may contain pipeline configurations. There are many options available, but Typesafe is one of the popular configuration library for JVM based languages. we are going to use use SBT build tool to manage dependencies.
In this article, will discuss about followings:
- How to add typesafe in build.sbt file
- How to create or read a conf file
Let’s get started !
Step 1: Create a application.conf file under “src/main/resources/”
app {
appname: "Test app"
owner : "Rahul roy"
email: "xyz@gmail.com"
}The application.conf contains configurations in the form of key-value pair.
Step 2: Add typesafe dependency to build.sbt file
libraryDependencies ++= Seq(
"com.typesafe" % "config" % "1.3.3"
)Step 3: Load application.conf file
import com.typesafe.config.{Config, ConfigFactory}
val applicationConf: Config = ConfigFactory.load("appliccation.conf")you can give any name instead of application.conf.
no need to mention absolute or relative path of application.conf file.
Step 4: Reading conf file
val appname = applicationConf.getString("app.owner")
print(appname)Output: Rahul roy
You can also pass this conf file with spark-submit as command line argument.
Library Github link — https://github.com/lightbend/config
Thanks for readings. Please follow me for more articles like this.
Please do share the article, if you liked it. Any comments or suggestions are welcome.
