Spark Scala Developer

Spark Scala Developer

Shift- Malaysian / EUR

Exp- 8 yrs

Full time Contract Opportunity

Fully Remote

please share resume at Khalil@weradioactive.com

Responsibilities-

Create Scala / Spark jobs for data transformation and aggregation

Produce unit tests for Spark transformations and helper methods

Write Scaladoc-style documentation with all code

Design data processing pipelines

Skills-

Scala (with a focus on the functional programming paradigm)

Scalatest, JUnit, Mockito, Embedded Cassandra

Apache Spark 2.x

Apache Spark RDD API

Apache Spark SQL Data Frame API

Apache Spark MLlib API

Apache Spark GraphX API

Apache Spark Streaming API

Spark query tuning and performance optimization

Data Modelling, Data Warehousing, Hadoop, Hive, ETL

SQL database integration ,Microsoft, Oracle, Postgres, and / or MySQL

Experience working with HDFS, S3, Cassandra, and / or DynamoDB

Deep understanding of distributed systems (e.g. CAP theorem, partitioning, replication, consistency, and consensus)