Position: Data Engineer
Data Engineer is responsible for various functions, which primarily involve providing
Analytical and technical skills necessary to innovate, build, and maintain well-managed data solutions and capabilities to solve business problems. It also includes working closely with the Development team to understand various features and events data to be captured for deriving better and more meaningful business insights. Senior Data Engineer’s job also includes delivering data and insights to the campaign, and gratification team to ensure revenue, customer satisfaction and platform growth goals are met.
DE is expected to:
· Design and development of low latency, high-performance data analytics applications.
· Design and manage database schemas and relations.
· Collaborate with peer data engineers and platform architects on various projects to maintain and enhance data platforms.
· Design flexible taxonomy for data ingestion from the core services to the Analytics system/HDFS.
· Work on open source big data technologies such as Hadoop, MapReduce, Spark etc.
· Work on relational database systems such as Hive etc.
· Use data to discover tasks that can be automated.
· Troubleshoot and provide fix for any production bug in a timely manner.
· Anticipate, identify and solve issues concerning data management and quality.
· Clean, prepare and optimize data at scale for ingestion and consumption.
· Mentor and develop fellow data engineering team in adopting best practices.
Requirements:
· 5+ years of working experience in data engineering.
· Expert Knowledge of programming languages such as Scala/Python/Java/Shell.
· Expert knowledge of Query Engines like Hive, Spark Sql.
· Hands-on-experience in ETL tools like Apache NiFi etc.
· Hands-on experience in Spark, Map reduce, HDFS.
· Familiar with monitoring and tuning tools like Dr. Elephant, Grafana etc.,
· Experience with some dashboard or reporting tools like Kibana, Zoom data, Tableau etc.
· Familiar with online analytics and reporting tools like Google Analytics, Elastic Search etc.
· Expertise in performance tuning and resource optimization techniques of Spark and Hive.
· Should have good understanding of query optimization techniques.
· Well familiar with concepts related to data modelling, metadata management, data lineage etc.
· Good understanding of GIT and CI-CD pipelines.
· Good knowledge of commonly used Linux and windows commands.
· BE/B.Tech/MCA from a reputed institute.
· Excellent teamwork skills.
Follow us:
.png)
.png)
.png)
