Senior Data Engineer

Role: Sr. Azure Databricks Data Engineer

Experience: 6+ Years

Location: Navi Mumbai

Duration: Fulltime

Role Overview

We are looking for a Data Engineer to design and build scalable, production-grade data pipelines on Azure using Databricks. The role involves working with high-volume, high-velocity enterprise data across domains such as telecom, retail, and regulatory systems (e.g., GST, eWay Bill).

You will be responsible for building reliable batch and real-time pipelines, ensuring data quality, auditability, and performance at scale.

Key Responsibilities

Data Engineering & Pipeline Development

• Design and implement end-to-end data pipelines using Azure Databricks and PySpark

• Build and maintain batch and real-time ingestion pipelines using:

o Azure Data Factory (ADF)

o Kafka / Azure Event Hubs

• Process TB–PB scale structured and semi-structured datasets (JSON, Parquet, CSV)

Lakehouse Architecture

• Implement and maintain Medallion Architecture (Bronze, Silver, Gold layers)

• Develop reusable data models for analytics, reporting, and downstream consumption

• Ensure data lineage, traceability, and auditability across layers

Delta Lake & Data Management

• Leverage Delta Lake features:

o MERGE (upserts), SCD Type 1/2 implementations

o Schema enforcement and evolution

o ACID-compliant pipelines

• Optimize Delta tables using:

o OPTIMIZE, Z-ORDER, VACUUM

• Handle incremental processing and CDC pipelines

Streaming & Real-Time Processing

• Build low-latency streaming pipelines using Structured Streaming

• Handle:

o Late-arriving data

o Watermarking and windowing

o Exactly-once processing semantics

• Integrate streaming pipelines with downstream Delta tables and serving layers

Performance Optimization & Scalability

• Optimize Spark jobs using:

o Partitioning strategies

o Broadcast joins

o Caching and persistence

o Adaptive Query Execution (AQE)

• Troubleshoot performance bottlenecks such as:

o Data skew

o Shuffle issues

o Memory constraints

Orchestration & Workflow Management

• Design orchestration workflows using Azure Data Factory:

o Pipeline dependencies

o Scheduling and triggers

o Retry and failure handling

• Integrate ADF with Databricks Jobs / Workflows for end-to-end execution

Data Quality & Governance

• Implement robust data quality checks, including:

o Schema validation

o Deduplication

o Null and integrity checks

o Data reconciliation across sources

• Handle schema drift and evolving data contracts

• Ensure compliance with regulatory and audit requirements

Domain-Specific Use Cases

• Build pipelines for:

o High-frequency transactional systems

o Regulatory datasets (GST, eWay Bill, financial reporting)

o Retail / telecom data platforms

• Ensure data consistency, reconciliation, and reporting accuracy

Technical Skills Required

Core

• Strong SQL:

o Window functions

o Complex joins

o Query optimization

• PySpark:

o Transformations and actions

o Performance tuning

Azure Ecosystem

• Azure Data Factory (ADF)

• Azure Data Lake Storage Gen2 (ADLS)

• Azure Databricks

Streaming

• Kafka or Azure Event Hubs

• Structured Streaming

Good to Have

• CDC pipeline implementation

• Databricks Autoloader

• Delta optimization techniques

• Multi-region / multi-source ingestion

• Data quality frameworks (e.g., expectation-based validation)

Soft Skills & Expectations

• Strong debugging skills in distributed data systems (Spark)

• Experience handling production incidents and RCA (Root Cause Analysis)

• Ability to work in high-scale, SLA-driven environments

• Effective collaboration with business, analytics, and downstream consumers

• Flexibility to work in WFO / hybrid setup

What Success Looks Like

• Reliable, scalable pipelines handling large-scale enterprise data

• Reduced pipeline failures and improved data SLAs

• High-quality, trusted datasets for business-critical reporting

• Efficient Spark jobs with optimized cost and performance