Sr. Data Engineer – Databricks / Azure

Sr. Data Engineer – Databricks / Azure

| Remote | Contract | 2–11 PM IST | Till end of 2026 (possibility of extension)

Design, build, and maintain automated testing frameworks and Python SDKs for data integration, parity verification, and user accessibility in the Industrial IoT / Smart Buildings domain. We're looking for a Senior Data Engineer with strong Python, PySpark, and Spark skills, with proven experience in automated Data Quality frameworks and CI/CD pipelines, delivering Data & Analytics products using Agile methodology.

Must-Have Skills: 5+ years hands-on data engineering experience in Azure ecosystem PySpark & DataFrames Apache Spark internals (Catalyst Optimizer, Logical Plans) Java Python (building Libraries/SDKs, Py4J) Databricks & Delta Lake optimization Automated Testing frameworks (PyTest, JUnit) for data pipelines Data Quality tools (Great Expectations, Deequ) Azure DevOps, GitHub, CI/CD, artifact management Data modeling & warehousing on Azure Databricks Data governance & security best practices in Azure Preferred Certifications:

Azure Fundamentals, Azure Data Engineer Associate, Databricks Certified Data Engineer Professional

Responsibilities: Implement scalable data solutions on Spark and Flink Develop Python SDK wrapping core libraries for use in Jupyter Notebooks Build an automated Parity Test Suite via CI/CD against Golden Datasets (Spark & Flink) Implement Data Quality rules (Completeness, Validity thresholds) Manage and optimize Azure & Databricks resources for performance and cost Transform, clean, and prepare data using SQL, Python, and Java Monitor and tune workloads/pipelines Maintain CI/CD pipelines and documentation Participate in Agile ceremonies and enforce DE best practices