Job Description
Sr. Data Engineer – Databricks / Azure
| Remote | Contract | 2–11 PM IST | Till end of 2026 (possibility of extension)
Design, build, and maintain automated testing frameworks and Python SDKs for data integration, parity verification, and user accessibility in the Industrial IoT / Smart Buildings domain. We're looking for a Senior Data Engineer with strong Python, PySpark, and Spark skills, with proven experience in automated Data Quality frameworks and CI/CD pipelines, delivering Data & Analytics products using Agile methodology.
Must-Have Skills:
5+ years hands-on data engineering experience in Azure ecosystem
PySpark & DataFrames
Apache Spark internals (Catalyst Optimizer, Logical Plans)
Java
Python (building Libraries/SDKs, Py4J)
Databricks & Delta Lake optimization
Automated Testing frameworks (PyTest, JUnit) for data pipelines
Data Quality tools (Great Expectations, Deequ)
Azure DevOps, GitHub, CI/CD, artifact management
Data modeling & warehousing on Azure Databricks
Data governance & security best practices in Azure
Preferred Certifications:
Azure Fundamentals, Azure Data Engineer Associate, Databricks Certified Data Engineer Professional
Responsibilities:
Implement scalable data solutions on Spark and Flink
Develop Python SDK wrapping core libraries for use in Jupyter Notebooks
Build an automated Parity Test Suite via CI/CD against Golden Datasets (Spark & Flink)
Implement Data Quality rules (Completeness, Validity thresholds)
Manage and optimize Azure & Databricks resources for performance and cost
Transform, clean, and prepare data using SQL, Python, and Java
Monitor and tune workloads/pipelines
Maintain CI/CD pipelines and documentation
Participate in Agile ceremonies and enforce DE best practices
Ready to Apply?
Don't miss this opportunity! Apply now and join our team.
Job Details
Posted Date:
March 7, 2026
Job Type:
Technology
Location:
India
Company:
Thankz Talent Solutions
Ready to Apply?
Don't miss this opportunity! Apply now and join our team.