Azure Data Engineer

Creospan is a growing tech collective of makers, shakers, and problem solvers, offering solutions today that will propel businesses into a better tomorrow. “Tomorrow’s ideas, built today!” In addition to being able to work alongside equally brilliant and motivated developers, our consultants appreciate the opportunity to learn and apply new skills and methodologies to different clients and industries.

Title: DATAENGINEER - AZURE DATABRICKS

Location: Pune

- Looking for Immediate joiners or someone already serving notice.

Job Title: Azure Databricks Engineer

Experience: 6–10 Years

Location: Pune, India (Hybrid)

Employment Type: Full Time

About the Role

We are seeking an experienced Azure Databricks Engineer to join its data engineering team in Pune. This role focuses on building, optimizing, and supporting enterprise-scale data platforms on Microsoft Azure, leveraging Azure Databricks, Apache Spark, and modern data engineering practices. The engineer will work closely with global teams to deliver secure, scalable, and high-performance analytics solutions.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks

• Build and optimize batch and streaming ETL/ELT workflows using Apache Spark

• Develop data processing solutions using PySpark, Spark SQL, and Delta Lake

• Integrate data from multiple Azure sources such as Azure Data Lake Storage (ADLS Gen2), Azure SQL, and APIs

• Optimize Spark jobs for performance, cost efficiency, and reliability on Azure

• Implement data quality, validation, and reconciliation frameworks

• Collaborate with data architects, analysts, and business stakeholders

• Ensure adherence to data security, governance, and compliance standards

• Support production deployments and troubleshoot pipeline failures

Required Skills & Qualifications

• 6–8 years of experience in Data Engineering

• Strong hands-on experience with Azure Databricks (mandatory)

• Deep expertise in Apache Spark, PySpark, and Spark SQL

• Experience working with Azure Data Lake (ADLS Gen2)

• Hands-on experience with Azure Data Factory (ADF) for orchestration

• Strong SQL skills and experience with relational databases

• Experience with CI/CD pipelines (Azure DevOps preferred)

• Working knowledge of cloud security and access controls on Azure