Senior Data Engineer

Principal / Senior Data Engineer – 5- 10+ Years Experience (hand on development/architecture) Specialization: Azure | Databricks | PySpark | Python | SQL | Data Warehousing | Data Products

Core Technical Expertise Azure Cloud Platform: Microsoft Azure (ADLS Gen2, ADF, Synapse, Azure SQL, Key Vault, App Services) Cloud-native architecture design & cost optimization Security & governance (RBAC, Managed Identity, Private Endpoints)

Databricks Engineering: Azure Databricks end-to-end development Lakehouse architecture implementation Delta Live Tables (DLT) & Unity Catalog Performance tuning & cluster optimization CI/CD for Databricks workloads

Big Data & Processing Frameworks: Apache Spark (PySpark advanced transformations, optimization) Structured Streaming & batch pipelines Delta Lake architecture & optimization techniques

Programming & Querying: Python (advanced data engineering & automation) PySpark (distributed data processing) SQL (complex query optimization, performance tuning, analytics engineering)

Data Architecture & Engineering Enterprise Data Warehouse Architecture (EDW) Dimensional modeling (Star/Snowflake schemas) Data Vault 2.0 modeling Metadata-driven ingestion frameworks CDC implementation (Change Data Capture) Medallion architecture (Bronze/Silver/Gold layers) Data lineage, governance, and cataloging Master Data Management (MDM)

Data Products & Analytics Enablement Design & delivery of scalable data products Business-aligned semantic layer design KPI frameworks & enterprise reporting enablement Integration of ERP, SaaS, and operational systems Lakehouse + EDW hybrid architecture

DevOps & Engineering Practices CI/CD pipelines (Azure DevOps, GitHub Actions, Bitbucket) Infrastructure as Code (Terraform, ARM templates) Automated testing (unit, integration, data quality) Monitoring & observability (logging, alerting) Agile & Scrum Methodologies

Strategic Impact Principal-level solution architecture design Cross-functional stakeholder engagement Technical roadmap Strategies and delivery

Advanced Competencies Performance optimization for large-scale distributed systems Data governance frameworks (GDPR, SOX compliance alignment) Cost optimization strategies in cloud data platforms Migration from legacy EDW (e.g., Teradata, Oracle, SQL Server) to Lakehouse