Lead Data Engineer

Job Description Notice Period:

Immediate Contract Duration: 10 Months Experience:

9+ years Location:

Pune/Hyderabad/Bangalore/ Gurgaon/Chennai/Coimbatore Work Mode :- Hybrid

Must have Skills: Python, AWS, Snowflake, DBT Mandatory Skills: Programming & Engineering : Expert-level Python (OOP, context managers, decorators, async, packaging), strong software design, and performance optimization. SQL & Warehousing : Strong SQL

Job Description : We are seeking a hands-on Tech Lead with deep expertise in Python, AWS, Snowflake, and modern data engineering to lead a team of four data engineers in designing, building, and operating robust, secure, and scalable data platforms and products. You will set technical direction, own architectural decisions, and ensure delivery excellence across the data lifecycle, from Infrastructure as Code (IaC) and pipeline orchestration to automated testing, deployment, and production reliability. You will operate at the intersection of software engineering and data engineering, partnering with R&D, analytics, and business stakeholders to translate strategic needs into production-grade solutions. You’ll mentor engineers, enforce engineering best practices, and establish standards for quality, reliability, and cost efficiency. This role requires strong stakeholder management, clear communication, and the ability to drive outcomes in a regulated, cloud-native environment. Key Responsibilities: Technical Leadership & Architecture : Define technical vision, reference architectures, and design patterns for data ingestion, transformation, governance, and serving across Python, DBT, and Snowflake. Hands-on Delivery : Design and develop end-to-end data pipelines, APIs, and backend systems using Python, DBT, and Snowflake; write first-class SQL and production-grade code. Engineering Excellence : Establish and enforce coding standards, testing strategies, branching models, and code review practices to ensure clean, maintainable, and secure code. Infrastructure as Code : Lead IaC using Terraform/CloudFormation/AWS CDK to automate environment provisioning, enforce guardrails, and enable repeatable deployments. AWS Data Stack Integration : Architect and implement integrations using AWS Glue, Lambda, Athena, S3, Step Functions, and event-driven/serverless patterns. Quality & Testing : Implement comprehensive test coverage (unit, integration, data quality) using Pytest, DBT tests, and Great Expectations; embed observability for data reliability. CI/CD & DevOps : Design and maintain CI/CD pipelines (GitHub Actions, Jenkins) for data workflows, DBT jobs, and Python services; enable automated deployments and rollback strategies. Reliability & Cost Optimization : Monitor production pipelines, performance, and spend; drive remediation, tuning, and resilience (e.g., retries, dead-letter queues, backpressure). Security & Compliance : Partner with InfoSec/Compliance to embed data security controls, IAM policies, encryption, lineage, auditability, and regulatory requirements (e.g., GxP, SOX, ISO 27001). Documentation & Standards : Create and maintain high-quality documentation for architectures, designs, runbooks, and operational procedures; promote knowledge sharing. Project & Roadmap Planning : Collaborate on backlog prioritization, estimations, release planning, and cross-team dependencies; track delivery against OKRs. Stakeholder Management : Engage business and R&D stakeholders to clarify requirements, define SLAs, and communicate status/risks; translate business needs into technical outcomes. Team Mentorship : Coach and mentor data engineers on best practices, architectural thinking, and career growth; foster a culture of continuous learning and accountability. Incident Management : Lead on-call rotations and incident response; perform root cause analysis and drive corrective actions and postmortems. Continuous Improvement : Evaluate new tools and approaches; standardize reusable components, templates, and accelerators to scale delivery across teams