AWS Data Engineer

Senior AWS Data Engineer - Python, SQL, Airflow

Location: Remote

Contract: 3 months initial

We are seeking a skilled Data Engineer with a strong background in Quality Assurance to join our client's team. The ideal candidate will be responsible for developing and executing a comprehensive data quality strategy as they migrate their SQL databases to AWS. Your expertise in Python, SQL, and ETL frameworks like Airflow and dbt will ensure high-quality data pipelines and effective diagnosis of pipeline failures.

Key Responsibilities:

- Data Quality Strategy Development: Design and implement a robust data quality strategy that addresses current challenges in SQL to AWS migration, ensuring data integrity and reliability throughout the process. - Data Quality Assurance: Conduct thorough quality assessments, validate data transformations, and ensure that all data quality metrics are met. Focus on identifying gaps and areas for improvement within existing ETL workflows. - Technical Proficiency: Utilize Python and SQL for extracting, transforming, and loading data. Implement automated testing frameworks (pytest) to validate data integrity and functionality across pipelines. - ETL Management: Leverage tools like Airflow and dbt to orchestrate data workflows and conduct regular checks to ensure pipeline performance. Identify and troubleshoot sources of failures. - Cloud Infrastructure: Manage data storage and processing on AWS platforms, specifically with Redshift and S3. Optimize data storage solutions and ensure best practices for data security and accessibility. - Root-Cause Analysis: Perform strong root-cause analysis to diagnose and resolve pipeline failures effectively. Document issues and fixes to improve future processes and communication with stakeholders. - Collaboration: Work closely with cross-functional teams, including data scientists and business analysts, to continuously improve data processes and quality metrics. Communicate findings clearly and provide recommendations to enhance system performance. - Documentation: Maintain comprehensive documentation of data quality processes, testing outcomes, and any identified issues, along with their resolutions.

Qualifications:

- Bachelor's degree in Computer Science, Data Science, Engineering, or a related field. - Proven experience as a Data Engineer, Data Analyst, or in a similar role with a focus on QA. - Strong proficiency in Python, SQL, and experience with data pipeline tools such as Airflow and dbt. - Familiarity with AWS services, particularly Redshift and S3. - Excellent English speaking and communication skills, with experience presenting technical concepts to non-technical stakeholders. - Strong analytical skills with a keen ability to perform root-cause analysis and troubleshoot complex data issues. - Knowledge of data quality frameworks and methodologies is a plus.