AWS Data Engineer-Unity Catalog

KPI Partners is seeking a skilled Data Engineer with expertise in Databricks, Unity Catalogue, Pyspark, Python, and SQL. As a member of our dynamic team, you will be responsible for designing, developing, and maintaining scalable data pipelines and solutions that support our data-driven decision-making processes.

Key Responsibilities: Develop and maintain data pipelines using Databricks and Pyspark to ensure efficient data processing and transformation. Utilize Unity Catalogue to manage and govern data assets and ensure data quality and security. Collaborate with data analysts, data scientists, and other stakeholders to understand data requirements and deliver actionable insights. Write efficient and optimized queries using SQL to extract and manipulate data from various sources. Monitor and troubleshoot data pipeline performance, implement improvements, and provide support for data-related issues. Document data engineering processes, architecture, and workflows to ensure knowledge sharing within the team. Qualifications: Bachelor’s degree in computer science, Data Science, or a related field. Proven experience as a Data Engineer or in a similar role with a strong understanding of data engineering principles Proficiency in Databricks, Unity Catalogue, Pyspark, Python, and SQL. Experience with data modelling, ETL processes, and data warehousing concepts. Strong problem-solving skills and the ability to work collaboratively in a team environment. Excellent communication skills to convey technical concepts to non-technical stakeholders.