Lead Data Engineer

Lead Data Engineer

Brazil- Remote Long Term Contract

Job Description:

Proven experience in designing and implementing data solutions on the Databricks platform. Solid understanding of Databricks fundamentals/architecture and have hands on experience in setting up Databricks cluster, working in Databricks modules (Data Engineering, ML and SQL warehouse). Create and maintain documentation for data pipelines, processes, and best practices. Knowledge of Performance Optimization, Monitoring and Automation would be a plus. Understanding of data governance, compliance, and security best practices. Knowledge in anyone of the cloud platforms AWS is a must. Strong Proficiency in Pyspark, Databricks Notebooks, SQL, Python, Scala, or Java. Certifications in Databricks would be beneficial. Strong understanding of data engineering best practices, data modeling, Data warehousing and ETL processes. Staying up-to-date with the latest Databricks features and best practices. Proficiency in SQL for data querying and manipulation. Strong problem-solving and troubleshooting skills. Excellent communication and collaboration skills