Senior Databricks AI / ML Engineer

Role Overview

We are seeking a highly skilled Senior Databricks AI/ML Engineer to lead the design, development, and deployment of advanced analytics and machine learning solutions. The ideal candidate will bring deep expertise in Databricks, strong hands-on experience in both AI and traditional ML models, and advanced proficiency in SQL and Python. The engineer will work closely with cross-functional teams to build scalable data and ML pipelines, optimize models, and deliver impactful insights for business growth.

Key Responsibilities

- Design, build, and deploy end-to-end AI and traditional ML models using Databricks. - Develop scalable ETL/ELT pipelines, feature engineering workflows, and model training pipelines. - Leverage Databricks Lakehouse capabilities for data preparation, model development, and production deployment. - Manage the entire machine learning lifecycle using MLflow, including experiment tracking, model registry, and reproducible workflows. - Write high-quality, optimized Python and SQL code for data processing and model operationalization. - Collaborate with data engineers, data architects, and business stakeholders to understand requirements and deliver analytical solutions. - Implement best practices for versioning, CI/CD for ML, and performance optimization. - Monitor model performance and retrain models as needed in production environments. - Troubleshoot performance bottlenecks, optimize cluster usage, and ensure cost-efficient Databricks operations. - Provide technical guidance and mentorship to junior team members.

Required Skills & Experience

- Minimum 5 years of hands-on experience in Databricks, including Spark, Delta Lake, and Lakehouse architecture. - Minimum 4 years of experience in AI/ML, covering both classical ML algorithms and advanced deep learning techniques. - Proven expertise in Python, including libraries such as Pandas, NumPy, Scikit-Learn, and PySpark. - Strong proficiency in SQL, including writing complex queries and performance tuning. - Hands-on experience with MLflow for tracking, versioning, and managing ML models. - Practical experience with data preprocessing, feature engineering, model evaluation, and deployment. - Solid understanding of distributed computing concepts and performance optimization in Spark. - Experience working in Agile environments and version control tools such as Git.

Good to Have

- Knowledge or hands-on experience with SAP systems, especially in data extraction or integration with SAP Datasphere, SAC, BW, or SAP S/4HANA. - Exposure to cloud platforms (Azure/AWS/GCP) and containerization technologies such as Docker or Kubernetes. - Experience in integrating ML models with business applications or APIs.