AI Engineer - Azure Databricks

Experience Required

10+ Years

Skills

Digital: Microsoft Azure

Digital: Azure Machine Learning (ML)

Technical Must-Haves

Hands-on experience with

Unity Catalog governance and lineage

Strong experience with Azure services:

ADLS Gen2

Azure Data Factory (ADF) Pipelines

Azure Key Vault

Azure Event Hub / Service Bus

Azure DevOps

Proven experience delivering

RAG and agentic patterns , including:

Embeddings pipelines

Chunking strategies

Vector indexing

Evaluation frameworks

Solid SQL expertise and strong understanding of:

Data modeling

Medallion Architecture (Bronze/Silver/Gold)

ACID transactions

Proven experience implementing

Model Context Protocol (MCP)

for agent-tool integrations:

Building MCP servers/adapters to expose tools/resources (SQL, REST, files, APIs, etc.)

Defining tool schemas, input/output contracts, and safety guardrails

Securing MCP integrations using Azure AD (Entra ID), Key Vault, secret scopes, and network controls

Connecting MCP to Unity Catalog governed assets, Vector Search, and Model Serving

Essential Skills

AI / GenAI Engineering (MCP & RAG)

Build GenAI-ready datasets, embeddings pipelines, and vector indexes for RAG workloads.

Use Databricks Genie to accelerate:

Code creation

Optimization

Documentation

Refactoring

Implement MCP patterns to connect AI agents to enterprise data and tools.

Design, build, and operate MCP servers/adapters that expose tools such as:

SQL

REST APIs

File systems

Enterprise APIs

Implement:

Tool schemas

Resource providers

Prompt and tool safety policies

Integrate MCP with Azure Databricks workloads including:

Delta Tables

Unity Catalog

Vector Search

Model Serving

Secure MCP endpoints using:

Key Vault

Secret scopes

Network rules

Role-based access control aligned with Unity Catalog

Operationalize RAG and agentic workloads using:

Vector Search

Model Serving

Feature Store

MLflow

Azure Cloud Integration

Integrate Azure Databricks with:

ADLS Gen2

ADF Pipelines

Key Vault

Event Hub / Service Bus

Azure Functions

Implement telemetry and observability using:

Azure Monitor

Log Analytics

Databricks metrics

Data Governance & Architecture

Implement Unity Catalog for:

Centralized governance

Data lineage

Fine-grained access controls across workspaces

Apply Medallion Architecture (Bronze / Silver / Gold).

Enforce data quality using:

Delta Live Tables

Expectations framework

Manage:

PII/PHI controls

Data masking

Secure data sharing (e.g., Delta Sharing where applicable)

DevOps & Automation

Build CI/CD pipelines using:

Azure DevOps

GitHub Actions

Use Databricks Repos for version control.

Orchestrate workloads using:

Databricks Workflows

Azure Data Factory

Implement environment promotion paths (Dev → Test → Prod).

#J-18808-Ljbffr