Data Engineering Architect

Data Engineering Architect

Experience: 8-12+ years Location: Kolkata/Bangalore Work Mode: Hybrid

Position Summary

: Own enterprise data architecture across OLTP, analytics and Gen AI Agentic AI workloads cloud on-prem. Define lakehouse warehouse target state standards; enable secure, scalable, cost efficient RAG agentic solutions. Partner with data engineering AI teams to deliver governed, retrieve already datasets metadata. Preferred: Primary experience on Azure Databricks + Synapse Platform + Purview OR AWS Snow flake Redshift + Glue with exposure to the other; ServiceNow CMDB/ITSM/ITOM as a governed knowledge source is a plus. Key Responsibilities

: Own roadmap for lakehouse, warehouse, MDM and streaming layers. Architect on Azure OR AWS Databricks Synapse or Snowflake Redshift with ADF Glue Airflow orchestration; design streaming with Kafka OR Kinesis and CDC. Model data dimensional, Data Vault 2.0, DDD; define canonical semantic layers. Build ELTETL SQL, Python, PySpark; batchstream pipelines Kafka, Kinesis, CDC, schema registry. Design vectorization pipelines chunking, embeddings operate vector stores Azure AI Search, Pinecone, FAISS. Data quality metadata lineage: Great Expectations Deequ; Purview AI Collaboration; Open Lineage. Security privacy FinOps: IAMRBACABAC, encryption, PIIGDPRISO; SLAs, cost performance monitoring. Integrate ServiceNow via Table REST APIs; align with CSDM; stream events to agents via Kafka. Must-have Skills Expert SQL; strong Python PySpark; ELTETL orchestration Airflow ADF Glue. Lakehouse warehouse: Databricks Delta; Snowflake Redshift Big Query; modeling Kimball, Data Vault. Streaming integration: Kafka OR Kinesis; CDC; Avro Proto buf. Cloud Azure OR AWS, security IAMTLSKMS, DR performance; governance Purview CollibraAlation, Great Expectations Deequ, PII controls. Good to Have Agentic RAG architectures Lang Chain, Lang Graph with vector stores Azure AI Search Pinecone FAISS and unification of ServiceNow knowledge with lakehouse. Exposure to ServiceNow data integration Table REST APIs, CSDMCMDB is a plus. Cost performance for GenAI data flows: Delta Iceberg Hudi tuning; Databricks Snowflake FinOps; real-time features via Kafka. Education BE / BTech/ ME/ MTech in CSIT Data or related Preferred Certifications: Azure AWS Data Engineer Architect, Databricks, Snowflake.