QA - Azure and Databricks

Job Role - QA with Azure and Databricks

Experience Required - 7 to 14 Years

Location - Indore/ Pune/ Chennai/ Bangalore

Immediate joiners are preferred.

- Key Responsibilities - 1) Test Strategy & Planning - Define and maintain test strategies for data pipelines, notebooks, jobs, and APIs across Databricks & Azure. - Translate business rules and data quality SLAs into testable acceptance criteria. - Establish risk-based test plans across functional, integration, system, and non-functional (performance, scalability, security) testing. - 2) Data & Pipeline Testing (Databricks / Spark / Delta) - Develop automated tests for Spark transformations (e.g., PySpark, Spark SQL) and Delta Lake features (MERGE, time travel, schema evolution). - Implement data quality checks (completeness, accuracy, consistency, referential integrity, drift detection). - Validate batch and streaming pipelines, job dependencies, and SLAs. - Create reproducible test datasets and use Delta Live Tables (if applicable) to validate expectations. - 3) Azure Cloud QA - Test data flows across Azure Data Factory / Synapse pipelines, Databricks Jobs, Azure Storage/ADLS, Azure SQL/SQL MI, Event Hub/Kafka, and Key Vault access patterns. - Verify IAM/ACLs/Unity Catalog permissions, secrets handling, and network boundary controls. - Validate infrastructure configurations across environments (Dev/Test/Prod) and support IaC validation (e.g., Bicep/Terraform plan diffs). - 4) Automation & DevOps - Integrate tests into CI/CD (Azure DevOps/GitHub Actions): unit, integration, data-validation, and post-deployment smoke tests. - Implement quality gates (coverage thresholds, schema checks, contract tests, static checks like Pylint/flake8). - Build test utilities and shared libraries for data assertions, fixtures, and synthetic data generation. - Use feature branches, PR checks, and merge policies to enforce quality. - 5) Observability & Reliability - Instrument tests and pipelines with logging & metrics (e.g., Azure Monitor, Log Analytics, Databricks metrics). - Define SLIs/SLOs for data quality and pipeline reliability; participate in incident reviews and RCA. - Establish quality dashboards (test pass rate, DQ incidents, recovery time, flaky test rate). - 6) Collaboration & Governance - Partner with data engineers, platform engineers, product owners, and data stewards. - Contribute to test data management practices and a data contract approach with upstream/downstream teams. - Document test plans, runbooks, and traceability from requirements to tests.

- Required Qualifications - 3–8+ years in QA/Test Engineering with 2+ years in data platform or Spark-based environments. - Hands-on with Databricks: Notebooks, Jobs, Delta Lake, cluster policies. - Strong Python for tests/utilities; solid SQL for validation and profiling. - Experience testing ETL/ELT patterns, data partitioning, late-arriving data, slowly changing dimensions, idempotency. - Azure experience: Data Factory, ADLS Gen2, Key Vault, Azure DevOps/GitHub. - CI/CD for data: pipelines, artifacts, environments, approvals, test stages and post-deploy validations. - Familiar with schema validation and API testing (REST/Databricks Jobs API). - Clear understanding of security (RBAC/ABAC, managed identities), cost & performance considerations.

- Nice-to-Have - Unity Catalog, data lineage, and governance testing. - Streaming (Structured Streaming, Delta Live Tables), event-driven validation. - Performance testing of Spark jobs (cluster sizing, shuffle hotspots, skew). - Exposure to privacy & regulatory controls (PII masking, GDPR/CCPA, SOX).

About InfoBeans:

InfoBeans is a global digital transformation and product engineering company, enabling businesses to thrive through innovation, agility, and cutting-edge technology solutions. With over 1,700 team members across the globe, we specialize in custom software development, enterprise solutions, cloud, AI/ML, UX, automation, and digital transformation services.

At InfoBeans, we live by our core purpose of “Creating WOW!”—for our clients, team members, and the community. Our collaborative culture, growth opportunities, and people-first approach make us one of the most trusted and rewarding workplaces.