Automation QA Engineer – Data Validation & Quality and AI/ML

Company Description ProfoundIQ offers CTO-as-a-Service, providing senior technology leadership to organizations where technology is fundamental for growth, operations, and long-term success. Our expertise includes guiding sound technology decisions, aligning engineering efforts with business objectives, and building scalable systems to adapt to increasing complexity. Beyond consulting, we take ownership of strategic direction, execution, and results, embedding ourselves deeply as technology leaders. Our business-focused, outcome-driven approach reduces risks, enhances decision-making, and creates robust technology foundations to support sustained growth and transformation.

Role Description This is a full-time remote role for an Automation QA Engineer – Data Validation Data Quality and AI/ML with 4-7 years experience for one of our clients.

Key Responsibilities Validate determinism and reproducibility in systems where identical inputs yield varying outputs, identifying causes like non-deterministic algorithms or environmental factors. Assess gold datasets and AI-generated content for quality, scalability, and reliability before use in testing. Develop automation to detect duplicates and near-duplicates in large text datasets using techniques like hashing or embedding similarity. QA automated scoring/evaluation functions to ensure pass/fail accuracy, including edge case verification. Test tool/API contracts for correct formatting and prevent silent failures in agent-based systems. Write stable UI tests with tools like Playwright, focusing on robust selectors and waits; structure API tests using Postman/Newman or pytest/requests for maintainability.

Required Skills and Experience Proven QA automation experience with Cypress or Selenium (mandatory as per request). Strong background in data validation and quality testing for AI/ML systems, including dataset validation and evaluation metrics. Proficiency in Playwright for UI testing, API testing frameworks (Postman, pytest), and handling non-determinism in RL environments or similar. Ability to scale tests for large datasets, with knowledge of duplicate detection and scoring system QA.

Nice-to-Haves Experience with RL environments, AI-generated data, or tool-using agents.

Work Hours US Eastern time (7.30PM to - 4.30AM IST) - 5 days a week. Remote