AI QA Engineer role at Konstant Infosolutions in Jaipur, Rajasthan. The employer's own role page states Location: Jaipur and shows a posting date of 27 Oct 2026; it was listed as open when we checked on 1 Nov 2026.

About Konstant Infosolutions

Konstant Infosolutions is a long-running Jaipur IT services company - its careers page cites 23+ years in business, 2,500+ clients and a 150+ person team, and it is a certified Great Place to Work. The company now markets itself heavily around AI engineering work.

The role in plain terms

This is quality engineering aimed at AI products: testing software whose behavior includes model outputs, which adds evaluation and edge-case work on top of normal QA.

What the employer asks for

These duties and requirements are quoted or closely condensed from the employer's listing so the details are not distorted:

Duties:

Design test strategies for AI applications including LLMs, RAG systems, conversational AI, and agentic workflows
Build automated evaluation pipelines for accuracy, groundedness, context relevance, reasoning, tool-use correctness, and latency
Create and maintain golden datasets, test scenarios, and evaluation metrics for regression testing
Implement LLM-as-a-Judge, schema validation, tracing, and calibrated scoring for multi-turn and long-horizon tasks
Test for hallucinations, bias, harmful content, policy violations, PII leaks, and brand drift using filters and guardrails
Conduct red-team testing for prompt injection, jailbreaking, data exfiltration, and adversarial inputs
Embed evaluation harnesses into CI/CD pipelines and monitor production telemetry for behavioral drift and failures
Analyze model and integration failures to identify root causes, controls, and mitigations
Translate business risks into measurable quality criteria, thresholds, and release recommendations
Collaborate with engineers, product managers, and security teams on testability, observability, and governance

Skills and experience:

3+ years of experience in QA engineering, test automation, ML QA, AI evaluation, or MLOps
Strong proficiency in Python and test frameworks such as Pytest, with experience building automated test suites
Hands-on experience testing LLM applications, RAG pipelines, or AI agents in production or pre-production environments
Understanding of AI failure modes: hallucinations, grounding errors, prompt sensitivity, context limits, and tool misuse
Familiarity with AI evaluation tools and platforms such as MLflow, Galileo, PyRIT, LangSmith, or equivalent
Knowledge of security testing concepts including prompt injection, jailbreaking, and data leakage risks
Experience working with CI/CD, containers, cloud environments, and observability tools
Strong analytical skills to interpret evaluation results and communicate risks clearly
Experience with multi-agent systems, autonomous workflows, or long-horizon task evaluation
Knowledge of responsible AI, fairness testing, and regulatory compliance for AI systems
Background in software development, data engineering, or ML engineering
Contributions to AI testing frameworks, open-source evaluation tools, or internal QA standards

Experience required: 3+ Years.

Who this role suits

This suits a QA engineer with 3+ years who wants to specialize; expect questions on how you test non-deterministic behavior, not only standard test-case design.

Practical notes

No salary is stated on the listing.

How to apply

Apply on the company's own role page: AI QA Engineer at Konstant Infosolutions. Before applying, re-read the employer's own description in full, since eligibility details, deadlines and process steps can change at any time and the employer's page is always the final word. Tailor your resume to the specific requirements above rather than sending a generic one: mirror the skills and tools the listing names where you genuinely have them. Never pay a fee to apply or interview; a genuine hiring process does not charge candidates.

Related openings on GetFirstJob

We write and maintain this page from the employer's public listing and link straight to their application page. We do not act as a recruiter, and applying is always free. If the link above stops working, check the employer's careers page for the current status of this role.

This is our short summary of the employer's listing. The full description, eligibility and deadline are on the employer's page.