Ai Evaluation Engineer

Description:

We are looking for a talented AI Evaluation Engineer / AI SDET to help assess the quality, accuracy, and reliability of AI-powered applications, including legal and business AI workflows.

Required Skills:
5+ years of software development experience using Python, Java, or C#
Foundational understanding of LLMs, AI capabilities, limitations, and failure modes
Experience with automated testing, API testing, and test automation
Knowledge of AI evaluation, test datasets, regression testing, and quality assessment
Understanding of precision, recall, sampling, and statistical measurement
Experience with Git, CI/CD, Jira, Jenkins, and SDLC practices
Ability to use coding agents effectively for development, testing, and analysis
Strong analytical, debugging, documentation, and problem-solving skills

Key Responsibilities:
Develop and execute automated evaluations for AI applications
Prepare and maintain evaluation datasets and test cases
Assess AI output accuracy, completeness, citations, and retrieval relevance
Perform functional and integration testing of AI workflows
Analyze AI failures and investigate model, retrieval, data, and application issues
Compare evaluation results across model, prompt, and application versions
Support grader calibration and regression testing
Integrate evaluation suites into CI/CD pipelines

Preferred Qualifications:
Practical experience evaluating AI applications or LLM-based systems
Experience with AI testing frameworks, LLM evaluation, or AI quality assurance
Academic, internship, or project experience in AI evaluation is also welcome