AI Rubric Writer

Role Title: AI Rubric Writer

Role Type: Contractor

Location: Remote

We are seeking an experienced AI Rubric Writer to develop high-quality evaluation rubrics and scoring frameworks for AI and Large Language Model (LLM) assessments. The ideal candidate will create clear, objective evaluation criteria to assess AI-generated responses and help improve model performance across a variety of subject areas.

Responsibilities

1. Design and develop evaluation rubrics and scoring guidelines for AI-generated content.
2. Create objective, consistent, and well-structured assessment criteria.
3. Evaluate AI responses for accuracy, reasoning, completeness, and quality.
4. Provide detailed feedback to improve AI model performance.
5. Collaborate with cross-functional teams to refine evaluation standards.
6. Ensure rubrics are clear, unbiased, and easy to apply.

Required Qualifications

1. Bachelor's degree or higher in a relevant field.
2. Strong analytical, critical thinking, and written communication skills.
3. Experience creating evaluation criteria, scoring frameworks, assessment guidelines, or rubrics.
4. Subject matter expertise in Software Engineering, Finance, Law, Consulting, or other quantitative/STEM disciplines.

Preferred Qualifications

1. Experience as an AI Evaluator, Prompt Evaluator, or similar AI quality assessment role.
2. Experience evaluating Large Language Models (LLMs) or Generative AI systems.
3. Familiarity with rubric development, AI quality assurance, or data annotation.
4. Experience reviewing and assessing technical or domain-specific content.

Originally posted on Himalayas