Data Annotation Engineer

Role: Data Annotation Engineer

Location - Louisville, Kentucky ( Day 1 onsite)

Job Description:

We are looking for an Annotation Program Lead to manage our AI Quality Annotation Program. This individual will oversee the operational execution of human review activities that establish the gold-standard datasets used to measure conversational AI performance, safety, and compliance.

The ideal candidate combines strong program management skills with experience in quality assurance, data annotation operations, and stakeholder engagement.

• Lead the day-to-day operation of the AI annotation program.
• Manage a team of annotators responsible for reviewing customer conversations and AI interactions.
• Develop annotation guidelines, procedures, and quality standards.
• Establish calibration programs and quality assurance processes to ensure consistency across reviewers.
• Partner with Data Scientists to create and maintain gold-standard datasets.
• Monitor annotation accuracy, throughput, and quality metrics.
• Coordinate reporting and deliverables for Legal, Compliance, Responsible AI, and executive stakeholders.
• Manage annotation workflows and tooling including setting up annotation jobs, running data processing scripts, and testing annotation UIs for quality before job launch.
• Plan capacity requirements to support seasonal increases in review volume.
• Support future expansion into multilingual evaluation programs.

Identify process improvements that increase efficiency and annotation quality.

• Bachelor's degree or equivalent experience in Linguistics/Psychology.
• 5+ years of experience in program management, operations, quality assurance, data labeling, or related fields.
• Experience leading teams and managing operational workflows.
• Strong organizational and stakeholder management skills.
• Ability to analyze quality metrics and drive continuous improvement initiatives.
• Excellent written and verbal communication skills.
• Familiarity with Python/R, SQL and Excel. Should be familiar with running scripts on language data.
• Strong organizational and stakeholder management skills.
• Ability to analyze quality metrics and drive continuous improvement initiatives.
• Excellent written and verbal communication skills.
• Familiarity with Python/R, SQL and Excel. Should be familiar with running scripts on language data.
• Experience supporting AI, machine learning, or data annotation programs.
• Familiarity with Responsible AI, compliance, legal review, or governance processes.
• Experience developing operational standards and quality frameworks.
• Experience managing vendor or contractor resources.