AI Quality & Knowledge Analyst

Responsibilities

This role combines two main functions:

• AI Quality Analyst: Review AI responses, define the expected answer, identify gaps or failure patterns, and recommend improvements to the knowledge base, instructions, or system behavior.
• Knowledge Base Editor: Write, organize, and update knowledge base content based on findings from the AI response review process.

This Is Not Primarily a Software Engineering Role. It Is More Suitable For Someone Who Is Analytical, Detail-oriented, Structured, And Able To Translate Complex Information Into Clear Rules And Documentation. Key Responsibilities:

• AI Response Review
• Review AI-generated responses to user or client questions.
• Compare AI responses against the available source documents, policies, guidelines, or expected business logic.
• Determine whether each response is correct, partially correct, incomplete, misleading, unsupported, or incorrect.
• Define the expected answer for each reviewed case.
• Identify important information that the AI missed.
• Identify answers that appear correct but are not sufficiently supported by the available knowledge base.
• Detect inconsistencies across similar AI responses.
• Review both the final answer and the reasoning or evidence used to produce it, where available.
• Root Cause Analysis

Investigate why the AI produced an incorrect or incomplete response.Requirements

• Knowledge Base Improvement
• Write and update knowledge base articles, rules, examples, and decision guidelines.
• Convert informal, incomplete, or ambiguous information into clear and explicit documentation.
• Add missing definitions, conditions, exceptions, examples, and edge cases.
• Ensure that each knowledge base entry can be understood consistently by both humans and AI systems.
• Structure information so that the AI can retrieve and apply the correct rule.
• Remove duplicated, outdated, or conflicting information.
• Clarify which rule should take priority when multiple rules apply.
• Create question-and-answer examples for situations that may be interpreted in different ways.
• Maintain version history and document the reason behind each knowledge base update.
• Quality Assurance and Testing
• Retest previously incorrect cases after the knowledge base has been updated.
• Identify high-risk questions that require stricter review or human approval.
• Escalate recurring system issues to the relevant product or engineering team.

Core Qualifications

• Strong analytical and critical-thinking skills.
• High attention to detail.
• Strong written communication skills in English and Indonesian.
• Ability to identify factual errors, logical gaps, inconsistencies, and ambiguity.
• Ability to convert complex information into clear, structured, and explicit documentation.
• Ability to understand rules, conditions, exceptions, and decision logic.
• Comfortable reviewing large amounts of text and repetitive cases.
• Able to make consistent judgments based on documented standards.
• Comfortable working with spreadsheets, documentation tools, dashboards, and internal software.
• Able to work independently while knowing when to ask for clarification.
• Able to explain why an answer is correct or incorrect, rather than relying only on intuition.

Preferred Backgrounds

Candidates may come from one or more of the following backgrounds:

• Business analyst.
• Operations analyst.
• Quality assurance analyst.
• Customer support quality analyst.
• Knowledge management specialist.
• Knowledge base writer or editor.
• Content quality analyst.
• Policy analyst.
• Compliance analyst.
• Research analyst.
• Process improvement analyst.
• Medical claims analyst.
• Insurance operations analyst.
• Legal researcher.
• Regulatory researcher.
• Auditor.
• Risk analyst.
• Fraud review analyst.
• Case review specialist.

A software engineering background is not required.

Preferred Experience

• Experience reviewing the quality of customer service, operational, policy, legal, healthcare, insurance, or financial decisions.
• Experience creating standard operating procedures, policies, decision trees, or knowledge base articles.
• Experience identifying gaps in documentation or operational processes.
• Experience creating test cases or quality evaluation frameworks.
• Experience handling information that contains multiple rules, exceptions, and approval conditions.
• Basic understanding of how AI assistants and large language models work.
• Experience with healthcare, insurance, claims, finance, legal, compliance, or other document-heavy industries.
• Experience reviewing customer conversations or support tickets.

What This Role Is Not

This role is not primarily responsible for:

• Developing machine learning models.
• Building backend systems.
• Writing production software.
• Managing cloud infrastructure.
• Training foundation models.
• Designing complex AI architecture.
• Fixing technical retrieval, database, or integration problems directly.

Technical issues will be escalated to software engineers or AI engineers. This role is responsible for identifying, documenting, and explaining the issue clearly.

Candidate Profile

The Ideal Candidate Is Someone Who:

• Enjoys investigating why something is wrong.
• Does not accept an answer simply because it sounds convincing.
• Can separate facts from assumptions.
• Can explain complex rules in simple language.
• Naturally thinks about exceptions and edge cases.
• Is comfortable making structured judgments.
• Is patient with repetitive review work.
• Can maintain consistency across many cases.
• Takes documentation quality seriously.
• Understands that small wording differences can change the meaning of a rule.