AI Rater Guidelines Writer, Linguist, Instructional Designer

📊 Check your resume score for this job

Improve your chances of getting an interview by checking your resume score before you apply.

Upload Resume

📋 Description

• Translate complex and ambiguous program requirements into clear, concise, and easy-to-follow rater guidelines.
• Design comprehensive evaluation rubrics and scoring frameworks that enable consistent decision-making across a variety of AI tasks.
• Review existing documentation to identify ambiguity, inconsistencies, contradictions, and coverage gaps, then revise guidelines to improve clarity and usability.
• Convert subject-matter requirements from domains such as finance, retail, insurance, legal, and sports into structured evaluation instructions that non-domain raters can apply confidently.
• Collaborate with research teams, program managers, and subject matter experts to maintain consistency across multiple guideline sets.
• Develop documentation that addresses edge cases, exceptions, and complex evaluation scenarios while minimizing reviewer escalation.

🎯 Requirements

• Minimum 3 years of professional experience in Linguistics, Instructional Design, Technical Writing, AI Content Development, or a closely related field.
• Direct experience creating, refining, or maintaining evaluation guidelines, rating rubrics, or reviewer instructions within Generative AI, RLHF, human evaluation, or AI data annotation environments.
• Demonstrated ability to work across multiple subject areas and convert domain-specific knowledge into clear, structured documentation.
• Proven experience resolving ambiguity and improving written specifications with measurable before-and-after improvements.
• Strong analytical thinking with exceptional attention to detail.
• Demonstrated career progression and professional growth.
• Ability to commit reliably to 35+ hours per week during weekdays.
• Outstanding written communication skills with the ability to explain nuanced concepts in a precise, consistent, and easy-to-understand manner.

Apply Now