Annotation Specialist, LearnWith.AI (Remote) - $30,000/year USD

This role is designed for individuals who value correctness over speed. The annotations you produce serve as training data for AI systems used daily by thousands of students. Accurate labeling strengthens the product; inconsistent labels teach the model incorrect patterns.LearnWith.AI develops AI-driven learning experiences grounded in learning science, data analytics, and expert knowledge. Your primary function will be to convert raw student session videos into reliable, rubric-aligned labels the team depends on. You will review recorded student sessions, pinpoint critical behavioral events, and apply precise classification rules to capture what occurred and when. Additionally, you will assess LLM-generated pre-annotations, correct errors, and record edge cases to help engineers refine the underlying system.This is not freelance, ad-hoc annotation work. It involves a continuous queue within one product area, supported by direct feedback, calibration against gold-standard datasets, and advancement tied to accuracy and consistency. If you value transparent expectations, quantifiable quality standards, and work that directly influences model outcomes, we would like to hear from you.What You Will Be DoingLabel student session videos by detecting, categorizing, and timestamping behavioral events according to a comprehensive rubricEvaluate and correct pre-annotations generated by LLMs, removing incorrect labels, adding overlooked events, and refining timestamp accuracyDocument reasoning for ambiguous decisions, referencing rubric guidelines and the logic behind your choicesRecord edge cases and questions about unclear scenarios, and maintain an annotation log with session metadataParticipate in calibration sessions, incorporate QA feedback, and adapt to rubric revisions to enhance accuracy consistentlyWhat You Won’t Be DoingDevelop AI models, conduct experiments, or perform research into student behavior patternsModify the annotation rubric or reinterpret category definitions based on subjective judgmentPrioritize speed over accuracy, consistency, or precise timestampingHandle sporadic, disconnected tasks across unrelated fields without context or quality feedbackAnnotation Specialist Key ResponsibilitiesYour role ensures that student session videos are transformed into labeled datasets with ≥95% accuracy and time-precise annotations, providing a reliable foundation for measuring model performance improvements or regressions.Basic RequirementsA minimum of 1 year of experience in data annotation, content moderation, QA evaluation, or comparable rubric-based review rolesExcellent English reading comprehension with the capacity to adhere to detailed written instructions without deviationCapacity to maintain focus and precision during 4–6 hours of video-based annotation work dailySkill in identifying subtle on-screen and visual behavioral signals and applying consistent classification across multiple sessionsProficient written communication for documenting edge cases, reasoning, and clarification requestsDependable internet connection suitable for video streamingExperience reviewing, correcting, and enhancing AI/LLM-produced annotationsAbout LearnWith.AILearnWith.AI is an edtech startup that leverages AI and subject matter experts to cultivate a new way of learning. Our unique approach leverages 50+ years of learning science, cutting-edge data analytics and AI-powered coaching. In doing so, we can help students learn more, learn faster, and learn better - and have fun while doing it. We are a remote-first company that hires globally via Crossover.There is so much to cover for this exciting role, and space here is limited. Hit the Apply button if you found this interesting and want to learn more. We look forward to meeting you!Working with usThis is a full-time (40 hours per week), long-term position. The position is immediately available and requires entering into an independent contractor agreement with Crossover as a Contractor of Record. The compensation level for this role is $15 USD/hour, which equates to $30,000 USD/year assuming 40 hours per week and 50 weeks per year. The payment period is weekly. Consult www.crossover.com/help-and-faqs for more details on this topic.Crossover Job Code: LJ-5785-US-Austin-AnnotationSpec