Remote AI/ML Data Annotation Engineer

About Rex.zoneRex.zone is a recruiting and talent solutions platform supporting modern AI/ML training workflows across the United States. This remote, full-time role focuses on building scalable annotation and evaluation programs that improve training data quality and model performance in production LLM pipelines.Role OverviewBuild and maintain high-quality annotation workflows for AI/ML, including RLHF-style preference data, prompt evaluation, and QA evaluation loops. You will design labeling schemas and write clear annotation guidelines to ensure consistent, measurable outcomes.ResponsibilitiesOwn training data quality metrics, error taxonomies, and continuous improvement plansDesign labeling schemas and write/update annotation guidelines and ambiguity-resolution rulesImplement QA evaluation programs (review queues, spot checks, gold tasks, inter-annotator agreement)Perform RLHF preference labeling and ranking to support alignment and helpfulnessRun prompt evaluation and regression test sets to track model performance improvementSupport NLP tasks such as named entity recognition and intent classificationSupport computer vision annotation (e.g., bounding boxes, segmentation) when neededExecute content safety labeling for policy-aligned model behaviorPartner with engineering to integrate tools and datasets into LLM training pipelinesQualifications3+ years in data operations, ML ops, evaluation, or annotation engineeringStrong understanding of NLP, LLM evaluation, and human feedback signals (RLHF)Experience with QA evaluation methods, sampling strategies, and disagreement analysisFamiliarity with annotation tools and workflow automationExcellent technical writing for guidelines and repeatable processesCompensationCompetitive hourly base pay: $30–$50/hr.