Remote AI/ML Data Annotation Engineer
ARCHIVED
We can't find an active application page for this role right now. It may reopen or be listed elsewhere. Use Next Steps to search for an active apply link and similar live jobs.
About Rex.zone Rex.zone is a recruiting and talent solutions platform supporting modern AI/ML training workflows across the United States. This remote, full-time role focuses on building scalable annotation and evaluation programs that improve training data quality and model performance in production LLM pipelines.
Role Overview:
Build and maintain high-quality annotation workflows for AI/ML, including RLHF-style preference data, prompt evaluation, and QA evaluation loops. You will design labeling schemas and write clear annotation guidelines to ensure consistent, measurable outcomes.
Responsibilities:
- Own training data quality metrics, error taxonomies, and continuous improvement plans
- Design labeling schemas and write/update annotation guidelines and ambiguity-resolution rules
- Implement QA evaluation programs (review queues, spot checks, gold tasks, inter-annotator agreement)
- Perform RLHF preference labeling and ranking to support alignment and helpfulness
- Run prompt evaluation and regression test sets to track model performance improvement
- Support NLP tasks such as named entity recognition and intent classification
- Support computer vision annotation (e.g., bounding boxes, segmentation) when needed
- Execute content safety labeling for policy-aligned model behavior
- Partner with engineering to integrate tools and datasets into LLM training pipelines
Qualifications:
- 3+ years in data operations, ML ops, evaluation, or annotation engineering
- Strong understanding of NLP, LLM evaluation, and human feedback signals (RLHF)
- Experience with QA evaluation methods, sampling strategies, and disagreement analysis
- Familiarity with annotation tools and workflow automation
- Excellent technical writing for guidelines and repeatable processes
- CompensationCompetitive hourly base pay: $30–$50/hr.