Operations Eng/Annotation Lead

Operations Eng/Annotation LeadPosition Details:Location: San Francisco, CA 94107 (Onsite)Type: Contract$40-50w2Communicate with vendorsPossibly build dashboardCommunication between engineers and annotation team internally and externally. SO the accuracy of the annotation is important.Responsibilities1. Annotation LeadOwn the end-to-end quality and coordination of annotation workflows across concurrent streams.Ensure annotation outputs consistently meet the quality bar; identify and resolve systematic error patterns (label inconsistencies, missed edge cases, boundary errors)Serve as the primary interface between ML engineers and the external annotation vendor — ensuring timely and crisp communication on annotation tasks including clarifications, guideline alignment, and QA feedbackWork with engineers to establish and maintain annotation guidelines, protocols, and gold-set standards; verify ground truth sets against engineer intent before large-scale annotation beginsIdentify and implement pre-annotation tooling or workflow improvements that increase annotator efficiency and reduce error ratesTrack inter-annotator agreement, QA pass rates, and rejection patterns over time; use findings to drive continuous guideline improvement2. System Performance MonitoringDevelop a deep understanding of where and how the system struggles, and keep engineering informed.Build intuition on hard corner cases and edge conditions where the system fails — through ongoing review of annotation outputs, rejection patterns, and model validation resultsCommunicate regularly with engineering to surface failure patterns, flag emerging issues, and play an active role in driving identified issues to resolutionTrack pipeline health metrics by building and maintaining whatever tooling is appropriate (SQL queries, scripts, spreadsheets, or lightweight dashboards)What We're Looking ForAble to pick up and use tools such as SQL, Python, dashboard platform to pull, summarize, and track metrics.Strong analytical eye — able to spot quality issues and failure patterns in structured label dataClear, crisp communicator; comfortable as the connective tissue between engineering and an external vendorOrganized and self-directed; able to maintain tracking systems and keep multiple workstreams moving without close oversightWork in the US business hoursRequirements:A Bachelor's Degree with 3-4 years of experience or a Master's Degree with 2 years' experience in Computer Science, Electrical Engineering, or a related field.Core Skills: General Software Engineering skills with 3+ years of programming experience in python and the surrounding tooling ecosystem along with familiarity in linux and expertise in infrastructure, cloud and/or MLOps,.Personal Attributes: Team player, good communication skills, self starter.Strong teamwork and communication skills to collaborate with cross-functional teams, including ML and software engineers.Nice to Have: Experience building MLOps pipelines for deep learning based perception solutions on AWS or GCPEEO Statement:Blackstone Talent Group is a division of Blackstone Technology Group, a global IT services and solutions firm that implements technological solutions across commercial industry verticals and the US Federal Government. Blackstone's global talent augmentation practice was founded in 1998. Blackstone Talent Group has offices in San Francisco, Denver, Houston, Colorado Springs, and Washington, DC. We specialize in providing clients the best talent across a variety of industries and sectors.