Data Science - Remote
Are you an experienced data science professional with expertise in LLMs, data analytics, and large-scale analytical pipelines? We’re looking for a Data Science & Analytic Methodology Subject Matter Expert (SME) to help evaluate and improve next-generation AI systems through advanced analytical scenarios, methodology assessments, and high-quality technical feedback.
Key Responsibilities:
- Evaluate analytical pipelines and map scenarios to established evaluation frameworks.
- Develop realistic analytical cases involving automation, data enrichment, and large-scale analysis.
- Assess LLM responses for reasoning quality, source accuracy, limitations, and practical usefulness.
- Identify model over-refusals and potential failures affecting analytical workflows.
- Document complex edge cases and recommend improvements to evaluation methodologies.
- Provide clear, actionable written and verbal feedback to improve AI performance.
- Required SkillsData Science and Data Analytics
- Large Language Models (LLMs)
- Pipeline Integration
- Scaled-Analysis Judgment
- Analytical Reasoning and Problem-Solving
- Technical Documentation and Written Communication
Preferred Qualifications:
Experience integrating LLMs into complex data or analytical workflows.
Strong understanding of large-scale analysis, automation, and evaluation metrics.
Background in data science leadership, teaching, or mentoring.
Experience with evaluation frameworks, benchmarks, or analytical methodology.
Familiarity with government, defense, intelligence, or public-sector analytics is an advantage.