Senior Machine Learning / Data Engineer - Evaluation

Job Description Salary: DOEAbout VTI AerospaceVTI Aerospace builds AI-powered perception and pilot assist technologies to unlock the future of aviation. With offices in Bozeman, MT and Seattle, WA, we are a team of engineers and technologists from Boeing, Airbus, Aurora, and beyond. We are passionate about pushing the boundaries of autonomy and aviation safety. Our company is dedicated to advancing the field of aviation with cutting-edge solutionsThe RoleAs a Senior Software Engineer Evaluation, you will design and implement systems that measure and monitor the performance of our computer vision, automatic speech recognition (ASR), and small language model (SLM) systems. You will develop evaluation methodologies, benchmarking pipelines, and monitoring tools that ensure our AI systems perform reliably in real-world environments. You will help establish the evaluation standards and performance benchmarks that guide the development of all AI systems at the company.You will work closely with machine learning and data engineering teams to evaluate model performance, identify failure modes, and guide improvements to data collection and model training, helping bring AI-powered aviation tools to market.This role is ideal for someone who enjoys designing robust evaluation systems and uncovering insights from model performance data, and wants to have a direct impact on the quality and reliability of our AI systems.What You'll DoDefine and implement evaluation methodologies for computer vision, ASR, and language systemsIdentify and track key performance indicators (KPIs) that measure system and model effectivenessPerform dataset coverage analysis to understand strengths, gaps, and biases in training and evaluation dataIdentify model deficiencies and collaborate with ML engineers to improve training data and model performanceBuild scalable model evaluation pipelines in Python for automated benchmarking and regression testingDesign and maintain systems for monitoring model performance and drift in production environmentsArchitect data models and storage systems for evaluation results using relational and time-series databasesBuild dashboards and reporting tools to visualize model performance and evaluation metricsCollaborate with ML and data engineering teams to improve evaluation workflows and data pipelinesContribute to ground-truth data pipelines and processes that support reliable evaluationWhat We're Looking For5+ years of experience developing production software systems in PythonStrong experience designing and implementing data processing or analysis pipelinesExperience building systems for machine learning evaluation, experimentation, or benchmarkingExperience working with large datasets and building scalable data workflowsFamiliarity with statistical methods, experimental design, and model performance analysisExperience building automated pipelines using tools such as Airflow or similar orchestration frameworksExperience working with evaluation or experiment tracking tools such as MLflow or similar systemsExperience designing data models and working with relational databases and time-series dataStrong collaboration skills and ability to work closely with machine learning and data engineering teamsBonus PointsExperience working with computer vision, ASR, or language models in production environmentsExperience using PyTorch or other deep learning frameworksExperience evaluating multimodal or large-scale AI systemsExperience building monitoring systems for ML models in productionExperience designing dataset analysis or data quality toolingExperience building dashboards or monitoring tools using Grafana or similar platformsWhy You'll Love Working HereOpportunity to work on meaningful problems in aviation and deliver products to industry at a rapid paceSmall team with significant impact on company directionCollaborative and innovative environmentCompetitive compensation and benefitsOpportunities for growth and leadershipLocationWork arrangement: On-sitePrimary location: Seattle or BozemanCompensation & BenefitsSalary range: DOEBenefits may include:Health, dental, and vision insuranceRetirement plan or 401(k)Paid time off and holidaysProfessional development opportunitiesEqual Opportunity StatementVTI is proud to be an equal opportunity employer. We value diverse perspectives and encourage candidates from all backgrounds to apply