Lead Software Engineer - Machine Learning Platform
ARCHIVED
We can't find an active application page for this role right now. It may reopen or be listed elsewhere. Use Next Steps to search for an active apply link and similar live jobs.
Overview:
As Lead Software Engineer in AI/ML Data Platforms, you will design and operate scalable ML training systems in the cloud, focusing on GPU workloads and reproducibility. You will partner across data engineering and platform teams to standardize interfaces, governance, and cost controls while advancing enterprise-grade AI practices. Your work enables GenAI/LLM workflows, observability, and secure, reusable training patterns at scale. This role offers impact on production-grade ML infrastructure and a path to shape how the firm trains and govern AI models.
Compensation / Benefitscompetitive total rewards packagecomprehensive health care coverageretirement savings planbackup childcaretuition reimbursementmental health support
Responsibilities:
Design, build, and maintain end-to-end ML training platformRun and optimize GPU training workloads (single-node and distributed) for throughput and reproducibilityBuild and operate training infrastructure on Kubernetes (EKS and other managed platforms) Enable Gen AI/LLM training workflows with governance and scalable GPU patternsImplement observability: metrics, logs, dashboards, alerting, runbooksCollaborate with data engineering and platform teams to define interfaces, standards, guardrailsImprove developer experience for training via containers, CI/CD, templates, and self-service workflowsPromote enterprise AI-assisted engineering practices and establish validation standardsLeverage SDLC toolchain and AI-assisted development to improve automation and outcomesEnsure secure, compliant adoption and training outputs validation
Key requirements:
Formal training or certification in software engineering with 5+ years of applied experienceExperience running ML training in cloud environments and debugging infrastructure/codeStrong Python skills with solid engineering practicesExperience building automation/CI for ML codebasesHands-on experience with deep learning workflows and PyTorch or TensorFlowUnderstanding of training performance, reproducibility, and evaluation methodologiesExperience with distributed training (DDP/FSDP/DeepSpeed) and related conceptsAbility to profile and optimize training systems (CPU/GPU/memory/I/O) Experience with Kubernetes and AWS services (EKS, ECR, S3, IAM, VPC, CloudWatch, EC2) Experience leading use of AI-assisted development tools with focus on correctness, performance, and securityPythonKubernetes (EKS) and cloud infrastructureGPU-based ML training workflowsPyTorch or TensorFlowDistributed training (DDP, FSDP, DeepSpeed)CI/CD for ML (build/test/release/deployment workflows)