Junior Software Engineer

Junior Software Engineer📍 Location: San Francisco, CA, 🏢 Industry: Technology, Information and Internet💼 Work Setting: HybridAre you a creative problem solver who enjoys software engineering challenges, working with AI systems, and designing rigorous technical evaluations?This role focuses on building and improving Reinforcement Learning (RL) evaluation tasks used to measure and improve frontier AI coding models. You'll design software engineering challenges, create grading infrastructure, analyze AI model behavior, and continuously refine tasks to ensure they accurately assess real-world coding capabilities.The ideal candidate combines strong Python programming skills, software engineering fundamentals, analytical thinking, and curiosity about AI systems with the ability to work independently and think creatively.Key ResponsibilitiesRL Task Design & DevelopmentDesign self-contained software engineering challenges.Develop realistic coding tasks that evaluate model capabilities.Create prompts that test reasoning, coding, debugging, and problem-solving skills.Design tasks that are challenging, fair, and measurable.Continuously improve evaluation quality.Areas of OwnershipTask DesignPrompt EngineeringEvaluation DevelopmentChallenge CreationBenchmark DesignAutomated Grading InfrastructureBuild automated grading systems for evaluating solutions.Develop scoring mechanisms that measure correctness and quality.Ensure tasks are objective and consistently evaluated.Validate grading reliability and accuracy.Improve testing and evaluation workflows.ResponsibilitiesTest DevelopmentAutomated ValidationScoring SystemsQuality AssuranceEvaluation FrameworksAI Model EvaluationRun frontier AI models against created tasks.Analyze model outputs and performance patterns.Identify strengths, weaknesses, and failure modes.Investigate where and why models make mistakes.Measure capability gaps across different scenarios.Focus AreasModel EvaluationFailure AnalysisPerformance AssessmentCapability TestingBenchmarkingAI Behavior AnalysisAnticipate how models interpret prompts.Identify situations where models may take shortcuts.Distinguish true capability limitations from evaluation flaws.Understand subtle model behaviors and edge cases.Improve task robustness based on observed outcomes.Key ActivitiesBehavioral AnalysisPrompt InterpretationError InvestigationEdge Case IdentificationEvaluation RefinementWorking with Coding AgentsLeverage AI coding agents extensively throughout development.Direct AI tools effectively to accelerate task creation.Review, validate, and improve AI-generated solutions.Identify cases where AI output appears correct but contains hidden flaws.Use AI systems as collaborative engineering tools.Experimentation & IterationContinuously refine tasks using evaluation results.Improve fairness, difficulty, and realism.Develop new ideas based on emerging AI capabilities.Adapt evaluation approaches as models evolve.Contribute to cutting-edge AI assessment methodologies.QualificationsEducationPreferredBachelor's Degree completed or in progress in:Computer ScienceSoftware EngineeringMathematicsEngineeringRelated Technical FieldStrong ConsiderationRecent graduatesStudents approaching graduationTechnical SkillsProgrammingRequiredPythonResponsibilitiesTask DevelopmentAutomationTestingEvaluation InfrastructureScriptingSoftware Engineering FundamentalsRequiredProblem SolvingDebuggingTestingCode ReviewSystem Design FundamentalsAnalytical ThinkingRequiredLogical ReasoningRoot Cause AnalysisPattern RecognitionTechnical InvestigationCritical ThinkingPreferred AttributesAI & Technology InterestCuriosity about AI systems.Interest in frontier technology.Experience using coding assistants or AI tools.Understanding of how generative AI behaves in development environments.NotePrior AI or Machine Learning experience is not required.Professional CompetenciesIndependent Working StyleCreativityTechnical CuriosityAttention to DetailProblem SolvingInnovationAdaptabilityExperimentation MindsetCommunication SkillsCore CompetenciesPythonSoftware EngineeringTest DevelopmentAutomationAI EvaluationProblem SolvingQuality AssuranceBenchmark DevelopmentPrompt EngineeringTechnical AnalysisDebuggingCoding AgentsExperimental DesignCritical Thinking