Game Engineer
Own the AI safety layer for Realtor.com's RealAssist, an LLM-powered real-estate assistant used across web and mobile. On the AI Integrations Team at News Corp, you will build and tune LLM safety guardrails that act as a launch gate for Fair Housing compliance and content moderation, with runtime prompt-injection and jailbreak screening designed for production.
What you'll do:
- Build, tune, and take end-to-end ownership of LLM-as-a-judge classifiers for Fair-Housing compliance and content moderation, targeting high recall on disallowed content without overblocking legitimate users
- Design and run guardrail evaluation pipelines, including labeled dataset curation, train/test/validate splits, and offline prod-replay to evaluate overblocking
- Use defensible metrics by reporting confusion matrices and precision/recall/F1 per category to guide safety decisions
- Integrate and operate cloud guardrail services as a runtime prompt-injection/jailbreak screening layer, including fail-open behavior and alerting
- Red-team the assistant and wire safety regression checks into CI to prevent silent guardrail degradation as the product grows
- Manage guardrail infrastructure as code using Terraform (templates, IAM, and project shape), and participate in technical design reviews and architecture discussions
- Partner with ML, backend, product, and legal/compliance stakeholders to define risk tiering and get new AI capabilities through safety review before launch
- Independently design, implement, and tune guardrail and evaluation systems with a focus on balancing safety, latency, and user experience; write clean, maintainable, well-tested code
- Keep eval datasets calibrated to real traffic and emerging attack patterns, staying current with AI-safety research and new guardrail tooling
- What you bringAI Safety & Evaluation Expertise (required)4+ years of professional software/ML engineering experience with hands-on LLM application work
- Bachelor's degree or equivalent experience
- Strong Python proficiency
- Real classification-metrics literacy: precision/recall/F1 (macro vs weighted), confusion matrix debugging, dataset curation, and calibration to production distribution
- Experience evaluating LLM prompts/classifiers with an eval framework such as DeepEval, RAGAS, Arize Phoenix, LangSmith, OpenAI Evals, or a homegrown harness
- Ability to turn small seed sets into robust labeled eval datasets using dataset-synthesis and prompt-optimization loops
- Familiarity with prompt-injection/jailbreak defense concepts, including OWASP LLM Top 10, input/output filtering, least-privilege tool access, and adversarial testing
- Hands-on experience with cloud guardrail services: Google Cloud Model ArmorAWS Bedrock Guardrails, or Azure AI Content Safety
- Terraform/IaC for cloud infrastructure
- Experience with Google Cloud Vertex AI / Gemini
- Understanding of monitoring and observability tools (for example, New Relic or similar)
- Exposure to regulated, compliance-sensitive domains (fair housing, fair lending, healthcare, finance, trust & safety)
- Red-teaming or AI-security background
- Tools you'll work with
- PythonDeepEvalRAGASArize Phoenix
- LangSmithOpenAI EvalsOWASP LLM Top 10 Google Cloud Model ArmorAWS Bedrock Guardrails
- Azure AI Content Safety
- TerraformGoogle Cloud Vertex AI / GeminiNew RelicOnsite role with strong support and collaboration
- The AI Integrations Team is a cross-functional squad building how Realtor.com integrates AI across its products. The work blends creativity and innovation with in-person collaboration. For most roles, employees work four or more days in office
Benefits:
- Inclusive and competitive medical, Rx, dental, and vision coverage
- Family forming benefits13 Paid Holidays
- Flexible Time Off8 hours of paid volunteer time off
- Immediate eligibility into the Company 401(k) plan with 3.5% company match
- Tuition Reimbursement for degreed and non-degreed programs1:1 personalized Financial Planning Sessions
- Free snacks and refreshments in each office location
- Student Debt Retirement Savings Match program