Technology System/Platform Engineer (contract)
Job Summary:
We are seeking a seasoned AI Operations professional to serve as the long-term operational owner of enterprise AI-powered solutions following implementation and hypercare. This role carries full accountability for the ongoing performance, quality, adoption, governance monitoring, and continuous improvement of production AI agents. Operating within a collaborative, highly matrixed Customer Operations environment, this individual will work across Product, Operations, Technology, Security, Governance, and AI engineering teams to ensure that deployed AI solutions deliver accurate, relevant, trusted, compliant, and timely outcomes. This is not an agent development role — it is a structured, evidence-driven operational position focused on translating production insights into measurable improvements and sustained business value.
Responsibilities:
Serve as the operational owner for production AI agents, maintaining accountability for performance, quality, adoption, governance compliance, and continuous improvement after deployment and hypercare. Establish and operate comprehensive performance monitoring covering answer accuracy, containment, escalation rates, satisfaction, response quality, retrieval effectiveness, search relevance, adoption, task completion, latency, and responsiveness. Execute AI quality-management activities including transcript and conversation review, search-result evaluation, knowledge-source validation, response grading, and evaluation scorecards. Identify and analyze hallucinations, incorrect responses, missing knowledge, misrouted interactions, low-confidence responses, retrieval failures, prompt failures, workflow breakdowns, misinformation, and unexpected escalation patterns. Own Knowledge Hub performance by monitoring search quality, identifying content gaps, assessing source effectiveness, reviewing unsuccessful searches, improving retrieval strategies, coordinating with SMEs, and managing knowledge freshness. Monitor compliance with enterprise AI standards including guardrails, prompt compliance, Responsible AI assessments, bias and fairness evaluation, privacy validation, audit support, and governance metrics. Investigate production AI incidents, coordinate root-cause resolution across business and technical teams, and convert findings into actionable backlog items. Partner with AI engineering teams to improve prompts, retrieval strategies, orchestration workflows, agent behavior, and overall user experience. Track value realization through productivity improvement, self-service adoption, reduction in manual work, customer-experience improvement, agent utilization, and ROI metrics; provide recommendations to Product Owners and leadership. Deliver production AI performance scorecards, recurring operational health reports, governance monitoring evidence, and value-realization reporting. Maintain prioritized continuous-improvement backlogs with owners, recommendations, and measurable expected outcomes. Develop operating procedures and knowledge-transfer documentation sufficient to reduce organizational dependency on the contractor by assignment end. SkillsDeep understanding of production AI operations supporting conversational AI agents, virtual assistants, or generative AI solutions post-deployment. Proficiency in AI quality evaluation techniques including transcript review, response grading, hallucination analysis, retrieval and search evaluation, and trend analysis. Operational analytics capability using dashboards and performance measures such as accuracy, containment, escalation, satisfaction, adoption, task completion, retrieval effectiveness, and latency. Knowledge and retrieval operations expertise including source validation, content-gap analysis, failed-search review, freshness management, and SME collaboration. Strong understanding of Responsible AI principles, privacy requirements, guardrails, bias and fairness assessment, audit evidence, and enterprise governance practices. Incident investigation and root-cause analysis skills with the ability to translate findings into engineering or product backlog items. Ability to work independently in a matrixed environment, owning analysis and follow-through while coordinating with domain owners and engineering teams. Strong written and verbal communication skills with the ability to prepare concise operational, governance, and value updates for Product Owners and leadership. Familiarity with AI observability, evaluation tooling, conversation analytics, and work-management platforms such as Jira. Experience with Microsoft Copilot Studio, Salesforce Agentforce, conversational AI platforms, knowledge-management systems, or comparable enterprise AI platforms is preferred. Background in a regulated, utility, privacy-sensitive, or audit-sensitive environment is preferred.
Experience:
Demonstrated experience in a production AI operations role supporting AI agents, conversational AI, or generative AI solutions after deployment.
Hands-on experience conducting AI quality reviews including transcript sampling, response grading, hallucination identification, and scorecard development.
Proven experience operating within customer-service, contact-center, or Customer Operations environments is strongly preferred.
Experience managing knowledge-base or retrieval system performance including gap analysis, source validation, and freshness oversight.
Experience producing governance monitoring evidence, audit support artifacts, and Responsible AI compliance documentation.
Qualifications:
Relevant Microsoft, cloud, AI, IT service management, data, or product-operations certifications are beneficial but are not a substitute for demonstrated production AI operations experience. The closing date for applications for this job posting is 10/21/2026.Should you require any accommodations in responding to this job posting or at any stage during the application process, kindly e-mail : xcelenergyprogramoffice@magnitglobal.comPay Rate Range73.85 - 98.46 USD hourly