{"schemaVersion":"jobsearcher.job.v1","id":"de67f0def22cf00eb2637ec9","url":"https://jobsearcher.com/jobs/de67f0def22cf00eb2637ec9","canonicalUrl":"https://jobsearcher.com/jobs/de67f0def22cf00eb2637ec9","title":"Technical Product Manager, Data Science","description":"CENTRL is a leading risk and compliance technology company that provides AI powered enterprise-grade risk, due diligence, cyber security and privacy management solutions to financial institutions worldwide. Our clients include some of the largest banks and investment management firms across the Americas, Europe and APAC. Headquartered in Silicon Valley, CENTRL has regional offices in New York, India, Australia, and the United Kingdom. Established in 2015, CENTRL is a high-growth, venture backed SaaS firm leading the way in innovative generative AI solutions to manage third party risk and due diligence.\r\nPosition Overview:We are looking for an owner for the accuracy, reliability, and cost-efficiency of the AI behind CentrlX, our agentic platform for Manager Research and Investor Relations teams at investment firms.\r\nWe have a strong engineering team building the platform: agents, skills, knowledge, automations, connectors, and governance. We are looking for a single owner for output accuracy and efficiency.\r\nYou will own the model and prompt layer end to end. You will build the evaluation datasets and harnesses that tell us whether a change actually helped, run structured comparisons across model providers on accuracy, latency, and cost, and drive the resulting changes into the product. You will help create and design agents and skills, and work closely with sales and PS in order to help drive standards for prompting and agents, and then feed that back into the product design. You will do a real share of this work yourself from prompt iteration to writing user facing stories like incorporating interactive feedback. While we have support teams to intake client issues and fix them, you will own AI Quality and be the point person for what is happening and what needs to improve.\r\nThis is a product role, not a research role. We want the analytical rigor of a data scientist paired with the judgment of a product manager: someone who can run the experiment, interpret it honestly, and then turn it into a shipped change.\r\nKey ResponsibilitiesEvaluation & AI QualityBuild and own CentrlX's evaluation foundation from the ground up: golden datasets, grading rubrics, LLM-as-judge pipelines calibrated against human labels, and regression suites that run before prompt or model changes ship.\r\nDefine what \"good\" means for each core workflow — document digitization and extraction, retrieval and groundedness, Smart Summary, Smart Response, Smart Evaluation, and full multi-step agent runs — and set a measurable quality bar for each.\r\nEvaluate agent behavior, not just single responses: tool selection, retrieval quality, step sequencing, and whether the finished deliverable holds up to a practitioner's review.\r\nTurn every real client failure into a permanent eval case, so the same class of error does not come back.\r\nModel Selection & Cost OptimizationContinuously benchmark models across providers — OpenAI, Anthropic, Google, open-weight, and specialized document models — on accuracy, latency, and cost for each workflow, and make the call on what we run where.\r\nOwn model migrations end to end, including our in-flight move off GPT-4.1 in document digitization, where current alternatives are materially faster, cheaper, and more accurate.\r\nTrack and manage AI spend by workflow, and use routing, model tiering, caching, and context strategy to hold quality while bringing cost down.\r\nMaintain a working view of the model landscape — releases, pricing changes, deprecations — and turn it into a recommendation with evidence attached, not a newsletter.\r\nInstrumentation & DataDefine the logging and tracing we need — prompt inputs, retrieved context, prompt text, outputs, tool calls, token counts, latency — and write the stories to get it built.\r\nPartner with Product and Design to build in-app feedback capture (ratings, corrections, structured reason codes) so labeled data accumulates as a byproduct of normal use instead of a periodic collection project.\r\nBuild and maintain the datasets yourself: pull the data, label it, curate the hard slices, and keep the sets honest with holdouts and rotation.\r\nOwnership & Cross-Functional PartnershipServe as the single point of contact for AI quality escalations from Client Success, Sales Engineering, and Professional Services — triage, reproduce, root-cause, and close the loop.\r\nWrite the user stories and acceptance criteria that turn findings into shipped changes, and make the prompt, configuration, and model changes yourself where that is the fastest path.\r\nPublish a regular quality and cost readout that leadership, engineering, and client-facing teams all treat as the same version of the truth.\r\nWork with CENTRL's Manager Research, Investor Relations, and diligence practitioners to encode domain judgment into rubrics — in our market, accuracy is defined by industry expertise, not by a generic benchmark.\r\nMinimum QualificationsMust have work authorization in the USA.\r\n3+ years across product management, data science, or applied AI, including at least 2 years working on LLM-based products in production.\r\nHands-on Python and SQL. You are comfortable in a notebook pulling data, running batch inference, and computing metrics. This role writes code; it does not only specify it.\r\nDemonstrated experience building evaluation datasets and harnesses for LLM systems: golden sets, rubric design, LLM-as-judge with human calibration, and regression testing against prompt and model changes.\r\nWorking fluency with at least one eval or LLM observability platform: Braintrust, LangSmith, Langfuse, Arize Phoenix, W&B Weave, Inspect, Promptfoo, or a comparable in-house harness.\r\nPractical understanding of RAG systems: retrieval quality, groundedness and faithfulness, hallucination detection, and chunking and context strategy.\r\nStatistical literacy: you can size a comparison, judge significance, and say plainly when a difference is not real.\r\nAbility to write clear user stories and acceptance criteria and work inside an agile engineering process.\r\nStrong written communication. This role produces recommendations that executives act on.\r\nPreferred QualificationsExperience with document AI: OCR and vision-language extraction, table and layout parsing, and structured output from complex PDFs.\r\nExperience evaluating agentic systems — multi-step trajectories, tool-use correctness, and long-run failure modes.\r\nExperience in financial services or investment management: due diligence, manager research, investor relations, or DDQ/RFP workflows.\r\nExperience reducing inference cost at scale through routing, tiering, batching, caching, or context compression.\r\nExperience designing in-product feedback mechanisms that generate labeled evaluation data.\r\nFamiliarity with agent frameworks and MCP.\r\nDegree in a quantitative or technical field.#J-18808-Ljbffr","company":"Centrl","rawCompany":"centrl","city":"Mountain View","state":"CA","isRemote":false,"isActive":false,"createdAt":"2026-09-27T01:10:23.599Z","occupations":[{"code":"15-2051.00","title":"Data Scientists","slug":"data-scientists"},{"code":"15-1221.00","title":"Computer and Information Research Scientists","slug":"computer-and-information-research-scientists"},{"code":"11-3021.00","title":"Computer and Information Systems Managers","slug":"computer-and-information-systems-managers"}],"industries":[{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"},{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Technical Product Manager, Data Science","description":"CENTRL is a leading risk and compliance technology company that provides AI powered enterprise-grade risk, due diligence, cyber security and privacy management solutions to financial institutions worldwide. Our clients include some of the largest banks and investment management firms across the Americas, Europe and APAC. Headquartered in Silicon Valley, CENTRL has regional offices in New York, India, Australia, and the United Kingdom. Established in 2015, CENTRL is a high-growth, venture backed SaaS firm leading the way in innovative generative AI solutions to manage third party risk and due diligence.\r\nPosition Overview:We are looking for an owner for the accuracy, reliability, and cost-efficiency of the AI behind CentrlX, our agentic platform for Manager Research and Investor Relations teams at investment firms.\r\nWe have a strong engineering team building the platform: agents, skills, knowledge, automations, connectors, and governance. We are looking for a single owner for output accuracy and efficiency.\r\nYou will own the model and prompt layer end to end. You will build the evaluation datasets and harnesses that tell us whether a change actually helped, run structured comparisons across model providers on accuracy, latency, and cost, and drive the resulting changes into the product. You will help create and design agents and skills, and work closely with sales and PS in order to help drive standards for prompting and agents, and then feed that back into the product design. You will do a real share of this work yourself from prompt iteration to writing user facing stories like incorporating interactive feedback. While we have support teams to intake client issues and fix them, you will own AI Quality and be the point person for what is happening and what needs to improve.\r\nThis is a product role, not a research role. We want the analytical rigor of a data scientist paired with the judgment of a product manager: someone who can run the experiment, interpret it honestly, and then turn it into a shipped change.\r\nKey ResponsibilitiesEvaluation & AI QualityBuild and own CentrlX's evaluation foundation from the ground up: golden datasets, grading rubrics, LLM-as-judge pipelines calibrated against human labels, and regression suites that run before prompt or model changes ship.\r\nDefine what \"good\" means for each core workflow — document digitization and extraction, retrieval and groundedness, Smart Summary, Smart Response, Smart Evaluation, and full multi-step agent runs — and set a measurable quality bar for each.\r\nEvaluate agent behavior, not just single responses: tool selection, retrieval quality, step sequencing, and whether the finished deliverable holds up to a practitioner's review.\r\nTurn every real client failure into a permanent eval case, so the same class of error does not come back.\r\nModel Selection & Cost OptimizationContinuously benchmark models across providers — OpenAI, Anthropic, Google, open-weight, and specialized document models — on accuracy, latency, and cost for each workflow, and make the call on what we run where.\r\nOwn model migrations end to end, including our in-flight move off GPT-4.1 in document digitization, where current alternatives are materially faster, cheaper, and more accurate.\r\nTrack and manage AI spend by workflow, and use routing, model tiering, caching, and context strategy to hold quality while bringing cost down.\r\nMaintain a working view of the model landscape — releases, pricing changes, deprecations — and turn it into a recommendation with evidence attached, not a newsletter.\r\nInstrumentation & DataDefine the logging and tracing we need — prompt inputs, retrieved context, prompt text, outputs, tool calls, token counts, latency — and write the stories to get it built.\r\nPartner with Product and Design to build in-app feedback capture (ratings, corrections, structured reason codes) so labeled data accumulates as a byproduct of normal use instead of a periodic collection project.\r\nBuild and maintain the datasets yourself: pull the data, label it, curate the hard slices, and keep the sets honest with holdouts and rotation.\r\nOwnership & Cross-Functional PartnershipServe as the single point of contact for AI quality escalations from Client Success, Sales Engineering, and Professional Services — triage, reproduce, root-cause, and close the loop.\r\nWrite the user stories and acceptance criteria that turn findings into shipped changes, and make the prompt, configuration, and model changes yourself where that is the fastest path.\r\nPublish a regular quality and cost readout that leadership, engineering, and client-facing teams all treat as the same version of the truth.\r\nWork with CENTRL's Manager Research, Investor Relations, and diligence practitioners to encode domain judgment into rubrics — in our market, accuracy is defined by industry expertise, not by a generic benchmark.\r\nMinimum QualificationsMust have work authorization in the USA.\r\n3+ years across product management, data science, or applied AI, including at least 2 years working on LLM-based products in production.\r\nHands-on Python and SQL. You are comfortable in a notebook pulling data, running batch inference, and computing metrics. This role writes code; it does not only specify it.\r\nDemonstrated experience building evaluation datasets and harnesses for LLM systems: golden sets, rubric design, LLM-as-judge with human calibration, and regression testing against prompt and model changes.\r\nWorking fluency with at least one eval or LLM observability platform: Braintrust, LangSmith, Langfuse, Arize Phoenix, W&B Weave, Inspect, Promptfoo, or a comparable in-house harness.\r\nPractical understanding of RAG systems: retrieval quality, groundedness and faithfulness, hallucination detection, and chunking and context strategy.\r\nStatistical literacy: you can size a comparison, judge significance, and say plainly when a difference is not real.\r\nAbility to write clear user stories and acceptance criteria and work inside an agile engineering process.\r\nStrong written communication. This role produces recommendations that executives act on.\r\nPreferred QualificationsExperience with document AI: OCR and vision-language extraction, table and layout parsing, and structured output from complex PDFs.\r\nExperience evaluating agentic systems — multi-step trajectories, tool-use correctness, and long-run failure modes.\r\nExperience in financial services or investment management: due diligence, manager research, investor relations, or DDQ/RFP workflows.\r\nExperience reducing inference cost at scale through routing, tiering, batching, caching, or context compression.\r\nExperience designing in-product feedback mechanisms that generate labeled evaluation data.\r\nFamiliarity with agent frameworks and MCP.\r\nDegree in a quantitative or technical field.#J-18808-Ljbffr","datePosted":"2026-09-27T01:10:23.599Z","dateModified":"2026-09-27T01:10:23.599Z","hiringOrganization":{"@type":"Organization","name":"Centrl","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Mountain View","addressRegion":"CA","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"de67f0def22cf00eb2637ec9"},"url":"https://jobsearcher.com/jobs/de67f0def22cf00eb2637ec9"}}