{"schemaVersion":"jobsearcher.job.v1","id":"f0d53bcea41d48bd481bf78a","url":"https://jobsearcher.com/jobs/f0d53bcea41d48bd481bf78a","canonicalUrl":"https://jobsearcher.com/jobs/f0d53bcea41d48bd481bf78a","title":"Quality Assurance Automation Engineer","description":"Our client is the largest physician-led and physician-owned practice in their area of specialty in the United States.NOT considering c2c candidates.Position: Software Quality Analyst – 6 month contract to hireLocation: Wisconsin – fully REMOTEWe are seeking an excellent Quality Analyst to validate our cutting-edge AI-powered reporting solution within our medical specialty product line. This role involves working closely with a development team leveraging Amazon AWS and Large Language Models (LLM) to automate the interpretation of radiology reports, and explicitly summarizing the radiologist’s findings to streamline the review process; thus, providing the most accurate and relevant information at-a-glance.As the Quality Analyst, you will be responsible for ensuring the accuracy and reliability of AI models through rigorous testing, automation, and data validation. The ideal candidate will have a strong background preferably in automation, scripting, and data integrity, with a focus on AWS-based systems and machine learning models. Any experience with evaluating LLMs against BERTscores, ROUGE (Recall-Oriented Understudy for Gisting Evaluation), BLEU (Bilingual Evaluation Understudy), METEOR (Metric for Evaluation of Translation with Explicit ORdering) would also be very helpful.Required SkillsExperience with Large Language Models (LLM) and machine learning-based testing.Strong expertise in AWS cloud services. (Lambda, S3, Bedrock, etc.)Proficiency in scripting languages such as Python or C#.Hands-on experience with Automation tools (e.g., Selenium, or similar).Ability to perform Data Integrity checking and strong SQL skills for writing queries.Nice to HaveFamiliarity with foundational and custom models like Bedrock, Sagemaker, etc.Knowledge of microservices architecture.Familiarity with healthcare or radiology domains.At Codeworks, we're committed to diversity, equity, and inclusion in our workforce and beyond. We believe in equal opportunities and value the unique perspectives that every individual brings to our team. Join us in creating an inclusive, innovative, and collaborative workplace where your talents can thrive.Qualified candidates able to work as our direct employee can apply by emailing resumes to pete.neja@codeworks-inc.com","company":"Codeworks It","rawCompany":"codeworks it","city":"Denver","state":"CO","isRemote":false,"isActive":false,"createdAt":"2026-08-14T11:07:39.244Z","occupations":[{"code":"15-1253.00","title":"Software Quality Assurance Analysts and Testers","slug":"software-quality-assurance-analysts-and-testers"},{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"},{"code":"17-2112.02","title":"Validation Engineers","slug":"validation-engineers"}],"industries":[{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541690","title":"Other Scientific and Technical Consulting Services","slug":"other-scientific-and-technical-consulting-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Quality Assurance Automation Engineer","description":"Our client is the largest physician-led and physician-owned practice in their area of specialty in the United States.NOT considering c2c candidates.Position: Software Quality Analyst – 6 month contract to hireLocation: Wisconsin – fully REMOTEWe are seeking an excellent Quality Analyst to validate our cutting-edge AI-powered reporting solution within our medical specialty product line. This role involves working closely with a development team leveraging Amazon AWS and Large Language Models (LLM) to automate the interpretation of radiology reports, and explicitly summarizing the radiologist’s findings to streamline the review process; thus, providing the most accurate and relevant information at-a-glance.As the Quality Analyst, you will be responsible for ensuring the accuracy and reliability of AI models through rigorous testing, automation, and data validation. The ideal candidate will have a strong background preferably in automation, scripting, and data integrity, with a focus on AWS-based systems and machine learning models. Any experience with evaluating LLMs against BERTscores, ROUGE (Recall-Oriented Understudy for Gisting Evaluation), BLEU (Bilingual Evaluation Understudy), METEOR (Metric for Evaluation of Translation with Explicit ORdering) would also be very helpful.Required SkillsExperience with Large Language Models (LLM) and machine learning-based testing.Strong expertise in AWS cloud services. (Lambda, S3, Bedrock, etc.)Proficiency in scripting languages such as Python or C#.Hands-on experience with Automation tools (e.g., Selenium, or similar).Ability to perform Data Integrity checking and strong SQL skills for writing queries.Nice to HaveFamiliarity with foundational and custom models like Bedrock, Sagemaker, etc.Knowledge of microservices architecture.Familiarity with healthcare or radiology domains.At Codeworks, we're committed to diversity, equity, and inclusion in our workforce and beyond. We believe in equal opportunities and value the unique perspectives that every individual brings to our team. Join us in creating an inclusive, innovative, and collaborative workplace where your talents can thrive.Qualified candidates able to work as our direct employee can apply by emailing resumes to pete.neja@codeworks-inc.com","datePosted":"2026-08-14T11:07:39.244Z","dateModified":"2026-08-14T11:07:39.244Z","hiringOrganization":{"@type":"Organization","name":"Codeworks It","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Denver","addressRegion":"CO","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"f0d53bcea41d48bd481bf78a"},"url":"https://jobsearcher.com/jobs/f0d53bcea41d48bd481bf78a"}}