Data Quality Engineer

Data Quality EngineerCharlotte, NC - Onsite 3 Days a week at Optimist HallHourly Rate: $55-$60/hour YOE12 Month Contract to Hire - W-2 only** not open to candidates who would be relocatingRole OverviewWe are seeking a Senior Data Quality Engineer to design, implement, and maintain automated data quality validations across our enterprise data engineering ecosystem. This role focuses on ensuring the accuracy, completeness, consistency, and timeliness of data flowing through both batch and streaming pipelines built on AWS.You will work closely with data engineers, analytics teams, and business stakeholders to embed data quality controls into pipelines built with AWS Glue, PySpark, Kafka, AWS DMS, Lambda, and Aurora PostgreSQL, supporting trusted analytics and reporting in Qlik.Key ResponsibilitiesData Quality EngineeringDesign and implement automated data quality checks across ingestion, transformation, and consumption layers.Define and enforce data quality rules for key dimensions such as completeness, validity, uniqueness, consistency, and timeliness.Build reusable Python and PySpark frameworks for validating large-scale datasets.Batch & Streaming ValidationEmbed data quality validations into AWS Glue (PySpark) batch pipelines.Implement real-time or near-real-time validations for Kafka-based streaming pipelines, including schema validation, duplicate detection, and latency checks.Monitor and validate event-time vs. processing-time behavior for streaming data.CDC & Ingestion QualityValidate AWS DMS change data capture pipelines, ensuring accuracy between source systems and downstream targets.Perform reconciliation checks (row counts, aggregates, checksums) between source and target systems.Detect and alert on data gaps, duplication, or schema drift in CDC pipelines.Data Stores & Analytics ReadinessWrite advanced SQL-based data quality checks against Amazon Aurora PostgreSQL and curated data layers.Ensure data delivered to Qlik meets defined quality thresholds and freshness SLAs.Validate semantic consistency and completeness of datasets used for reporting and dashboards.Monitoring, Alerting & Incident ManagementImplement data quality monitoring, logging, and alerting using AWS Lambda, CloudWatch, and pipeline metrics.Create dashboards and alerts for data quality failures and SLA breaches.Perform root-cause analysis of data quality incidents and drive long-term remediation.Standards, Governance & CollaborationPartner with data engineers to embed quality gates into CI/CD and deployment workflows.Contribute to data quality standards, documentation, and operational runbooks.Act as a subject-matter expert for data quality best practices across batch and streaming architectures.Required Qualifications6+ years of experience in data engineering, analytics engineering, or data quality engineering.Strong hands-on experience with AWS Glue, PySpark, and Python.Experience validating batch and streaming data pipelines.Practical knowledge of Kafka for streaming ingestion and validation use cases.Experience working with AWS DMS for CDC pipelines and data reconciliation.Advanced SQL skills and experience with Amazon Aurora PostgreSQL.Experience implementing serverless workflows using AWS Lambda.Understanding of data modeling concepts and multi-layer data architectures.Strong analytical and problem-solving skills with attention to detail.Ability to communicate data quality issues clearly to technical and non-technical stakeholders.Preferred QualificationsExperience supporting BI tools such as Qlik or similar analytics platforms.Familiarity with data observability concepts and quality metrics.Knowledge of schema management and schema evolution in streaming systems.Experience in regulated or highly governed data environments.Exposure to CI/CD pipelines and Infrastructure-as-Code practices.What Success Looks LikeCritical datasets have automated, repeatable data quality validations.Data quality issues are detected early and resolved before impacting analytics.Streaming and batch pipelines meet defined quality and freshness SLAs.Business users trust analytics and reporting outputs with minimal manual intervention.