Lead Data Engineer
ARCHIVED
We can't find an active application page for this role right now. It may reopen or be listed elsewhere. Use Next Steps to search for an active apply link and similar live jobs.
Location
United States
Start and end date
12 Oct 2026 – 24 Oct 2027
Indicative hourly rate
$80 – 85/hr
Respond by
10 Oct 2026
Skills
Security Controls, Machine learning, Architecture, Enterprise Data Management, Python programming, Data, Confluent, Amazon, Feature engineering, Data streaming, Data Quality, Code reviews, EcoSYSTEMS, Identity and access management, Extract/transform/load, System Migration, Software applications, PySpark, Artificial Intelligence, SQL, Cloud Migration, Performance tuning, Language models, airflow, Business Analytics, Engineering, Data Governance, Data Architecture, Data analysis, Data management, Github, Apache Spark, Data warehousing, Vector, Distributed Computing Environment, Apache Kafka, Cloud Engineering, Continuous Delivery, Analytics, Databricks, Big Data, Event Driven Architecture, Infrastructure management, Supervision, Data modelling, Microsoft, CloudWatch, Planning, Workflow Management Systems, Amazon S3, Database administrationSecurity Controls, Machine learning, Architecture, Enterprise Data Management, Python programming, Data, Confluent, Amazon, Feature engineering, Data streaming, Data Quality, Code reviews, EcoSYSTEMS, Identity and access management, Extract/transform/load, System Migration, Software applications, PySpark, Artificial Intelligence, SQL, Cloud Migration, Performance tuning, Language models, airflow, Business Analytics, Engineering, Data Governance, Data Architecture, Data analysis, Data management, Github, Apache Spark, Data warehousing, Vector, Distributed Computing Environment, Apache Kafka, Cloud Engineering, Continuous Delivery, Analytics, Databricks, Big Data, Event Driven Architecture, Infrastructure management, Supervision, Data modelling, Microsoft, CloudWatch, Planning, Workflow Management Systems, Amazon S3, Database administration View all
Hybrid to Charlotte is preferred
Lead Data Engineer
Core Technical Skills
8+ years of experience in Data Engineering with at least 5+ years working extensively within AWS ecosystems
Expert-level experience with AWS services including S3, EMR, Glue Jobs, Lambda, Athena, CloudTrail, SNS, SQS, CloudWatch, and Step Functions
Experience creating AI applications with AWS Bedrock
Strong experience designing and implementing enterprise-scale data lake and data warehouse solutions using Lake Formation, Amazon Redshift, and Amazon Athena
Extensive experience with Kafka-based streaming architectures, preferably Confluent Kafka
Advanced SQL and data modeling expertise, including dimensional modeling, data vault, and large-scale data warehousing solutions
Deep experience designing, developing, and optimizing scalable, resilient data pipelines within AWS environments
Strong understanding of distributed data processing frameworks, particularly PySpark and EMR
Expert knowledge of database management principles, performance tuning, and data architecture best practices
Advanced Python development skills with extensive hands-on experience using PySpark
Expertise in Infrastructure as Code using Terraform
Experience designing and implementing CI/CD frameworks using GitHub and GitHub Actions
Deep knowledge of AWS IAM roles, policies, governance, and security best practices
Strong experience with workflow orchestration tools such as AWS Step Functions, Apache Airflow, or equivalent platforms
Experience leading cloud migration, modernization, and enterprise data platform initiatives
Strong understanding of data governance, metadata management, data quality frameworks, and observability principles
Core Responsibilities:
Lead the design, architecture, and implementation of enterprise data engineering solutions across the AWS ecosystem
Collaborate with Lead Developers, Data Scientists, Architects, Product Owners, and business stakeholders to define technical strategy and scalable solutions
Provide technical leadership and mentorship to Data Engineers and development teams, promoting engineering excellence and best practices
Drive key architectural decisions in partnership with Data Architects and Solution Architects to ensure scalability, security, reliability, and maintainability
Design and oversee data warehouse and data lake solutions that balance business usability, performance, and long-term sustainability
Establish engineering standards for data modeling, ETL frameworks, pipeline reliability, monitoring, and operational excellence
Lead end-to-end solution delivery, ensuring alignment with business requirements, enterprise architecture standards, and regulatory requirements
Oversee production support and operational management of AWS-based data platforms, driving root-cause analysis and performance optimization initiatives
Champion data quality, governance, observability, and data stewardship practices across platforms and teams
Identify opportunities to modernize data architecture and improve operational efficiency through automation and cloud-native technologies
Core Experience and Abilities
Ability to lead hands-on development efforts while providing technical direction, code reviews, and engineering oversight across multiple initiatives
Experience establishing development environments, infrastructure standards, security controls, and migration strategies across multiple AWS accounts and environments
Ability to architect, develop, and govern enterprise-scale data pipelines, ETL processes, data ingestion frameworks, and orchestration workflows
Proven ability to identify data gaps, define strategic remediation plans, and implement scalable automation solutions that improve analytical capabilities across the organization
Ability to design and implement highly reliable data pipelines with strong focus on data quality, observability, resiliency, and operational supportability
Experience building and optimizing large-scale data warehousing solutions that prioritize both business user experience and system performance
Ability to drive technical decision-making, influence architectural direction, and effectively communicate complex technical concepts to both technical and non-technical stakeholders
Proven track record mentoring engineers, fostering technical growth, and building high-performing data engineering teams
Experience leading cross-functional initiatives involving data engineering, analytics, architecture, platform engineering, and business stakeholders
Strong problem-solving and leadership skills with the ability to manage competing priorities in fast-paced enterprise environments
Experience building AI-ready data pipelines and ML workflows, including feature engineering and MLOps
Proficiency in Python, SQL, Spark, and Generative AI technologies for scalable data and analytics solutions
Hands-on experience with AWS, Azure, or Databricks and cloud-based AI/data platforms
Knowledge of LLMs, RAG, and vector databases to support AI-powered applications and intelligent search
Strong understanding of data governance, quality, security, and responsible AI practices
Preferred Qualifications:
AWS Certified Data Engineer, AWS Solutions Architect, or equivalent cloud certifications
Experience implementing enterprise data governance and metadata management platforms
Experience with real-time analytics, event-driven architectures, and streaming data platforms
Knowledge of modern data architecture patterns including Data Mesh, Lakehouse, and domain-oriented design
Experience leading large-scale cloud transformation or enterprise data modernization programs
Requirements:
Skills
Security Controls — Intermediate
Machine learning — Intermediate
Architecture — Intermediate
Enterprise Data Management — Intermediate
Python programming — Intermediate
Data — Intermediate
Confluent — Intermediate
Amazon — Intermediate
Feature engineering — Intermediate
Data streaming — Intermediate
Data Quality — Intermediate
Code reviews — Intermediate
EcoSYSTEMS — Intermediate
Identity and access management — Intermediate
Extract/transform/load — Intermediate
System Migration — Intermediate
Software applications — Intermediate
PySpark — Nice to have
Artificial Intelligence — Intermediate
SQL — Intermediate
Cloud Migration — Intermediate
Performance tuning — Intermediate
Language models — Intermediate
airflow — Intermediate
Business Analytics — Intermediate
Engineering — Intermediate
Data Governance — Intermediate
Data Architecture — Intermediate
Data analysis — Intermediate
Data management — Intermediate
Github — Intermediate
Apache Spark — Intermediate
Data warehousing — Intermediate
Vector — Intermediate
Distributed Computing Environment — Intermediate
Apache Kafka — Intermediate
Cloud Engineering — Intermediate
Continuous Delivery — Intermediate
Analytics — Intermediate
Databricks — Intermediate
Big Data — Intermediate
Event Driven Architecture — Intermediate
Infrastructure management — Intermediate
Supervision — Intermediate
Data modelling — Intermediate
Microsoft — Intermediate
CloudWatch — Intermediate
Planning — Intermediate
Workflow Management Systems — Intermediate
Amazon S3 — Intermediate
Database administration — Nice to have
Languages
English — Professional working proficiency