{"schemaVersion":"jobsearcher.job.v1","id":"5e735c641ffe920eccaf5b78","url":"https://jobsearcher.com/jobs/5e735c641ffe920eccaf5b78","canonicalUrl":"https://jobsearcher.com/jobs/5e735c641ffe920eccaf5b78","title":"Lead Data Engineer – AWS, Databricks & PySpark","description":"**FULLY REMOTE**ScolerTec is seeking Lead Data Engineer – AWS, Databricks & PySpark to design, build, and lead enterprise-scale cloud data engineering solutions.This is a hands-on technical leadership role responsible for scalable ETL/ELT pipelines, batch and near-real-time data integration, lakehouse/data-platform implementation, cloud data services, CI/CD, data quality, governance, performance optimization, and technical direction of data engineers.The ideal candidate has strong recent experience with AWS, Databricks, Apache Spark/PySpark, Python, SQL, Delta Lake, ETL/ELT, and technical leadership.Key ResponsibilitiesDesign and implement scalable cloud-based data platforms, including data lake, lakehouse, and data warehouse architectures.Build enterprise-scale ingestion and ETL/ELT pipelines for batch and near-real-time workloads.Develop data transformation and integration solutions using Python, SQL, Spark/PySpark, and modern cloud data technologies.Implement Databricks/Delta Lake-based data engineering solutions and downstream curated/gold data layers.Integrate pipelines with AWS compute, storage, messaging, and analytics services.Lead technical design reviews, code reviews, engineering standards, and reusable data-engineering patterns.Provide day-to-day technical guidance and mentoring to data engineers.Implement data-quality checks, validation rules, error handling, observability, and pipeline monitoring.Support streaming and event-driven data processing.Implement Infrastructure as Code and CI/CD for repeatable data-platform and pipeline deployment.Optimize pipeline performance, scalability, reliability, and cloud cost.Support production incident resolution and root-cause analysis.Partner with architecture, governance, migration, QA, DevSecOps, and security teams.Maintain technical documentation, runbooks, and audit-ready engineering artifacts.Qualifications7+ years of specialized data-engineering experience.Strong hands-on experience building enterprise-scale data pipelines.Strong production experience with AWS data and compute services.Strong hands-on Databricks experience.Strong Apache Spark/PySpark and Delta Lake experience.Strong Python and SQL skills.Strong ETL/ELT and data-transformation experience.Experience with batch and streaming/near-real-time ingestion.Experience with data orchestration tools.Experience with CI/CD and Infrastructure as Code.Experience with data quality, governance, metadata, and classification.Proven technical leadership, design review, code review, and mentoring experience.Federal, public-sector, or regulated-environment experience preferred.Preferred CertificationsDatabricks Certified Data Engineer ProfessionalAWS Certified Data EngineerAWS Certified Solutions Architect – ProfessionalAWS Certified Data Analytics – SpecialtyCDMP","company":"Scolertec","rawCompany":"scolertec","city":"Denver","state":"CO","isRemote":false,"isActive":false,"createdAt":"2026-10-15T11:57:30.372Z","occupations":[{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-1243.00","title":"Database Architects","slug":"database-architects"},{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"518210","title":"Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services","slug":"computing-infrastructure-providers-data-processing-web-hosting-and-related-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Lead Data Engineer – AWS, Databricks & PySpark","description":"**FULLY REMOTE**ScolerTec is seeking Lead Data Engineer – AWS, Databricks & PySpark to design, build, and lead enterprise-scale cloud data engineering solutions.This is a hands-on technical leadership role responsible for scalable ETL/ELT pipelines, batch and near-real-time data integration, lakehouse/data-platform implementation, cloud data services, CI/CD, data quality, governance, performance optimization, and technical direction of data engineers.The ideal candidate has strong recent experience with AWS, Databricks, Apache Spark/PySpark, Python, SQL, Delta Lake, ETL/ELT, and technical leadership.Key ResponsibilitiesDesign and implement scalable cloud-based data platforms, including data lake, lakehouse, and data warehouse architectures.Build enterprise-scale ingestion and ETL/ELT pipelines for batch and near-real-time workloads.Develop data transformation and integration solutions using Python, SQL, Spark/PySpark, and modern cloud data technologies.Implement Databricks/Delta Lake-based data engineering solutions and downstream curated/gold data layers.Integrate pipelines with AWS compute, storage, messaging, and analytics services.Lead technical design reviews, code reviews, engineering standards, and reusable data-engineering patterns.Provide day-to-day technical guidance and mentoring to data engineers.Implement data-quality checks, validation rules, error handling, observability, and pipeline monitoring.Support streaming and event-driven data processing.Implement Infrastructure as Code and CI/CD for repeatable data-platform and pipeline deployment.Optimize pipeline performance, scalability, reliability, and cloud cost.Support production incident resolution and root-cause analysis.Partner with architecture, governance, migration, QA, DevSecOps, and security teams.Maintain technical documentation, runbooks, and audit-ready engineering artifacts.Qualifications7+ years of specialized data-engineering experience.Strong hands-on experience building enterprise-scale data pipelines.Strong production experience with AWS data and compute services.Strong hands-on Databricks experience.Strong Apache Spark/PySpark and Delta Lake experience.Strong Python and SQL skills.Strong ETL/ELT and data-transformation experience.Experience with batch and streaming/near-real-time ingestion.Experience with data orchestration tools.Experience with CI/CD and Infrastructure as Code.Experience with data quality, governance, metadata, and classification.Proven technical leadership, design review, code review, and mentoring experience.Federal, public-sector, or regulated-environment experience preferred.Preferred CertificationsDatabricks Certified Data Engineer ProfessionalAWS Certified Data EngineerAWS Certified Solutions Architect – ProfessionalAWS Certified Data Analytics – SpecialtyCDMP","datePosted":"2026-10-15T11:57:30.372Z","dateModified":"2026-10-15T11:57:30.372Z","hiringOrganization":{"@type":"Organization","name":"Scolertec","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Denver","addressRegion":"CO","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"5e735c641ffe920eccaf5b78"},"url":"https://jobsearcher.com/jobs/5e735c641ffe920eccaf5b78"}}