{"schemaVersion":"jobsearcher.job.v1","id":"5dfe53ba8d86b507b8bdba01","url":"https://jobsearcher.com/jobs/5dfe53ba8d86b507b8bdba01","canonicalUrl":"https://jobsearcher.com/jobs/5dfe53ba8d86b507b8bdba01","title":"Lead Data Engineer","description":"Enterprise Data Lake Technical LeadRemote Role14+ yearsFully Remote RoleONLY Independent visaPosition OverviewThe Enterprise Data Lake (EDL) Technical Lead is responsible for owning the design, implementation, and engineering leadership of the enterprise data lake platform. This role ensures the platform is scalable, reliable, secure, and consumable across analytics, reporting, and operational workloads.The Tech Lead drives technical decisions and collaborates closely with Architecture, Product Management, Data Engineering, Security, and downstream consumers to deliver robust data platform capabilities.Key ResponsibilitiesTechnical LeadershipOwn the design and implementation of enterprise-scale data lake solutions on Azure/AWS CloudDefine technical standards and best practices for data platform componentsDrive technical decision-making across data ingestion, storage, processing, and governance layersProvide technical mentorship and guidance to data engineering and platform teamsCollaborate with Architecture team on overall architectural alignment and strategyPartner with Product Management, Data Engineering, Security, SRE, and downstream consumersData Governance & Catalog Management (Critical)Own the design and implementation of enterprise data catalog and metadata management layerBuild and operate data catalog using DataHub or equivalent metadata platformDesign and implement automated metadata ingestion, lineage tracking, and data discovery capabilitiesDesign and implement data governance policies, data quality rules, and compliance frameworks or Chrono data OrchestratorEnable self-service data discovery and access management for downstream consumersEstablish data ownership, stewardship models, and metadata standards across the enterpriseBuild APIs and services for metadata-driven data operations and workflowsPlatform Engineering & DevelopmentDesign and implement real-time and batch data ingestion pipelines using Apache KafkaOptimize data processing workflows on Databricks platform (Delta Lake, Spark optimization, Unity Catalog)Design integration patterns between RDBMS sources and data lake (CDC, batch ETL, replication)Optimize database performance, query tuning, and indexing strategies across relational and distributed systemsImplement infrastructure as code using Terraform for automated provisioning and managementDesign and deploy containerized data services on Kubernetes clustersDevelop data platform services and tooling using C# and GoBuild database migration tools, data validation frameworks, and connectivity servicesOperations & ReliabilityEnsure platform scalability, reliability, and security across all data lake componentsImplement monitoring, logging, and observability solutions for data infrastructureOptimize Linux-based systems for data processing workloadsEstablish CI/CD pipelines for data platform deploymentsEnsure platform security, compliance, and data governance standards (GDPR, SOC2, etc.)Drive cost optimization and performance tuning initiativesImplement database backup/recovery strategies and disaster recovery planningRequired QualificationsTechnical ExpertiseData Streaming & Processing5+ years with Apache Kafka (streaming architecture, Kafka Connect, Schema Registry, stream processing)3+ years with Databricks (Delta Lake, Apache Spark optimization, Unity Catalog, cluster management)Deep understanding of data lake architecture patterns (Bronze/Silver/Gold, medallion architecture)Data Governance & Cataloging3+ years with DataHub or similar metadata management platforms (Alation, Collibra, Apache Atlas)Deep experience building and operating enterprise data catalog systemsExpertise in automated metadata extraction, lineage tracking, and impact analysisExperience with data quality frameworks and metadata-driven data operationsKnowledge of data governance policies, data classification, and compliance automationUnderstanding of data discovery, access control, and self-service analytics enablementRelational Database Systems5+ years with enterprise RDBMS platforms including:SQL Server (T-SQL, SSIS, SSRS, replication, Always On Availability Groups)PostgreSQL (advanced query optimization, partitioning, extensions, streaming replication)MySQL (replication, clustering, performance tuning)Strong SQL skills (complex queries, stored procedures, window functions, query optimization)Database design principles (normalization, indexing strategies, schema design, partitioning)Change Data Capture (CDC) patterns and implementation (Debezium, Azure Data Factory, AWS DMS, custom solutions)Database migration experience (schema migration, data migration, zero-downtime migrations)Cloud & Infrastructure5+ years with Azure or AWS Cloud including:Azure: Data Lake Storage (Gen2), Event Hubs, AKS, Azure SQL, Key Vault, Azure AD, MonitorAWS: S3, MSK/Kinesis, EKS, RDS/Aurora, Secrets Manager, IAM, CloudWatchCloud-native data services, networking, security, and IAM3+ years with Kubernetes (deployment strategies, scaling, monitoring, service mesh, Helm)3+ years with Terraform (modules, state management, multi-environment deployments, multi-cloud)Programming & DevelopmentStrong proficiency in any programming languages like C#, GoExpert-level SQL across multiple database platformsExperience with Python for data engineering tasks (preferred)Familiarity with Shell scripting (Bash)Leadership & Experience7+ years in data engineering, platform engineering, or database engineering roles3+ years in technical leadership capacity (Tech Lead, Principal Engineer)Proven track record of delivering large-scale data infrastructure projectsExperience leading teams of 5-10+ engineersStrong architectural design and system thinking capabilitiesExperience migrating legacy RDBMS workloads to modern data lake architecturesDemonstrated ability to balance technical excellence with business needsPreferred QualificationsAdditional TechnologiesExperience with cloud ETL services (Azure Data Factory, AWS Glue, Azure Stream Analytics, AWS Lambda)Knowledge of managed database features (Azure SQL elastic pools/hyperscale, AWS RDS/Aurora serverless)Knowledge of additional streaming technologies (Apache Flink)Experience with database sharding and horizontal partitioning strategies at scaleFamiliarity with NoSQL databases (Cosmos DB, DynamoDB, MongoDB, Cassandra, Redis)Experience with Apache IcebergKnowledge of data observability tools","company":"Accion Labs","rawCompany":"accion labs","city":"Denver","state":"CO","isRemote":false,"isActive":false,"createdAt":"2026-11-05T07:28:34.597Z","occupations":[{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-1243.00","title":"Database Architects","slug":"database-architects"},{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Lead Data Engineer","description":"Enterprise Data Lake Technical LeadRemote Role14+ yearsFully Remote RoleONLY Independent visaPosition OverviewThe Enterprise Data Lake (EDL) Technical Lead is responsible for owning the design, implementation, and engineering leadership of the enterprise data lake platform. This role ensures the platform is scalable, reliable, secure, and consumable across analytics, reporting, and operational workloads.The Tech Lead drives technical decisions and collaborates closely with Architecture, Product Management, Data Engineering, Security, and downstream consumers to deliver robust data platform capabilities.Key ResponsibilitiesTechnical LeadershipOwn the design and implementation of enterprise-scale data lake solutions on Azure/AWS CloudDefine technical standards and best practices for data platform componentsDrive technical decision-making across data ingestion, storage, processing, and governance layersProvide technical mentorship and guidance to data engineering and platform teamsCollaborate with Architecture team on overall architectural alignment and strategyPartner with Product Management, Data Engineering, Security, SRE, and downstream consumersData Governance & Catalog Management (Critical)Own the design and implementation of enterprise data catalog and metadata management layerBuild and operate data catalog using DataHub or equivalent metadata platformDesign and implement automated metadata ingestion, lineage tracking, and data discovery capabilitiesDesign and implement data governance policies, data quality rules, and compliance frameworks or Chrono data OrchestratorEnable self-service data discovery and access management for downstream consumersEstablish data ownership, stewardship models, and metadata standards across the enterpriseBuild APIs and services for metadata-driven data operations and workflowsPlatform Engineering & DevelopmentDesign and implement real-time and batch data ingestion pipelines using Apache KafkaOptimize data processing workflows on Databricks platform (Delta Lake, Spark optimization, Unity Catalog)Design integration patterns between RDBMS sources and data lake (CDC, batch ETL, replication)Optimize database performance, query tuning, and indexing strategies across relational and distributed systemsImplement infrastructure as code using Terraform for automated provisioning and managementDesign and deploy containerized data services on Kubernetes clustersDevelop data platform services and tooling using C# and GoBuild database migration tools, data validation frameworks, and connectivity servicesOperations & ReliabilityEnsure platform scalability, reliability, and security across all data lake componentsImplement monitoring, logging, and observability solutions for data infrastructureOptimize Linux-based systems for data processing workloadsEstablish CI/CD pipelines for data platform deploymentsEnsure platform security, compliance, and data governance standards (GDPR, SOC2, etc.)Drive cost optimization and performance tuning initiativesImplement database backup/recovery strategies and disaster recovery planningRequired QualificationsTechnical ExpertiseData Streaming & Processing5+ years with Apache Kafka (streaming architecture, Kafka Connect, Schema Registry, stream processing)3+ years with Databricks (Delta Lake, Apache Spark optimization, Unity Catalog, cluster management)Deep understanding of data lake architecture patterns (Bronze/Silver/Gold, medallion architecture)Data Governance & Cataloging3+ years with DataHub or similar metadata management platforms (Alation, Collibra, Apache Atlas)Deep experience building and operating enterprise data catalog systemsExpertise in automated metadata extraction, lineage tracking, and impact analysisExperience with data quality frameworks and metadata-driven data operationsKnowledge of data governance policies, data classification, and compliance automationUnderstanding of data discovery, access control, and self-service analytics enablementRelational Database Systems5+ years with enterprise RDBMS platforms including:SQL Server (T-SQL, SSIS, SSRS, replication, Always On Availability Groups)PostgreSQL (advanced query optimization, partitioning, extensions, streaming replication)MySQL (replication, clustering, performance tuning)Strong SQL skills (complex queries, stored procedures, window functions, query optimization)Database design principles (normalization, indexing strategies, schema design, partitioning)Change Data Capture (CDC) patterns and implementation (Debezium, Azure Data Factory, AWS DMS, custom solutions)Database migration experience (schema migration, data migration, zero-downtime migrations)Cloud & Infrastructure5+ years with Azure or AWS Cloud including:Azure: Data Lake Storage (Gen2), Event Hubs, AKS, Azure SQL, Key Vault, Azure AD, MonitorAWS: S3, MSK/Kinesis, EKS, RDS/Aurora, Secrets Manager, IAM, CloudWatchCloud-native data services, networking, security, and IAM3+ years with Kubernetes (deployment strategies, scaling, monitoring, service mesh, Helm)3+ years with Terraform (modules, state management, multi-environment deployments, multi-cloud)Programming & DevelopmentStrong proficiency in any programming languages like C#, GoExpert-level SQL across multiple database platformsExperience with Python for data engineering tasks (preferred)Familiarity with Shell scripting (Bash)Leadership & Experience7+ years in data engineering, platform engineering, or database engineering roles3+ years in technical leadership capacity (Tech Lead, Principal Engineer)Proven track record of delivering large-scale data infrastructure projectsExperience leading teams of 5-10+ engineersStrong architectural design and system thinking capabilitiesExperience migrating legacy RDBMS workloads to modern data lake architecturesDemonstrated ability to balance technical excellence with business needsPreferred QualificationsAdditional TechnologiesExperience with cloud ETL services (Azure Data Factory, AWS Glue, Azure Stream Analytics, AWS Lambda)Knowledge of managed database features (Azure SQL elastic pools/hyperscale, AWS RDS/Aurora serverless)Knowledge of additional streaming technologies (Apache Flink)Experience with database sharding and horizontal partitioning strategies at scaleFamiliarity with NoSQL databases (Cosmos DB, DynamoDB, MongoDB, Cassandra, Redis)Experience with Apache IcebergKnowledge of data observability tools","datePosted":"2026-11-05T07:28:34.597Z","dateModified":"2026-11-05T07:28:34.597Z","hiringOrganization":{"@type":"Organization","name":"Accion Labs","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Denver","addressRegion":"CO","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"5dfe53ba8d86b507b8bdba01"},"url":"https://jobsearcher.com/jobs/5dfe53ba8d86b507b8bdba01"}}