Databricks Data Engineer
Senior Databricks Data Engineer Jahnel Group's mission is to provide the absolute best environment for software creators to pursue their passion by connecting them with great clients doing meaningful work. This is a full-time position with one of our closest clients supporting a Federal government project. Who We're Looking ForWe're looking for a hands-on Senior Databricks Data Engineer who enjoys building reliable, scalable data platforms and solving complex data engineering problems. This is a highly technical role focused on designing, building, and maintaining production-grade data pipelines and data processing systems using Databricks, PySpark, and SQL. You'll work across batch and streaming workloads, data ingestion, transformation, data modeling, orchestration, and data quality. The ideal candidate is comfortable working independently in a cloud-based environment, understands modern data architecture patterns, and can take ownership of data solutions from design through production.
The ideal candidate:
- Has strong hands-on experience building production data pipelines with Databricks and PySparkThinks in terms of data architecture, scalability, reliability, and maintainability
- Can work across both batch and real-time data processing environments
- Understands how to structure and curate data using modern lakehouse architecture
- Is comfortable working with structured, semi-structured, and unstructured data
- Can troubleshoot and optimize Spark workloads for both performance and cost
- Takes ownership of data quality, monitoring, documentation, and operational reliability
- Communicates technical decisions clearly and works effectively in an Agile environment
Responsibilities:
- Design, build, and maintain production-grade batch and streaming data pipelines using Databricks, PySpark, SQL, and Python
- Develop data workflows using Databricks Workflows and Delta Live Tables (DLT)
- Implement Medallion architecture patterns across bronze, silver, and gold data layers
- Build scalable ingestion frameworks for structured, semi-structured, and unstructured data
- Integrate data from files, databases, APIs, and streaming sources
- Work with technologies such as Kafka, Kinesis, and Auto Loader for data ingestion and streaming workloads
- Design dimensional and domain-specific data models to support downstream analytics and applications
- Build and consume APIs to support data integration with downstream systems
- Optimize Spark jobs through partitioning, caching, cluster configuration, and other performance techniques
- Implement data quality checks, validation, monitoring, and pipeline health processes
- Work with Unity Catalog to support data organization, access controls, and governance
- Document data flows, lineage, architecture, and technical processes
- Collaborate with technical and business stakeholders to understand data requirements and deliver reliable solutions
- Participate in code reviews, testing, deployment, and ongoing production support
- Contribute to CI/CD workflows and engineering practices that improve the reliability and maintainability of the data platform
- Some Must-HavesActive U.S. Secret security clearance or higher 5+ years of professional data engineering experience 2+ years of hands-on Databricks experience
- Strong production experience with PySpark and SQLStrong Python development skills
- Experience building and supporting both batch and streaming data pipelines
- Solid understanding of data modeling and modern data architecture principles
- Experience working with structured, semi-structured, and unstructured data
- Experience with Git-based development and CI/CD workflows
- Experience working with Unity Catalog, including catalogs, schemas, and grants
- Experience optimizing Spark workloads for performance and cost
- Strong understanding of data quality, pipeline monitoring, and production supportU.S. citizenship and current residence in the United States
- Technical Requirements
- DatabricksPySparkSQLPythonDatabricks Workflows
- Delta Live Tables (DLT)
- KafkaAmazon Kinesis
- Auto LoaderUnity Catalog
- GitCI/CDAPIsSparkData modeling
- Batch and streaming data processing
Nice-to-Haves:
- Experience with Delta Live Tables (DLT)
- Experience with Kafka
- Experience with Amazon Kinesis
- Experience with Databricks Auto Loader
- Experience building reusable data ingestion frameworks
- Experience with dimensional data modeling
- Experience documenting data lineage and data flows
- Experience working in Federal or other highly regulated environments
- Experience supporting cloud-based data platforms
- Bachelor's degree in Computer Science, Engineering, or a related field
- Experience working in Agile development environments
- Where We're Looking For ItRemote - U.S.-based
Compensation Package: (Salary Transparency for US-Based Employees)
Salary Range:
$100,000 - $130,000 Candidates must reside in the United States and be U.S. citizens. Active Secret security clearance or higher is required. Candidates without an active Secret clearance will not be considered.