Lead Databricks Developer
ARCHIVED
We can't find an active application page for this role right now. It may reopen or be listed elsewhere. Use Next Steps to search for an active apply link and similar live jobs.
Job Title: Lead Databricks Developer
Location: Wilmington, DE or Plano, TX – 5 Days onsite role
Summary:
We are looking for a Lead Databricks Developer with strong experience in Python and Apache Spark to design, develop, and optimize scalable data pipelines and cloud-based data solutions. The ideal candidate will have hands-on expertise in Databricks, PySpark, SQL, and modern data engineering practices.
Key Responsibilities: -
Design, develop, and maintain ETL/ELT pipelines using Databricks and PySpark- Build scalable batch and real-time data processing solutions using Apache Spark- Develop data transformation workflows using Python, PySpark, and Spark SQL- Implement Delta Lake solutions and optimize data processing performance- Create and manage Databricks notebooks, jobs, and workflows- Ensure data quality, validation, logging, and exception handling- Optimize Spark performance using partitioning, caching, and tuning techniques- Collaborate with cross-functional teams to gather requirements and deliver solutions- Support CI/CD, deployment automation, and production releases- Troubleshoot and resolve issues in data pipelines and platform processes
Required Skills: -
Strong hands-on experience with Databricks
Strong Python programming skills
Strong in communication and proactive.
Expertise in Apache Spark, PySpark, and Spark SQL- Strong SQL and data transformation skills
Experience with Delta Lake and lakehouse concepts- Knowledge of ETL/ELT and data pipeline design
Experience with AWS, Azure, or GCP- Familiarity with Git, CI/CD, and Agile practices
Good problem-solving and communication skills
Preferred Skills: -
Experience with Airflow, Azure Data Factory, or Databricks Workflows
Knowledge of Kafka or streaming frameworks
Experience with MLflow or analytics platforms
Understanding of data governance and security best practices