Lead Databricks Developer

About UsA Canadian technology and workforce solutions company headquartered in Vaughan, Ontario. The company helps businesses, enterprises, and public-sector organizations modernize their operations through secure digital transformation, custom software development, staffing, recruitment, payroll, and workforce management solutions.Job DescriptionJob Title: Lead Databricks DeveloperLocation: Wilmington, DE or Plano, TX – 5 Days onsite roleSummary: We are looking for a Lead Databricks Developer with strong experience in Python and Apache Spark to design, develop, and optimize scalable data pipelines and cloud-based data solutions. The ideal candidate will have hands-on expertise in Databricks, PySpark, SQL, and modern data engineering practices.Key Responsibilities:-Design, develop, and maintain ETL/ELT pipelines using Databricks and PySpark- Build scalable batch and real-time data processing solutions using Apache Spark- Develop data transformation workflows using Python, PySpark, and Spark SQL- Implement Delta Lake solutions and optimize data processing performance- Create and manage Databricks notebooks, jobs, and workflows- Ensure data quality, validation, logging, and exception handling- Optimize Spark performance using partitioning, caching, and tuning techniques- Collaborate with cross-functional teams to gather requirements and deliver solutions- Support CI/CD, deployment automation, and production releases- Troubleshoot and resolve issues in data pipelines and platform processesRequired Skills:-Strong hands-on experience with DatabricksStrong Python programming skillsStrong in communication and proactive.Expertise in Apache Spark, PySpark, and Spark SQL- Strong SQL and data transformation skillsExperience with Delta Lake and lakehouse concepts- Knowledge of ETL/ELT and data pipeline designExperience with AWS, Azure, or GCP- Familiarity with Git, CI/CD, and Agile practicesGood problem-solving and communication skillsPreferred Skills:-Experience with Airflow, Azure Data Factory, or Databricks WorkflowsKnowledge of Kafka or streaming frameworksExperience with MLflow or analytics platformsUnderstanding of data governance and security best practicescheck(event) ; career-website-detail-template-2 => apply(record.id,meta)" mousedown="lyte-button => check(event)" final-style="background-color:#0055FF;border-color:#0055FF;color:white;" final-class="lyte-button lyteBackgroundColorBtn lyteSuccess" lyte-rendered="">