Lead Data Engineer Databricks Data Ingestion
Hi, Our client is looking Lead Data Engineer Databricks Data Ingestion--for San Francisco, CA project and below is the detailed requirement.
Job Title: Lead Data Engineer Databricks Data Ingestion
Location: San Francisco, CA Duration : Contract
Job Description:
Bachelor's degree in Information Systems, Business Administration, Computer Science, or related field.8+ years of experience in data engineering, data integration, and enterprise data platforms.
Strong hands-on experience with Databricks Lakeflow Connect and Databricks-native data ingestion capabilities.
Experience with Databricks Native Data Loader or related native ingestion tooling.
Strong knowledge of Apache Spark, PySpark, and Delta Lake.
Hands-on experience ingesting and migrating data from Oracle, MySQL, and Microsoft SQL Server.
Strong experience designing and implementing ETL/ELT pipelines and large-scale data integration solutions.
Understanding of incremental ingestion, CDC, schema evolution, and data reconciliation.
Experience with batch and streaming data processing.
Strong SQL skills and knowledge of relational database concepts.
Experience with data quality, pipeline monitoring, troubleshooting, and performance tuning.
Proven technical leadership, mentoring, and stakeholder collaboration skills.
Experience with Agile methodologies, CI/CD, automated testing, and technical documentation. Strong documentation, communication, and stakeholder management skills.