Lead Data Engineer
Job Title: Databricks Architect
Location: San Jose, CA (Hybrid)
Fulltime and Contract
Requirements
Key Responsibilities:
Work on client projects to deliver Microsoft Azure based Data engineering & Analytics solutions.
Provide hands on technical leadership for relevant technical skills, in multiple projects if required.
Design and implement high performing scalable analytics solutions on Azure following well-architected framework guidelines.
Design pipelines to ingest & integrate structured/unstructured data from heterogeneous sources.
Collaborate with delivery leadership to deliver projects on time adhering to the quality standards
Mandatory Skills
In-depth knowledge Data processing techniques and handling large data sets.
Hands on experience using different storage services on Azure for storing data, like ADLS, SQL DB, SYnapse, Delta tables, Hive.
Expert level hands on experience using Databricks in at least 4 projects, creating notebooks using Scala/Python.
Expertise using Pyspark and Spark SQL for processing data for various use cases.
Expertise in code/performance optimization, and troubleshooting PySpark notebooks.
Good understanding of Delta lake and how it can be used in conjunction with ADLS and other storage.
Experience working with atleast 1 big data project using map-reduce.
Experience with Databricks scheduling features to automate Databricks jobs.
Problem-solving skills along with good interpersonal & communication skills.
Self-starter who can pick up any other relevant Azure Services in the Analytics space
Good-to-Have Skills
Databricks Certified Data Engineer
Knowledge of HDInsight, and Azure DevOps is a big plus
DP 203 - Microsoft Certified Azure Data Engineer
Knowledge in Power BI
Exposure to Databricks on AWS
Job Type: Contract
Pay: From $60.00 per hour
Expected hours: 8 per week
Benefits:
401(k)
Dental insurance
Health insurance
Compensation Package:
Yearly pay
Schedule:
Monday to Friday
Experience:
Databricks: 10 years (Required)
Power BI: 1 year (Required)
Azure Devops: 1 year (Required)
Microsoft Certified Azure Data Engineer: 1 year (Required)
AWS: 5 years (Required)
ADLs: 5 years (Required)
SQL DB: 5 years (Required)
Synapse: 5 years (Required)
Delta Tables: 5 years (Required)
Ability to Relocate:
San Jose, CA 95123: Relocate before starting work (Required)
Work Location: Hybrid remote in San Jose, CA 95123