Data Engineer
Job Title
Data Engineer Job Type
Full TimeClient Location
Quincy, MAWork Arrangement
Onsite Duration
Full TimePay Rate / Salary
$120-$135k/yr.
Job Summary:
We are seeking an experienced Azure Databricks Data Engineer with 8+ years of hands-on experience in data engineering, ETL/ELT, distributed data processing, and large-scale data pipeline development. The ideal candidate will have strong expertise in Databricks, Apache Spark, Scala, PySpark, SQL, Azure, and data lakehousearchitectures, with proven experience designing and optimizing batch and streaming data solutions. The role involves developing scalable data ingestion, transformation, and processing pipelines using Azure Databricks, implementing Delta Lake and Medallion architectures, optimizing Spark workloads, and supporting high-volume streaming environments. The candidate will also work closely with DevOps teams to troubleshoot pipelines, Azure services, AKS environments, and production Databricks workloads.
Key Responsibilities:
Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Databricks, Spark, Scala, PySpark, and SQL.Develop complex data ingestion, transformation, aggregation, and processing workflows for large-volume structured and unstructured datasets.
Build and optimize Databricks notebooks, Spark jobs, DataFrames, Datasets, and SQL-based solutions.
Implement Delta Lake, Unity Catalog, Auto Loader, Lakehouse, and Medallion architecture patterns.
Develop real-time and near-real-time data pipelines using Spark Structured Streaming.
Design streaming solutions capable of processing high-volume data with appropriate performance, scalability, fault tolerance, and data-quality controls.
Configure and optimize Databricks clusters, including sizing, autoscaling, compute configuration, and workload optimization.
Develop and manage Databricks job orchestration, scheduling, dependencies, retries, alerts, and production workflows.
Perform Spark, Scala/PySpark, and SQL performance tuning, including optimization of transformations, joins, partitioning, caching, file formats, and query execution.
Design and implement enterprise data lake and data warehouse/lakehouse solutions using Azure and Databricks.
Develop data pipelines using Databricks Serverless capabilities where appropriate.
Monitor, debug, troubleshoot, and resolve issues in live/production Databricks jobs and data pipelines.
Required Qualifications:
Must have atleast 8+ years of extensive Data engineering experience in Databricks, ETL/ELT using data pipelines, SQL/Procesurr experience Must have 8 years of hands on development proficiency in implementing Databricks solutions using scala and spark with data frames and notebooks/sqlMust have 8+ years of hands on development and performance tuning/ enhancement experience in scala/pyspark and sqlMust have strong experience with Databricks, including developing and optimizing spark jobs, data transformations, and data processing workflows Must have experience and a good understanding on the below topics Databricks delta lake storage Unity catalog AutoloaderLakehouse architecture Medallion architecture Databricks serverless options Dataset and dataframe Databricks clusters configuration and sizing Databricks job orchestration Spark structured streaming Must have experience with complex and large volume data streaming and data transformations using Azure Databricks Must have extensive experience in datawarehouse / data lakehouse implementation Must have experience with monitoring, debugging, and resolving issues in kive Databricks jobs Must have strong hands on expertise in troubleshooting devops pipelines, azure, and AKS services The candidate shd be able to provide data ingestion, streaming and transformations solutions and do development work Experience with apache big data implementation with programming proficiency like pyspark, R or Java prior to Databricks is advantageous Good to have experience with containerization technologies such as docker and Kubernetes