{"schemaVersion":"jobsearcher.job.v1","id":"fb2979d19dd5b27e763cb220","url":"https://jobsearcher.com/jobs/fb2979d19dd5b27e763cb220","canonicalUrl":"https://jobsearcher.com/jobs/fb2979d19dd5b27e763cb220","title":"Data Engineer","description":"Job Title\r\nData EngineerJob Type\r\nFull TimeClient Location\r\nQuincy,MAWork Arrangement\r\nOnsiteDuration\r\nFull TimePay Rate / Salary\r\n$120-$135k/yr.Job SummaryWe are seeking an experienced Azure Databricks Data Engineer with 8+ years of hands-on experience in data engineering, ETL/ELT, distributed data processing, and large-scale data pipeline development.The ideal candidate will have strong expertise in Databricks,Apache Spark, Scala, PySpark, SQL, Azure, and data lakehousearchitectures, with proven experience designing and optimizing batch and streaming data solutions.The role involves developing scalable data ingestion, transformation, and processing pipelines using Azure Databricks, implementing Delta Lake and Medallion architectures, optimizing Spark workloads, and supporting high-volume streaming environments.The candidate will also work closely with DevOps teams to troubleshoot pipelines, Azure services, AKS environments, and production Databricks workloads.Key ResponsibilitiesDesign, develop, and maintain scalable ETL/ELT data pipelines using Azure Databricks, Spark, Scala, PySpark, and SQL.Develop complex data ingestion, transformation, aggregation, and processing workflows for large-volume structured and unstructured datasets.Build and optimize Databricks notebooks, Spark jobs, DataFrames, Datasets, and SQL-based solutions.Implement Delta Lake, Unity Catalog, Auto Loader, Lakehouse, and Medallion architecture patterns.Develop real-time and near-real-time data pipelines using Spark Structured Streaming.Design streaming solutions capable of processing high-volume data with appropriate performance, scalability, fault tolerance, and data-quality controls.Configure and optimize Databricks clusters, including sizing, autoscaling, compute configuration, and workload optimization.Develop and manage Databricks job orchestration, scheduling, dependencies, retries, alerts, and production workflows.Perform Spark, Scala/PySpark, and SQL performance tuning, including optimization of transformations, joins, partitioning, caching, file formats, and query execution.Design and implement enterprise data lake and data warehouse/lakehouse solutions using Azure and Databricks.Develop data pipelines using Databricks Serverless capabilities where appropriate.Monitor, debug, troubleshoot, and resolve issues in live/production Databricks jobs and data pipelines.Required QualificationsMust have atleast 8+ years of extensive Data engineering experience in Databricks, ETL/ELT using data pipelines, SQL/Procesurr experienceMust have 8 years of hands on development proficiency in implementing Databricks solutions using scala and spark with data frames and notebooks/sqlMust have 8+ years of hands on development and performance tuning/ enhancement experience in scala/pyspark and sqlMust have strong experience with Databricks, including developing and optimizing spark jobs, data transformations, and data processing workflowsMust have experience and a good understanding on the below topicsDatabricks delta lake storageUnity catalogAutoloaderLakehouse architectureMedallion architectureDatabricks serverless optionsDataset and dataframeDatabricks clusters configuration and sizingDatabricks job orchestrationSpark structured streamingMust have experience with complex and large volume data streaming and data transformations using Azure DatabricksMust have extensive experience in datawarehouse / data lakehouse implementationMust have experience with monitoring, debugging, and resolving issues in kive Databricks jobsMust have strong hands on expertise in troubleshooting devops pipelines, azure , and AKS servicesThe candidate shd be able to provide data ingestion, streaming and transformations solutions and do development workExperience with apache big data implementation with programming proficiency like pyspark, R or Java prior to Databricks is advantageousGood to have experience with containerization technologies such as docker and Kubernetes#J-18808-Ljbffr","company":"Sfe","rawCompany":"sfe","city":"Quincy","state":"MA","isRemote":false,"isActive":false,"createdAt":"2026-10-14T01:32:35.703Z","occupations":[{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-1243.00","title":"Database Architects","slug":"database-architects"},{"code":"15-2051.00","title":"Data Scientists","slug":"data-scientists"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Data Engineer","description":"Job Title\r\nData EngineerJob Type\r\nFull TimeClient Location\r\nQuincy,MAWork Arrangement\r\nOnsiteDuration\r\nFull TimePay Rate / Salary\r\n$120-$135k/yr.Job SummaryWe are seeking an experienced Azure Databricks Data Engineer with 8+ years of hands-on experience in data engineering, ETL/ELT, distributed data processing, and large-scale data pipeline development.The ideal candidate will have strong expertise in Databricks,Apache Spark, Scala, PySpark, SQL, Azure, and data lakehousearchitectures, with proven experience designing and optimizing batch and streaming data solutions.The role involves developing scalable data ingestion, transformation, and processing pipelines using Azure Databricks, implementing Delta Lake and Medallion architectures, optimizing Spark workloads, and supporting high-volume streaming environments.The candidate will also work closely with DevOps teams to troubleshoot pipelines, Azure services, AKS environments, and production Databricks workloads.Key ResponsibilitiesDesign, develop, and maintain scalable ETL/ELT data pipelines using Azure Databricks, Spark, Scala, PySpark, and SQL.Develop complex data ingestion, transformation, aggregation, and processing workflows for large-volume structured and unstructured datasets.Build and optimize Databricks notebooks, Spark jobs, DataFrames, Datasets, and SQL-based solutions.Implement Delta Lake, Unity Catalog, Auto Loader, Lakehouse, and Medallion architecture patterns.Develop real-time and near-real-time data pipelines using Spark Structured Streaming.Design streaming solutions capable of processing high-volume data with appropriate performance, scalability, fault tolerance, and data-quality controls.Configure and optimize Databricks clusters, including sizing, autoscaling, compute configuration, and workload optimization.Develop and manage Databricks job orchestration, scheduling, dependencies, retries, alerts, and production workflows.Perform Spark, Scala/PySpark, and SQL performance tuning, including optimization of transformations, joins, partitioning, caching, file formats, and query execution.Design and implement enterprise data lake and data warehouse/lakehouse solutions using Azure and Databricks.Develop data pipelines using Databricks Serverless capabilities where appropriate.Monitor, debug, troubleshoot, and resolve issues in live/production Databricks jobs and data pipelines.Required QualificationsMust have atleast 8+ years of extensive Data engineering experience in Databricks, ETL/ELT using data pipelines, SQL/Procesurr experienceMust have 8 years of hands on development proficiency in implementing Databricks solutions using scala and spark with data frames and notebooks/sqlMust have 8+ years of hands on development and performance tuning/ enhancement experience in scala/pyspark and sqlMust have strong experience with Databricks, including developing and optimizing spark jobs, data transformations, and data processing workflowsMust have experience and a good understanding on the below topicsDatabricks delta lake storageUnity catalogAutoloaderLakehouse architectureMedallion architectureDatabricks serverless optionsDataset and dataframeDatabricks clusters configuration and sizingDatabricks job orchestrationSpark structured streamingMust have experience with complex and large volume data streaming and data transformations using Azure DatabricksMust have extensive experience in datawarehouse / data lakehouse implementationMust have experience with monitoring, debugging, and resolving issues in kive Databricks jobsMust have strong hands on expertise in troubleshooting devops pipelines, azure , and AKS servicesThe candidate shd be able to provide data ingestion, streaming and transformations solutions and do development workExperience with apache big data implementation with programming proficiency like pyspark, R or Java prior to Databricks is advantageousGood to have experience with containerization technologies such as docker and Kubernetes#J-18808-Ljbffr","datePosted":"2026-10-14T01:32:35.703Z","dateModified":"2026-10-14T01:32:35.703Z","hiringOrganization":{"@type":"Organization","name":"Sfe","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Quincy","addressRegion":"MA","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"fb2979d19dd5b27e763cb220"},"url":"https://jobsearcher.com/jobs/fb2979d19dd5b27e763cb220"}}