Data Engineer with Databricks Experience

Job DescriptionWe are seeking an experienced Data Engineer with strong expertise in Databricks to join our team in Boston, MA. The ideal candidate will have 8+ years of experience in designing, implementing, and optimizing data solutions, including building robust ETL pipelines and advanced analytics platforms. You will work closely with cross-functional teams to manage and process large-scale datasets using Databricks and cloud technologies.Key ResponsibilitiesDesign, develop, and maintain ETL pipelines for data ingestion, processing, and analytics using Databricks.Collaborate with data scientists, analysts, and business stakeholders to deliver end-to-end data solutions.Optimize data workflows and performance, ensuring scalability and efficiency.Develop, test, and deploy SQL and Python scripts for data processing and transformation.Work with cloud platforms, primarily Azure or AWS, to manage large-scale datasets.Implement data governance and security best practices.Perform data validation, cleansing, and quality assurance to ensure accurate results.Troubleshoot and resolve data-related issues.Qualifications8+ years of hands-on experience in data engineering, with expertise in Databricks.Proficiency in Apache Spark, PySpark, and Scala for large-scale data processing.Strong experience with SQL for querying and manipulating databases.Proven experience with cloud platforms like Azure, AWS, or GCP.Experience with Azure Data Lake, Azure Synapse, or similar tools is a plus.Familiarity with data warehousing concepts and ETL processes.Excellent problem-solving skills and ability to work in a collaborative environment.Strong communication skills and ability to work with cross-functional teams