Senior Data Engineer-Databricks

Overview We are seeking a highly skilled and motivated Senior Data Engineer specializing in Databricks to join our dynamic data team. In this role, you will lead the design, development, and optimization of large-scale data systems, enabling advanced analytics and business intelligence solutions. Your expertise will drive innovative data management strategies across cloud platforms, ensuring seamless integration and high-performance processing of complex datasets. This position offers an exciting opportunity to work with cutting-edge big data technologies and contribute to impactful data-driven decision-making. Responsibilities Design, develop, and maintain scalable ETL pipelines using Databricks, Spark, Python, and SQL to process vast amounts of structured and unstructured data. Collaborate with cross-functional teams to translate business requirements into robust data models and architectures aligned with best practices in dimensional modeling and data warehousing design. Optimize big data systems leveraging Hadoop, Azure Data Lake, AWS, and other cloud-based platforms to ensure high availability and performance. Implement data management integration strategies utilizing Informatica, Talend, RESTful APIs, and other tools to streamline data ingestion and transformation processes. Develop comprehensive documentation for data workflows, models, and pipelines while ensuring adherence to security standards and compliance regulations. Support model training, analysis activities, and business intelligence reporting using Looker, SQL databases such as Oracle or Microsoft SQL Server, and cloud databases. Participate in Agile development cycles to continuously improve data engineering solutions while maintaining flexibility for evolving project needs. Requirements Proven experience in software development with a strong background in data engineering within big data environments. Extensive hands-on expertise with Databricks platform, Spark (including Apache Hive), Hadoop ecosystems, and cloud services such as Azure Data Lake or AWS Public Cloud. Proficiency in programming languages including Python, Java, Bash (Unix shell), Shell Scripting, and VBA for automation and analysis tasks. Deep understanding of ETL pipeline development, SQL query management, database design (including dimensional modeling), and data warehousing concepts. Familiarity with business intelligence tools like Looker or similar platforms for visualization and reporting purposes. Knowledge of data modeling techniques for linked data, Oracle databases, SQL databases, cloud databases, and enterprise data warehouse architecture. Experience working with Big Data systems such as Hadoop or Spark for large-scale processing projects. Strong analytical skills combined with excellent communication abilities to translate complex technical concepts into clear insights. Ability to work effectively within an Agile environment while managing multiple priorities simultaneously. Join us as a Senior Data Engineer-Databricks to shape the future of our enterprise’s data landscape! Your expertise will empower smarter decisions through innovative big data solutions while collaborating with talented professionals dedicated to excellence in analytics and technology innovation. Pay: $110,000.00 - $150,000.00 per year Benefits: 401(k) 401(k) matching Dental insurance Health insurance Health savings account Life insurance Paid time off Parental leave Retirement plan Vision insurance Work Location: Remote