Databricks Data Engineer

Company DescriptionAlephys specializes in delivering data products, managed services, and consulting solutions to help organizations unlock the full potential of their data. Our expertise lies in creating innovative, data-driven strategies and technologies that empower businesses to make informed decisions. By providing tailored solutions, Alephys enables companies to achieve operational excellence and maximize their data value. We are committed to fostering collaboration and innovation to meet the unique needs of our clients and drive sustainable growth.About the RoleWe are seeking a highly skilled Databricks Data Engineer to design, build, and optimize our next-generation data architecture. In this role, you will be instrumental in modernizing our data infrastructure, transitioning legacy ETL workloads to highly scalable, cloud-native solutions, and ensuring our data assets are secure, discoverable, and performant. You will leverage the latest capabilities of the Databricks platform—from declarative pipelines to advanced governance—to deliver high-quality data products to the business.Key ResponsibilitiesPipeline Engineering: Design, build, and maintain robust, scalable ETL/ELT pipelines using PySpark and Spark SQL to process large volumes of structured and semi-structured data.Declarative Frameworks: Implement and manage Spark Declarative Pipelines (Delta Live Tables) to simplify pipeline development, automate data quality checks, and streamline operations.Legacy Modernization: Lead the migration of complex workloads from legacy enterprise ETL systems into modernized, scalable PySpark architectures.Data Governance & Security: Architect and enforce centralized data governance, access controls, auditing, and data lineage tracking across the organization utilizing Unity Catalog.CI/CD & Automation: Automate the deployment lifecycle of data pipelines, notebooks, and infrastructure using DABs integrated with enterprise CI/CD workflows.Performance Optimization: Profile, tune, and optimize complex PySpark jobs and Spark SQL queries for maximum performance and cost-efficiency.Required Skills & QualificationsExperience: 5+ years of experience in Data Engineering, with a heavy focus on the Databricks Data Intelligence Platform.Core Languages: Expert-level proficiency in Python (PySpark) and advanced Spark SQL.Databricks Ecosystem: Deep hands-on experience with modern Databricks features, specifically Spark Declarative Pipelines and Unity Catalog for centralized governance.DevOps / DataOps: Proven ability to implement CI/CD pipelines for data assets using DABs, Git, and automation tools (e.g., GitHub Actions, Jenkins, or GitLab CI).Architecture & Migration: Strong understanding of distributed computing principles and experience migrating legacy on-premise ETL logic (e.g., DataStage, Informatica) to cloud-native Spark environments.Problem Solving: Strong analytical skills with the ability to troubleshoot complex data processing issues and optimize massive data transformations.Nice-to-HavesExperience with event-driven data ingestion architectures and storage-layer triggers (e.g., S3).Familiarity with cloud infrastructure (AWS, Azure, or GCP) and infrastructure-as-code (Terraform).Experience building and designing internal frameworks to accelerate data pipeline development.