Senior Databricks Platform & Data Engineer
A senior Databricks Platform Engineer who is also a strong, hands-on Databricks Data Engineer. The candidate must be able to design, configure, secure and govern enterprise Databricks environments while personally developing and optimizing production-grade batch and streaming data pipelines. This is not a purely advisory architect, platform administrator or governance-only resource.Databricks platform engineering requirements:Design and deploy enterprise-scale Databricks Lakehouse environmentsConfigure and administer Databricks workspaces, networking, security and private connectivityImplement Unity Catalog, RBAC, data lineage, auditability and multi-workspace governanceEstablish cluster policies, workload-management standards and platform guardrailsImplement CI/CD, Infrastructure as Code, DevOps automation and environment-promotion processesOptimize platform performance, scalability, reliability and costStrong AWS and Azure Databricks experience; GCP is preferred but not requiredHands-on data engineering requirements:Strong development experience using Python, PySpark and/or ScalaDesign, build and optimize production-grade batch and streaming pipelinesStrong Apache Spark development, troubleshooting and performance tuningDeep Delta Lake experience, including architecture, optimization and reliable data-processing patternsBuild scalable ingestion, transformation and orchestration frameworksDevelop reusable data products and domain-oriented pipelinesImplement data quality, validation, observability, monitoring and operational-support frameworksLead use-case delivery from requirements gathering and data modeling through development, testing and production deploymentTroubleshoot failed jobs, pipeline bottlenecks, data-quality issues and workload-performance problemsExperience supporting real-time analytics, reporting and machine-learning workloadsModern Databricks capabilities desired:Lakeflow/Declarative PipelinesAI/BI Dashboards and GenieModel Serving and ML deploymentSelf-service analytics and reusable ingestion frameworksClient-facing requirements:Lead architecture workshops, solution-design sessions and platform-strategy discussionsTranslate business requirements into technical architectures and delivery plansCommunicate effectively with data, architecture, security, risk, compliance and business stakeholdersLead technical teams and support client solutioning and proposal activities