Data Platform Engineer

Position Description: Senior Cloud Systems EngineerStart Date – End Date: ASAP– 12/31/2026Note: Period of performance may extend beyond 12/31/2026.Hands-on experience administering Databricks (workspace administration,clusters/compute policies, jobs, SQL warehouses, repos, runtime management) andexpertise using Databricks CLI.• Strong Unity Catalog administration: metastores; catalogs/schemas; grants; serviceprincipals; external locations; storage credentials; governed storage access.• Identity & Access Management proficiency: SSO concepts, SCIM provisioning, group-based RBAC, service principals, least-privilege patterns.• Security fundamentals: secrets management, secure connectivity, audit logging, accessmonitoring, and evidence-ready operations.• Cloud platform expertise (AWS ): IAM roles/policies, object storage security patterns,networking basics (VPC concepts), logging/monitoring integration.• Automation skills: scripting and/or IaC using Terraform/CLI/REST APIs for repeatableconfiguration and environment promotion.• Experience implementing data governance controls (classification/tagging,lineage/metadata integrations) in partnership with governance teams.• CI/CD practices for jobs/notebooks/config promotion across SDLC environments.• Understanding of lakehouse concepts (e.g., Delta, table lifecycle management, separationof storage/compute).• SQL proficiency and data engineering fundamentals for troubleshooting queryperformance issues, understanding ETL/ELT workflow patterns, and debugging datapipeline failures; basic Python/Scala familiarity for notebook/code issue diagnosis.• Experience with compliance and regulatory frameworks (FedRAMP, HIPAA, SOC2, orsimilar) including implementation of data residency requirements, retention policies, andaudit-ready evidence collection.• Hands-on experience with AWS security and networking services including PrivateLink,Secrets Manager/Systems Manager integration, CloudWatch/CloudTrail integration, S3bucket policies, cross-account access patterns, and KMS encryption key management.• Experience administering Databricks serverless compute, Workspace Git integrations(GitLab), Databricks Asset Bundles (DABs) for deployment automation, and modernworkspace features supporting DevOps workflows.• SLA/SLO management and stakeholder communication skills; ability to define platformservice levels, produce operational reports, translate technical issues to businessstakeholders, and manage vendor relationships (Databricks account teams).Education / Experience/Certifications/Accreditations• Bachelor’s degree in a related field or equivalent practical experience.• 7+ years in cloud/data platform administration and operations, including 4+ yearssupporting Databricks or similar platforms.• Databricks Platform Administrator/Databricks AWS Platform Architect• Databricks Certified Data Engineer Associate/Professional• AWS Certified Solutions Architect Associate or ProfessionalThe Contractor shall deliver, but not limited to, the following:• Administer Databricks account and workspaces across SDLC environments; standardizeconfiguration, naming, and operational patterns.• Configure and maintain clusters/compute, job compute, SQL warehouses, runtimeversions, libraries, repos, and workspace settings.• Implement platform monitoring/alerting, operational dashboards, and health checks;maintain runbooks and operational procedures.• Provide Tier 2/3 operational support: troubleshoot incidents, perform root-cause analysis,and drive remediation and preventive actions.• Manage change control for upgrades, feature rollouts, configuration changes, andintegration changes; document impacts and rollback plans.• Enforce least privilege across platform resources (workspaces, jobs, clusters, SQLwarehouses, repos, secrets) using role/group-based access patterns.• Configure and manage secrets and secure credential handling (secret scopes / keymanagement integrations) for platform and data connectivity.• Enable and maintain audit logging and access/event visibility; support security reviewsand evidence requests.• Administer Unity Catalog governance: metastores, catalogs/schemas/tables, ownership,grants, and environment/domain patterns.• Configure and manage external locations, storage credentials, and governed access tocloud object storage.• Partner with governance stakeholders to support metadata/lineage integration,classification/tagging, and retention controls where applicable.• Coordinate secure connectivity and guardrails with cloud/network teams: privateconnectivity patterns, egress controls, firewall/proxy needs.• Configure cloud integrations required for governed data access and service connectivity(roles/permissions, endpoints, storage integrations).• Implement cost guardrails: cluster policies, auto-termination, scheduling, workload sizingstandards, and capacity planning.• Produce usage/cost insights and optimization recommendations; address waste drivers(idle compute, oversized clusters, inefficient jobs).• Automate administration and configuration using APIs/CLI/IaC (e.g., Terraform) toreduce manual drift and improve repeatability.• Maintain platform documentation: configuration baselines, security/governancestandards, onboarding guides, and troubleshooting references.• Design and implement backup and disaster recovery procedures for workspaceconfigurations, notebooks, Unity Catalog metadata, and job definitions; maintainrecovery runbooks and perform periodic DR testing aligned to RTO/RPO objectives.• Monitor and optimize platform performance, including SQL warehouse query tuning,cluster autoscaling configuration, Photon enablement, and Delta Lake optimizationguidance (OPTIMIZE, VACUUM, Z-ordering strategies).• Administer Delta Live Tables (DLT) pipelines and coordinate with data engineeringteams on pipeline health, data quality monitoring, failed job remediation, and pipelineconfiguration best practices.• Manage third-party integrations and ecosystem connectivity, including BI toolintegrations (e.g., Power BI), and external metadata catalog integrations.• Implement Databricks Asset Bundles (DABs) for standardized deployment patterns;automate workspace resource deployment (jobs, pipelines, dashboards) across SDLCenvironments using bundle-based CI/CD workflows.• Conduct capacity planning and scalability analysis, including forecasting concurrentuser/workload growth, platform scaling strategies, and proactive resource allocationduring peak usage periods.• Facilitate user onboarding and enablement, including new user/team onboarding procedures, training coordination, workspace access provisioning, and creation of self-service documentation/guides