Senior Data Engineer
Senior Data EngineerLocation: Glendale, CA (2-3 days/week)Type of Hire: ContractRate: $90.60As a Senior Data Engineer, you will be pivotal in transforming data into actionable insights. Collaborate with our dynamic team of technologists to develop cutting-edge data solutions that drive innovation and fuel business growth. You will own and operate the Core Data platform on Databricks, delivering reliable batch and streaming Spark pipelines, platform governance, and solution architecture across AWS, Kubernetes, and Airflow. Your expertise will be essential in explaining Spark architecture, recommending best-fit Databricks solutions to stakeholders, and optimizing our data-driven decision-making processes. If you're passionate about leveraging data to make a tangible impact, we welcome you to join us in shaping the future of our organization.Key Responsibilities:Design, write, test, and deploy data pipelines using PySpark, Scala, SQL, PythonMeet with stakeholders to gather requirements and translate them into scalable data platform solutionsUnderstanding of Databricks platform and developer tooling to diagnose errors, audit platform activity, and automate updates across pipelines, objects, and integrationsAbility to explain Spark architecture and pipeline behavior to stakeholders to diagnose root causes and recommend solutionsProvide solution architecture across AWS, Databricks, Kubernetes, and Airflow (MWAA), including cross-platform integrationsManage Databricks platform governance, including Unity Catalog, ACLs, lineage, and data discovery and privacy toolingBuild and maintain Kubernetes containers and containerized utilities supporting deployed data platform servicesApply networking knowledge to troubleshoot connectivity and integration errors across platform componentsPerform platform administration: provision and remove access, assess resource utilization, monitor platform health and cost, and evaluate stakeholder requestsCollaborate with engineers, architects, and product managers to drive Core Data platform success; participate in agile/scrum ceremoniesMaintain documentation of platform changes, standards, and pipeline configurations to support data quality and governanceQualifications:5+ years of data engineering experience developing and operating large-scale data pipelinesDeep hands-on experience with Databricks and Apache Spark (batch and streaming), including pipeline development in PySpark and/or ScalaStrong understanding of Spark architecture—executors, stages, partitioning, shuffle, and performance tuning—with ability to explain tradeoffs to technical and non-technical stakeholdersProficiency with Databricks platform tooling (API, SDK, CLI) for automation, auditing, governance, and operational troubleshootingProficient in SQL with advanced performance tuning capabilitiesHands-on production experience with Airflow (MWAA) for orchestrating data pipelinesExperience managing Databricks platform governance: ACLs, Unity Catalog, lineage, and access provisioningProficiency in Python and at least one additional language (Scala, Kotlin, or SQL-driven pipeline tooling)Experience designing and optimizing scalable ETL/ELT pipelines integrating diverse structured and unstructured data sourcesAWS-primary experience (compute, storage, networking, IAM); experience with other cloud providers is transferableProficiency with Docker and Kubernetes for building and maintaining containerized data platform servicesWorking knowledge of networking concepts to diagnose cross-platform integration and connectivity issuesFamiliarity with Snowflake and comparable tooling relative to the Databricks ecosystemExperience designing and implementing CI/CD and DevOps practices (Git-based workflows)Experience implementing data quality checks, monitoring, and logging for pipeline reliabilitySelf-starting problem solver with strong analytical and communication skills; willingness to learn new tooling and trendsFamiliar with Scrum and Agile methodologiesExperience with Snowflake is a plusBachelor’s Degree in Computer Science, Information Systems, or a related field, or equivalent work experience, master’s degree is a plusRequired Skills (Must-Have)• Databricks experience (primary requirement)• Apache Airflow• Advanced SQL skills• Python• Spark / PySpark• Scala• Experience building and maintaining data pipelines and workflowsRequired Education: BA/BS