{"schemaVersion":"jobsearcher.job.v1","id":"95ef097d276665e15a45bc6a","url":"https://jobsearcher.com/jobs/95ef097d276665e15a45bc6a","canonicalUrl":"https://jobsearcher.com/jobs/95ef097d276665e15a45bc6a","title":"Sr. Data Engineer (Databricks Engineer)","description":"Mandatory skills –• Databricks experience (primary requirement)• Apache Airflow• Advanced SQL skills• Python• Spark / PySpark• Scala• Experience building and maintaining data pipelines and workflowsKey Responsibilities:● Design, write, test, and deploy data pipelines using PySpark, Scala, SQL, Python● Meet with stakeholders to gather requirements and translate them into scalable data platform solutions● Understanding of Databricks platform and developer tooling to diagnose errors, audit platform activity, and automate updates across pipelines, objects, and integrations● Ability to explain Spark architecture and pipeline behavior to stakeholders to diagnose root causes and recommend solutions● Provide solution architecture across AWS, Databricks, Kubernetes, and Airflow (MWAA), including cross-platform integrations● Manage Databricks platform governance, including Unity Catalog, ACLs, lineage, and data discovery and privacy tooling● Build and maintain Kubernetes containers and containerized utilities supporting deployed data platform services● Apply networking knowledge to troubleshoot connectivity and integration errors across platform components● Perform platform administration: provision and remove access, assess resource utilization, monitor platform health and cost, and evaluate stakeholder requests● Collaborate with engineers, architects, and product managers to drive Core Data platform success; participate in agile/scrum ceremonies● Maintain documentation of platform changes, standards, and pipeline configurations to support data quality and governanceQualifications:● 5+ years of data engineering experience developing and operating large-scale data pipelines● Deep hands-on experience with Databricks and Apache Spark (batch and streaming), including pipeline development in PySpark and/or Scala● Strong understanding of Spark architecture—executors, stages, partitioning, shuffle, and performance tuning—with ability to explain tradeoffs to technical and non-technical stakeholders● Proficiency with Databricks platform tooling (API, SDK, CLI) for automation, auditing, governance, and operational troubleshooting● Proficient in SQL with advanced performance tuning capabilities● Hands-on production experience with Airflow (MWAA) for orchestrating data pipelines● Experience managing Databricks platform governance: ACLs, Unity Catalog, lineage, and access provisioning● Proficiency in Python and at least one additional language (Scala, Kotlin, or SQL-driven pipeline tooling)● Experience designing and optimizing scalable ETL/ELT pipelines integrating diverse structured and unstructured data sources● AWS-primary experience (compute, storage, networking, IAM); experience with other cloud providers is transferable● Proficiency with Docker and Kubernetes for building and maintaining containerized data platform services● Working knowledge of networking concepts to diagnose cross-platform integration and connectivity issues● Familiarity with Snowflake and comparable tooling relative to the Databricks ecosystem● Experience designing and implementing CI/CD and DevOps practices (Git-based workflows)● Experience implementing data quality checks, monitoring, and logging for pipeline reliability● Self-starting problem solver with strong analytical and communication skills; willingness to learn new tooling and trends● Familiar with Scrum and Agile methodologies● Experience with Snowflake is a plus● Bachelor’s Degree in Computer Science, Information Systems, or a related field, or equivalent work experience, Master’s Degree is a plus.","company":"Okaya Infocom","rawCompany":"okaya infocom","city":"Glendale","state":"CA","isRemote":false,"isActive":false,"createdAt":"2026-10-01T23:14:28.102Z","occupations":[{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"},{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"518210","title":"Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services","slug":"computing-infrastructure-providers-data-processing-web-hosting-and-related-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Sr. Data Engineer (Databricks Engineer)","description":"Mandatory skills –• Databricks experience (primary requirement)• Apache Airflow• Advanced SQL skills• Python• Spark / PySpark• Scala• Experience building and maintaining data pipelines and workflowsKey Responsibilities:● Design, write, test, and deploy data pipelines using PySpark, Scala, SQL, Python● Meet with stakeholders to gather requirements and translate them into scalable data platform solutions● Understanding of Databricks platform and developer tooling to diagnose errors, audit platform activity, and automate updates across pipelines, objects, and integrations● Ability to explain Spark architecture and pipeline behavior to stakeholders to diagnose root causes and recommend solutions● Provide solution architecture across AWS, Databricks, Kubernetes, and Airflow (MWAA), including cross-platform integrations● Manage Databricks platform governance, including Unity Catalog, ACLs, lineage, and data discovery and privacy tooling● Build and maintain Kubernetes containers and containerized utilities supporting deployed data platform services● Apply networking knowledge to troubleshoot connectivity and integration errors across platform components● Perform platform administration: provision and remove access, assess resource utilization, monitor platform health and cost, and evaluate stakeholder requests● Collaborate with engineers, architects, and product managers to drive Core Data platform success; participate in agile/scrum ceremonies● Maintain documentation of platform changes, standards, and pipeline configurations to support data quality and governanceQualifications:● 5+ years of data engineering experience developing and operating large-scale data pipelines● Deep hands-on experience with Databricks and Apache Spark (batch and streaming), including pipeline development in PySpark and/or Scala● Strong understanding of Spark architecture—executors, stages, partitioning, shuffle, and performance tuning—with ability to explain tradeoffs to technical and non-technical stakeholders● Proficiency with Databricks platform tooling (API, SDK, CLI) for automation, auditing, governance, and operational troubleshooting● Proficient in SQL with advanced performance tuning capabilities● Hands-on production experience with Airflow (MWAA) for orchestrating data pipelines● Experience managing Databricks platform governance: ACLs, Unity Catalog, lineage, and access provisioning● Proficiency in Python and at least one additional language (Scala, Kotlin, or SQL-driven pipeline tooling)● Experience designing and optimizing scalable ETL/ELT pipelines integrating diverse structured and unstructured data sources● AWS-primary experience (compute, storage, networking, IAM); experience with other cloud providers is transferable● Proficiency with Docker and Kubernetes for building and maintaining containerized data platform services● Working knowledge of networking concepts to diagnose cross-platform integration and connectivity issues● Familiarity with Snowflake and comparable tooling relative to the Databricks ecosystem● Experience designing and implementing CI/CD and DevOps practices (Git-based workflows)● Experience implementing data quality checks, monitoring, and logging for pipeline reliability● Self-starting problem solver with strong analytical and communication skills; willingness to learn new tooling and trends● Familiar with Scrum and Agile methodologies● Experience with Snowflake is a plus● Bachelor’s Degree in Computer Science, Information Systems, or a related field, or equivalent work experience, Master’s Degree is a plus.","datePosted":"2026-10-01T23:14:28.102Z","dateModified":"2026-10-01T23:14:28.102Z","hiringOrganization":{"@type":"Organization","name":"Okaya Infocom","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Glendale","addressRegion":"CA","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"95ef097d276665e15a45bc6a"},"url":"https://jobsearcher.com/jobs/95ef097d276665e15a45bc6a"}}