Senior Data Engineer - Full Stack
Senior Data Engineer – Full StackWe are seeking a highly skilled Senior Data Engineer – Full Stack to build and maintain internal tools, automation frameworks, and workflows that enhance the efficiency, reliability, and scalability of our data and machine learning platforms. This role will work closely with Data Engineers, Data Scientists, and ML Engineers to streamline operations across the data lifecycle.Total Exp: 8+ YearsLocation: Santa Clara, CA (Hybrid)Key ResponsibilitiesDesign and develop CLI tools, scripts, and internal utilities to automate repetitive tasks across the data platform, including:Pipeline execution and orchestrationData governance workflowsMetadata synchronizationEnvironment setup and configurationTest harness developmentAutomate workflows on Databricks, including:Job deployment and schedulingEnvironment provisioningMLOps processes using APIs, Terraform, or Databricks SDKBuild and implement robust testing frameworks:Integration testing for pipelinesEnd-to-end validation of ETL/ELT workflowsTesting and validation for ML inference workflowsImprove overall productivity, scalability, and reliability of the data and ML engineering ecosystemDevelop lightweight internal tools and dashboards using frameworks such as React, Streamlit, or similar technologies to:Visualize data pipelines and workflowsDemonstrate model inference capabilitiesProvide configuration and operational controlsEnable internal productivity monitoring and dashboardsCollaborate with cross-functional teams to identify automation opportunities and implement best practicesRequired Skills & QualificationsStrong experience in Python and scripting for automation and backend developmentHands-on experience with Databricks platform and ecosystemExperience with APIs, Terraform, and/or Databricks SDK for automationSolid understanding of ETL/ELT pipelines and data platform architectureExperience building testing frameworks for data pipelines and ML workflowsFamiliarity with CLI tool development and system automationKnowledge of MLOps principles and practicesExperience with modern development practices, including:Spec-driven developmentUse of coding agents or automation-assisted development toolsVersion control and CI/CD pipelinesNice to HaveExperience building dashboards or internal tools using React, Streamlit, or similar frameworksFamiliarity with Databricks AI/BI or other data visualization toolsExposure to data governance and metadata management frameworksExperience working with cloud platforms (AWS preferred)Preferred Experience8+ years of experience in Data Engineering, Platform Engineering, or related rolesExperience working in data-driven or ML-focused environmentsWhat You'll BringStrong problem-solving mindset with a focus on automation and efficiencyAbility to work in a fast-paced, collaborative environmentPassion for building scalable internal tools and improving developer productivity