Databricks Engineer

Job Title : Databricks Engineer – Data Operations & Production SupportJob Location : Seattle, WA (Hybrid)Job Duration :Long-term Contract(12+ months)Pay rate : $55-60/hr on W2 & $65-70/hr on C2CJob Description:We are seeking a highly skilled Databricks Engineer to support and maintain enterprise data platforms and large-scale data processing environments. This role is responsible for ensuring the reliability, performance, and availability of data pipelines built on Databricks, Apache Airflow, and PySpark. The ideal candidate will possess strong production support experience and the ability to troubleshoot and optimize distributed data workloads.Key ResponsibilitiesDatabricks & Spark OperationsSupport and administer Databricks-based data platforms.Monitor and maintain Spark and PySpark processing jobs.Identify and resolve performance bottlenecks and processing failures.Optimize workloads for improved scalability and resource utilization.Workflow ManagementManage and monitor Apache Airflow workflows and DAG executions.Ensure successful and timely completion of scheduled data pipelines.Troubleshoot workflow failures and implement corrective actions.Production SupportProvide L2/L3 support for enterprise data processing applications.Perform root cause analysis for production incidents.Implement permanent solutions for recurring issues.Participate in on-call support and incident management activities.Data Quality & ReliabilityValidate and reconcile datasets to ensure accuracy and consistency.Support data quality controls and monitoring frameworks.Maintain high availability of business-critical data pipelines.Unix/Linux AdministrationPerform log analysis and troubleshooting activities.Manage file systems and batch execution processes.Utilize shell scripting and Unix commands to support operations.Collaboration & DocumentationPartner with Data Engineering, QA, and business teams.Support enhancements and continuous improvement initiatives.Maintain operational documentation, runbooks, and knowledge repositories.Mandatory SkillsCandidates must have:4+ years of experience with Databricks and Spark/PySparkStrong hands-on experience with Apache Airflow (DAG development, scheduling, monitoring, and troubleshooting)Proficiency in Python and SQLExperience providing L2/L3 production support for data platformsStrong understanding of ETL/ELT frameworks and data warehousing conceptsExperience troubleshooting large-scale distributed processing environmentsKnowledge of Spark performance tuning and optimization techniquesHands-on experience with Unix/Linux commands, shell scripting, and log analysisExperience with incident management and root cause analysisStrong analytical and problem-solving skillsPreferred SkillsExperience with Azure Databricks or AWS DatabricksKnowledge of Azure Data Services or Microsoft FabricExposure to data governance and data quality frameworksExperience with monitoring and observability toolsUnderstanding of CI/CD and DevOps practicesFamiliarity with cloud platforms such as Azure or AWSIdeal BackgroundThis position is best suited for candidates with a strong support and operations mindset who have extensive experience managing Databricks, Airflow, and PySpark environments in production and ensuring the stability of enterprise data pipelines.Regards,Vishwajeet Verma