Senior Data Engineer / Databricks Specialist

*1. Position Overview* Satsyil Corp is seeking a highly skilled and motivated Senior Data Engineer / Databricks Specialist to join our team. In this role, you will play a crucial part in designing, developing, and optimizing high-performance Spark applications and managing enterprise data infrastructure within AWS Databricks. Utilizing PySpark, Python, SQL, and Delta Lake, you will architect robust ETL/ELT pipelines, maintain medallion architecture, and establish efficient big data solutions supporting mission-critical public sector and enterprise environments. *2. Key Roles & Responsibilities*

• Develop and optimize scalable Spark applications in AWS Databricks using Python, PySpark, and SQL to support data processing and analytics needs.
• Design, build, and maintain robust ETL/ELT pipelines and implement Medallion Architecture (Bronze, Silver, Gold layers) ensuring data integrity, completeness, and lineage.
• Administer Databricks Workspaces, including cluster provisioning, job scheduling, access controls, Unity Catalog, and notebook management.
• Manage AWS cloud infrastructure assets including S3, Lambda, IAM, Aurora, Glue, and provision environments using Terraform Infrastructure-as-Code (IaC).
• Perform query optimization, performance tuning, troubleshooting, and debugging for complex Spark jobs and large-scale data solutions.
• Merge, consolidate, and process structured, semi-structured, and unstructured data sets from diverse API endpoints and databases.
• Embed security gates and compliance controls within DevSecOps CI/CD pipelines (Jenkins, GitLab CI) to support FISMA/FedRAMP requirements, ATO reassessments, and POA&M resolution.
• Implement rigorous data quality controls, anomaly detection, validation rules, and automated lineage tracking across all pipelines.
• Collaborate within cross-functional Agile/Scrum teams in multi-contractor federal environments, providing technical leadership and production on-call support.

*3. Candidate Qualifications & Requirements* *Required Experience & Technical Skills:*

• *Experience:* 3+ years of dedicated hands-on experience in Databricks development.
• *Core Big Data Stack:* Production proficiency with PySpark, Spark SQL, Apache Spark, and Delta Lake.
• *Cloud Ecosystem:* Strong experience in AWS Cloud services including S3, IAM, Lambda, Glue, and Aurora.
• *Automation & DevOps:* Production experience with Terraform (IaC) and CI/CD tools (Jenkins, GitLab CI, or equivalent).
• *Languages & Frameworks:* Deep proficiency in Python and SQL; working knowledge of REST APIs for ingestion.
• *Clearance & Citizenship:* U.S. Citizenship required; must have the ability to obtain and maintain a Public Trust clearance.
• *Methodology & Education:* Hands-on experience in Agile/Scrum delivery environments. Bachelor's degree in CS, IT, Data Science, or related engineering discipline.
• *Preferred Qualifications & Nice-to-Haves:*
• Hands-on experience with Unity Catalog, Delta Live Tables (DLT), and AI/ML pipeline support (MLflow, Model Registry).
• Experience in Federal Government contracting and compliance frameworks (FISMA / FedRAMP environments).
• Active security clearance (Public Trust or higher).

*4. Preferred Certifications* Candidates possessing any of the following certifications will be prioritized:

• *Databricks Certified Data Engineer Associate*
• *Databricks Certified Data Engineer Professional*
• *Databricks Certified Machine Learning Associate*
• *AWS Certified Solutions Architect – Associate or Professional*
• *AWS Certified Data Analytics – Specialty*