Senior Data Engineer – Databricks AWS
ARCHIVED
We can't find an active application page for this role right now. It may reopen or be listed elsewhere. Use Next Steps to search for an active apply link and similar live jobs.
*Senior Data Engineer – AWS + Databricks Lakehouse*
*Healthcare / Life Sciences*
*Hybrid – Indianapolis, IN *
The role involves close collaboration with data engineering teams, analytics stakeholders, and senior leadership on enterprise-wide data modernization initiatives. Requires strong expertise in Databricks Lakehouse architecture, PySpark-based engineering, cloud-native data processing, data governance, and enterprise-scale ETL/ELT modernization in regulated environments.
*Key Responsibilities*
_AWS + Databricks Lakehouse Engineering_
- Design, develop, and optimize scalable data pipelines and Lakehouse architectures using AWS and Databricks
- Build enterprise-grade ETL/ELT workflows for ingestion, transformation, and analytics
- Develop scalable PySpark processing frameworks for structured and unstructured data
- Implement Delta Lake best practices: partitioning, optimization, schema evolution, performance tuning
- Support Medallion architecture (Bronze, Silver, Gold layers)
_Data Ingestion & Processing_
- Build ingestion pipelines for RWD, healthcare, operational, and analytics datasets
- Develop reusable ingestion and transformation frameworks
- Support batch and incremental processing workloads
- Optimize Spark workloads, cluster utilization, and pipeline performance
_AWS Cloud Integration_
- Work with AWS-native services: S3, Glue, Lambda, IAM, CloudWatch, Redshift
- Design secure, scalable integration patterns between Databricks and AWS
- Support cloud resource optimization and cost governance
_Governance, Security & Compliance_
- Implement governance standards using Unity Catalog and RBAC
- Ensure HIPAA/PHI compliance for healthcare data handling
- Support data lineage, auditability, and access controls
_Performance Optimization & Reliability_
- Optimize workloads via partitioning, caching, ZORDER, cluster sizing, autoscaling
- Improve pipeline resiliency, monitoring, alerting
- Support CI/CD and deployment best practices
_Collaboration & Leadership_
- Translate business requirements into technical solutions with teams
- Establish coding standards and reusable frameworks
- Mentor junior engineers; facilitate documentation and knowledge transfer
- Provide status updates to senior leadership
*Required Skills & Experience*
_Core Technical_
- Hands-on Databricks Lakehouse platform experience
- PySpark and Apache Spark optimization
- Delta Lake and Medallion architecture
- Scalable ETL/ELT framework design
- Strong SQL and data modeling
- Large-scale enterprise data platform experience
_AWS_
- AWS cloud ecosystem and data engineering services
- S3, Glue, Lambda, IAM, Redshift, CloudWatch
- Secure cloud architecture and integration patterns
_Governance & Enterprise Architecture_
- Enterprise governance and security models
- Unity Catalog and RBAC
- Regulated environment experience
- Healthcare/life sciences dataset familiarity preferred
*Preferred Qualifications*
- Databricks Certification (Associate/Professional)
- Enterprise modernization initiative experience
- Healthcare/Pharma/RWD/HEOR analytics experience
- Strong stakeholder communication and leadership
- Experience mentoring engineering teams
Application Question(s):
- Are you able to work in a hybrid in Indianapolis, IN?
- How many years of hands-on experience do you have with Databricks?
- What hands-on experience do you have with Delta Lake and Medallion architecture?
- How many years of hands-on PySpark/Spark experience do you have, and what is the largest data volume you have worked with?
- Have you worked with Unity Catalog, RBAC, data lineage, or governance for sensitive/regulated data?
- Have you worked with healthcare, life sciences, pharma, RWD, HEOR, claims, EHR, or PHI data? If yes, what type of data and what was your role?
- How many years of hands-on AWS data engineering experience do you have, and which AWS services have you worked with, such as S3, Glue, Lambda, IAM, Redshift, or CloudWatch?
- Total work experience - This is a Senior Role, which requires minimum 10 plus years work experience. This is a W2 contract role.
Work Location: Hybrid remote in Indianapolis, IN 46201