Senior Databricks Data Engineer
div:has([data-free-thinking-preview-answer=true])+:is(.text-message,.relative:has(>.text-message))]:-mt-2 grow">Job Summary We are seeking an experienced Senior Databricks Data Engineer with strong expertise in designing, developing, and optimizing large-scale data engineering solutions on the Databricks platform. The role will focus on building enterprise-grade data pipelines, processing high-volume datasets, and delivering scalable analytics solutions using PySpark, Spark SQL, Apache Spark, and AWS cloud technologies.
Key Responsibilities:
Design, develop, and maintain scalable data engineering solutions using Databricks, PySpark, Spark SQL, and Apache Spark.
Build and optimize end-to-end ETL/ELT pipelines integrating Databricks with AWS S3, Snowflake, Airflow, and AWS Glue.
Process and transform large-scale structured and semi-structured datasets to support enterprise reporting, analytics, and regulatory requirements.
Develop reusable frameworks and components for data ingestion, cleansing, transformation, validation, reconciliation, enrichment, and business rule implementation.
Implement robust data quality checks, monitoring, and governance controls to ensure data accuracy and reliability.
Perform performance tuning and optimization of Spark applications and Databricks workloads to improve scalability and processing efficiency.
Collaborate with business stakeholders, architects, and analytics teams to understand data requirements and deliver scalable solutions.
Troubleshoot and resolve production data pipeline issues while maintaining high availability and reliability.
Support advanced analytics, customer segmentation, campaign analytics, AML and regulatory reporting, and data validation initiatives.
Contribute to large-scale data migration and modernization programs.
Support production data platforms and ensure reliable delivery of data engineering solutions.
Required Qualifications:
7+ years of hands-on experience with Databricks and modern data engineering platforms.
Strong expertise in PySpark, Spark SQL, Apache Spark, and Databricks Lakehouse architecture.
Experience building and managing ETL/ELT pipelines on cloud platforms.
Strong hands-on experience with AWS S3, Snowflake, Apache Airflow, and AWS Glue.
Strong knowledge of data modeling, data warehousing, and big data processing techniques.
Experience working with structured and semi-structured data formats, including JSON, Parquet, Avro, and CSV.Hands-on experience implementing data quality frameworks and monitoring solutions.
Strong SQL and data analysis skills.
Experience developing scalable and production-ready data pipelines.
Strong troubleshooting and problem-solving skills for production data engineering environments.
Preferred Qualifications:
- Experience with regulatory reporting, AML, compliance, or financial data domains.
- Exposure to CI/CD and DevOps practices for data engineering.
- Experience with Delta Lake, Databricks Workflows, and Unity Catalog.
- Knowledge of Agile/Scrum delivery methodologies.
- Experience with customer segmentation analytics.
- Experience with campaign analytics.
- Experience supporting AML and regulatory reporting.
- Experience with enterprise data platforms.
- Experience with data validation and reconciliation.
- Experience with large-scale data migration and modernization programs.notes:
- Mandatory AreasMust Have Skills 7+ years of hands-on experience with Databricks and modern data engineering platforms.
- Strong expertise in: PySpark Spark SQL Apache Spark Databricks Lakehouse Architecture
- Experience building and managing ETL/ELT pipelines on cloud platforms.
- Strong experience with: AWS S3 Snowflake Apache Airflow AWS Glue Knowledge of data modeling, data warehousing, and big data processing techniques.
- Domain Experience (If any) - Good to have healthcare experience
- Must have Certifications - None
Location: RemoteOnsite Requirement - RemoteNumber of days onsite - Remote