Senior Databricks Data Engineer
Who We Are:
Artmac Soft is a technology consulting and service-oriented IT company dedicated to providing innovative technology solutions and services to customers.
Job Description
Job Title: Senior Databricks Data Engineer
Job Type: C2C / W2Experience : 8-15 Years
Location: RemoteRole Summary
We are looking for an experienced Senior Databricks Data Engineer to design, develop, and optimize scalable data engineering solutions using Databricks, Apache Spark, PySpark, SQL, and Delta Lake. The ideal candidate will have strong expertise in modern data architecture, Lakehouse design, ETL/ELT pipelines, data integration, and cloud-based data platforms.
Key Responsibilities:
- Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Apache Spark, and SQL.Build and optimize ETL/ELT workflows to process large volumes of structured and semi-structured data.
- Implement Lakehouse solutions using Delta Lake and Medallion Architecture (Bronze, Silver, and Gold layers).Develop batch and streaming data processing solutions to support business and analytics requirements.
- Optimize Spark jobs, cluster configurations, partitioning, and data storage to improve performance and control processing costs.
- Integrate data from multiple sources, including databases, APIs, cloud storage, and enterprise applications.
- Implement data validation, reconciliation, error handling, and monitoring to ensure data accuracy and reliability.
- Work with cloud platforms such as Microsoft Azure, AWS, or Google Cloud, depending on project requirements.
- Implement data governance, access controls, and secure data management using Unity Catalog and related platform capabilities.
- Develop automated workflows, job scheduling, and dependency management for production data pipelines.
- Troubleshoot production issues, identify root causes, and implement long-term solutions.
- Required Skills And Experience8 15 years of experience in data engineering, big data development, or related roles.
- Strong hands-on experience with Databricks and Apache Spark.
- Advanced programming skills in Python and PySpark.
- Strong SQL skills, including complex queries, joins, aggregations, and query optimization.
- Experience building production-grade ETL/ELT pipelines and data integration workflows.
- Hands-on experience with Delta Lake, Delta tables, and Lakehouse architecture.
- Understanding of Medallion Architecture and scalable data modeling practices.
- Experience with at least one major cloud platform: Azure, AWS, or GCP.Experience with data pipeline orchestration, scheduling, monitoring, and troubleshooting.
- Knowledge of Git, CI/CD pipelines, automated testing, and deployment practices.
- Strong understanding of data quality, data governance, security, and performance optimization
- Familiarity with Terraform, infrastructure as code, or cloud resource management.
- Experience working with enterprise-scale data platforms and high-volume datasets.
- Databricks certifications or relevant cloud data engineering certifications.
Qualification: Bachelor's degree or equivalent combination of education and experience.