Lead Databricks Engineer
Who We Are:
Artmac Soft is a technology consulting and service-oriented IT company dedicated to providing innovative technology solutions and services to customers.
Job Description
Job Title: Lead Databricks Engineer
Job Type: C2C / W2Experience : 8-15 Years
Location: RemoteRole Summary
We are seeking an experienced Lead Databricks Engineer to lead the design, development, and implementation of scalable, high-performance data engineering solutions using the Databricks Lakehouse Platform. The ideal candidate will possess strong hands-on expertise in Databricks, Apache Spark, PySpark, SQL, Delta Lake, and modern data architecture.
Key Responsibilities:
Own the technical roadmap for Databricks-based data engineering initiatives, aligning platform architecture with enterprise data strategy and business goals.
Lead architecture reviews and make high-impact technical decisions involving data platform design, technology selection, scalability, and maintainability.
Define reusable engineering frameworks, reference architectures, coding standards, and development guidelines across multiple data engineering projects.
Drive the implementation of domain-oriented data architecture and data product strategies to improve data accessibility, ownership, and reuse.
Establish architectural patterns for multi-environment data platforms, including development, testing, staging, and production.
Lead the adoption of Infrastructure as Code and automated platform provisioning to standardize Databricks environment management.
Partner with security, governance, and compliance teams to establish enterprise-wide policies for sensitive data, auditability, and regulatory requirements.
Manage technical dependencies, delivery risks, and architectural trade-offs across multiple teams and concurrent data initiatives.
Required Skills And Experience8 15 years of experience in designing and delivering enterprise data platforms using Databricks and cloud-native technologies.
Strong proficiency in Apache Spark, PySpark, Python, SQL, and distributed data processing.
Expertise in lakehouse architecture, Delta Lake, data architecture patterns, and large-scale data platform design.
Proven experience establishing data governance, metadata management, access controls, and data lineage practices.
Strong understanding of cloud architecture, networking, identity management, and security across Azure, AWS, or GCP.Experience with platform automation, Infrastructure as Code, CI/CD, and production deployment strategies.
Ability to evaluate complex technical requirements and translate them into scalable architectural solutions.
Knowledge of Databricks Asset Bundles and automated deployment practices.
Understanding of data governance, security, data quality, and monitoring.
Qualification: Bachelor's degree or equivalent combination of education and experience.