Data Bricks Lead Engineer

Company: Quanta Systems Technologies LLP

Employment Type: Contract

Location: Washington DC

Experience: 10 Years

Job Description: We are seeking a DatabricksEngineer to build of a new Databricks solution that ingests source files and constructs a semantic/serving layer that powers a Power BI dashboard. A semantic layer already exists; the new solution must conform to the same structure so that a replicated Power BI solution works seamlessly against it with no rework to reports, measures, or relationships. The engineer owns the Databricks data and semantic layer and partners with a dedicated Power BI developer/team, who own the dashboard build, to deliver the end solution.

Key Responsibilities: Lead the design and delivery of file-ingestion pipelines in Azure Databricks (batch and/or streaming) that land and process incoming files into Delta Lake using a Medallion architecture (Bronze ? Silver ? Gold).Design the Gold/serving layer to replicate the existing semantic model exactly — matching table names, grain, column names, data types, keys, and relationships so the existing Power BI dataset can be repointed without redesign. Expose the model through Databricks SQL endpoints (Direct Query) so the Power BI developer can connect and validate the dashboard with no redesign. Collaborate with the Power BI developer/team as the data-side owner — align on the semantic contract (star schema, measures, RLS expectations), support their dashboard build, and resolve any data or model issues to land the end deliverable. Document the semantic mapping and work with the Power BI team to reconcile the replicated dashboard's output against the existing solution to prove parity.