Lead Data Engineer -Databricks

Alphanumeric is hiring a DATA ENGINEER IV LEAD to work remotely (EST hours preferred) with an established leader in the financial and insurance industries. Candidates located in or near Richmond, VA are strongly preferred. This is a contract-to-hire opportunity with an approximate conversion salary of $135,000 annually.Pay Range: $93.00 - $100.00/hr. W2No third-party agencies please. Sponsorship is not available for this position. As a member of the Data Solutions and Data Engineering team, you will play a key role in transforming enterprise data capabilities using Databricks and modern cloud-based data engineering practices. This position focuses heavily on enhancing out-of-the-box Databricks functionality, developing scalable frameworks, optimizing Delta Lake infrastructure, and supporting large-scale data transformation initiatives utilizing medallion architecture principles.You will help design and build robust data ingestion, standardization, and curation pipelines across bronze, silver, and gold layers while improving the usability and accessibility of enterprise data. This includes supporting initiatives involving AI-powered extraction of underwriting and medical data from images and PDFs to improve policy cost analysis and business insights.What You'll Be DoingPartner with business users to gather and define data requirementsCollaborate with architects and technical leads to design scalable Databricks solutionsBuild and support data engineering solutions throughout the full SDLC lifecycleDevelop frameworks and reusable components for pipeline standardization, SLA monitoring, and data quality managementCreate and optimize Delta tables, Databricks dashboards, and orchestration workflowsDesign and maintain dimensional and ER-based data models utilizing medallion architectureImplement batch and streaming data pipelines within DatabricksCreate unit tests, perform SIT testing, and support UAT troubleshooting effortsDebug and resolve data defects and performance issuesImplement and maintain data security policies and compliance standardsWork closely with upstream/downstream teams, including offshore and vendor partnersProduce technical documentation, training materials, and knowledge transfer sessionsResearch and evaluate emerging Databricks capabilities including Intelligent Document Processing, Genie AI coding, and self-service workspace featuresJob RequirementsRequired QualificationsMinimum 3 years of hands-on Databricks experience in Azure or AWS environmentsStrong experience utilizing medallion architecture and modern data modeling techniquesExpertise creating reusable data engineering frameworks and standardization processesExperience developing dimensions and fact tables with SCD2 trackingStrong Databricks orchestration experience with both batch and streaming pipelinesExperience performing testing, debugging, and production supportStrong understanding of Databricks compute and storage optimizationExperience with CI/CD, GitLab, and deployment automation toolsStrong understanding of Agile methodologies, story creation, and effort estimationHands-On Technical SkillsExpert-level SQL skillsIntermediate or higher proficiency in PySparkExperience with Lakeflow and data quality expectations codingStrong understanding of Databricks features including Unity Catalog, Spark UI, Job Scheduling, and related capabilitiesAbility to quickly learn and implement newer Databricks technologies and AI-driven capabilitiesPreferred Qualifications5+ years of Databricks experience building enterprise-scale capabilities from scratchExperience working within insurance or financial services environmentsExperience with data governance and observability platformsUnderstanding of enterprise data models and schema evolution strategiesSoft SkillsExcellent communication and presentation skillsAbility to create training materials and technical documentationStrong problem-solving and analytical thinkingCollaborative mindset with strong teamwork skillsDesign-thinking approach to solution developmentNo third-party agencies please. Sponsorship is not available for this position.