Lead Data Engineer

Position - Lead Data EngineerLocation - San Francisco, CAMain Skills - SQL, Databricks, ADF, Datastage (or other ETL tool), SSAS cubes, Cognos/other, Tableau/other, Thoughtspot/otherJob DescriptionWe are seeking a Lead Data Engineer to design, implement, and maintain our data infrastructure and business intelligence solutions. This role requires a technical expert who can architect scalable data solutions while collaborating across teams to drive business insights.Required Experience 8-10 years of software development and deployment experienceAt least 7 years of hands-on experience with:SQLDatabricksAzure Data Factory (ADF)Datastage (or other ETL tools)SSAS cubesBI tools (Cognos, Tableau, ThoughtSpot, etc.)Strong Database knowledge (SQL & No-SQL preferred)Advanced SQL and Python skillsExperience working with APIs for data collection/ingestionKey ResponsibilitiesTechnical DevelopmentWrite SQL for processing raw data, Kafka ingestions, ADF pipelines, data validation, and QALead, design and implement innovative analytical solutions using Tableau, SQL, and Big Data technologiesExperience with storing, joining, filtering, and analyzing data using SQL, Spark, Hive, etc.Data Architecture & DesignDesign comprehensive data architecture and data modeling strategies for BI systemsCreate logical and physical data models using best practices to ensure high data quality and reduced redundancyCollaborate with stakeholders to translate business requirements into robust, scalable data modelsPerform detailed analysis of business problems and technical environments to design quality solutionsIntegration & ImplementationIntegrate data from various sources (databases, warehouses, external systems)Develop strategies for ETL processes ensuring data accuracy, consistency, and availabilityEvaluate and select appropriate BI tools based on organizational needsRun POCs and implement BI software, including visualization tools and analytics solutions.Optimization & GovernanceMonitor and optimize BI system performance for efficient data retrieval, processing, and reportingIdentify and resolve performance bottlenecks through query tuning and data structure optimizationEstablish data governance policies ensuring data quality, consistency, and securityDefine access controls, privacy measures, and retention policies to meet regulatory requirementsCollaboration & LeadershipBuild partnerships with architects, technical leads, and stakeholders across functional groupsActively participate in code reviews and testing to ensure adherence to best practicesWork with product and engineering teams to evaluate requirements and new featuresBuild and foster a high-performance engineering cultureMentor team members and provide tools and motivation for successAdditional Skills & RequirementsExperience building data ingestion pipelines and data warehouse architectureAbility to write design documentation, source-to-target mapping documentation, and manage Confluence pagesExperience converting business functionalities into technical Jira storiesStrong experience with data modeling, design patterns, and building scalable BI solutionsKnowledge of cloud platforms (Azure, AWS or equivalent)Experience analyzing large datasets to identify trends, patterns, and outliers