Databricks Engineer

LightFeather is currently seeking a skilled Databricks Engineer to join our dynamic team and play a pivotal role in our data engineering efforts. The successful candidate will be responsible for designing, implementing, and optimizing data pipelines that integrate data from multiple sources into Databricks. In this role, your primary focus will be to ensure seamless data flow and enable efficient data processing, storage, and analysis.This Position is Full Time, Remote.Responsibilities:Develop and maintain ETL processes to extract, transform, and load data from various sources including Google Analytics (GA4), Splunk, Medallion, and others into DatabricksDesign and implement data pipelines and workflows using Databricks, ensuring scalability, reliability, and performanceCollaborate with data scientists, analysts, and other stakeholders to understand data requirements and provide appropriate data solutionsDevelop and maintain Python Notebooks within Databricks for data analysis and processing, optimizing data workflows for efficiency and accuracyOptimize and tune data processing jobs for performance and cost-efficiencyEnsure data quality and consistency through robust data validation and cleansing techniquesMonitor and troubleshoot data pipeline issues, ensuring timely resolution and minimal downtimeLeverage Terraform for infrastructure as code (IaC) practices to automate and manage infrastructure provisioning and scalingStay updated with the latest trends and advancements in data engineering and Databricks technologiesQualifications:US CitizenshipActive clearance at thePublic Trust level or higher. IRS clearance preferredBachelor’s degree preferred or equivalent experience5+ years of hands-on experience with Databricks, including designing and managing large-scale data pipelinesProficiency in ETL tools and techniques, with a strong understanding of data integration from sources like Google Analytics (GA4), Splunk, and MedallionSolid experience with SQL, Python, and Spark for data processing and transformationFamiliarity with cloud platforms such as AWS, Azure, or Google Cloud, with a focus on their data servicesExperience with other big data technologies such as Apache AirflowKnowledge of data warehousing concepts and best practicesFamiliarity with data visualization tools such as Tableau, Power BI, or LookerProven experience in designing and deploying Databricks infrastructure on cloud platforms, preferably Amazon AWSDeep understanding of Apache Spark, Delta Lake, and their integration within the Databricks environmentProficient in Terraform for implementing infrastructure as code (IaC) solutionsStrong expertise in Python, especially in developing Notebooks for data analysis within DatabricksDemonstrated ability to design and implement complex data pipelines with ETL processes for large-scale data aggregation and analysisKnowledge of best practices for infrastructure scaling and data management, with a keen focus on security and robustnessStrong problem-solving skills and the ability to troubleshoot complex data issuesExcellent communication and collaboration skills to work effectively with cross-functional teamsWhy Join LightFeather?You'll be part of a team dedicated to meaningful impact, working on solutions that address mission-critical needs. Experience variety, fulfillment, and the opportunity to work with some of the best in the industry. We are committed to fostering a diverse and inclusive environment where everyone is valued and respected.Commitment to DiversityLightFeather is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees, regardless of race, color, religion, sex, sexual orientation, gender identity, national origin, veteran, or disability status.Powered by JazzHRCd9p3S6WIT