Lead Data Engineer

Data Engineering Team LeaderAs a key figure in our data engineering team, you will play a critical role in designing and implementing scalable data pipelines and architectures. You will leverage your expertise in Python and PySpark to enhance our data processing capabilities, with a particular focus on migrating data from Snowflake to Databricks. This position requires a strong background in big data technologies and excellent leadership skills.Required Skills & QualificationsMinimum of 8 years of experience in data engineering, with a strong focus on Databricks and big data technologies.Proficient in Python and PySpark, with a proven track record of building data pipelines and ETL processes.Experience with Snowflake and a demonstrated ability to migrate data solutions to Databricks.Strong understanding of data modeling, data warehousing concepts, and distributed computing.Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and their data services.Excellent problem-solving skills and the ability to work in a fast-paced environment.Strong communication and interpersonal skills, with the ability to collaborate effectively with technical and non-technical stakeholders.Bachelor's degree in Computer Science, Engineering, or a related field; advanced degree preferred.Prior work experience at client or in client's industry.Applicants must be able to work directly for the company on W2.Previous experience working with large datasets in a high-volume environment.Knowledge of machine learning frameworks and analytics tools is a plus.Experience in Agile methodologies and DevOps practices.Day-to-Day ResponsibilitiesLead the design, development, and optimization of data pipelines and ETL processes in Databricks.Collaborate with cross-functional teams to gather requirements and translate them into robust data solutions.Migrate existing data solutions from Snowflake to Databricks, ensuring minimal disruption and high data integrity.Utilize Python and PySpark to build and maintain scalable data processing frameworks.Implement best practices for data governance, security, and performance optimization.Mentor and guide junior data engineers, fostering a culture of continuous learning and improvement.Monitor system performance and troubleshoot issues to ensure optimal data flow and availability.Stay updated with industry trends and emerging technologies related to data engineering and analytics.Company Benefits & CultureInclusive and diverse work environment.Opportunities for professional growth and development.Flexible work arrangements to support work-life balance.