{"schemaVersion":"jobsearcher.job.v1","id":"3f365e9ecde376952e8d4d78","url":"https://jobsearcher.com/jobs/3f365e9ecde376952e8d4d78","canonicalUrl":"https://jobsearcher.com/jobs/3f365e9ecde376952e8d4d78","title":"Lead Data Engineer","description":"Lead Data Engineer - Job DescriptionResponsibilitiesDevelopment Tasks:Collect metrics based on user interactions.Visualize data for business teams.Develop and redesign data pipelines using Kafka streams.Implement solutions using Spring Boot Java and Databricks Spark streaming.Leadership Duties:Lead the measurement processes from requirements gathering to production delivery.Collaborate with other team leads, business partners, and product managers.Balance between hands-on engineering (50%) and team leadership (50%).Collaboration Structure:Onsite: Lead role (this resource)Nearshore: Senior developer.Offshore: Data engineer role.Required Skills & Experience:Hands-on code mindset with deep understanding in technologies / skillset and an ability to understand larger picture.Sound knowledge to understand Architectural Patterns, best practices and Non-Functional RequirementsOverall, 8-10 years of experience in heavy volume data processing, data platform, data lake, big data, data warehouse, or equivalent.5+ years of experience with strong proficiency in Python and Spark (must-have).3+ years of hands-on experience in ETL workflows using Spark and Python.4+ years of experience with large-scale data loads, feature extraction, and data processing pipelines in different modes – near real time, batch, realtime.Solid understanding of data quality, data accuracy concepts and practices.3+ years of solid experience in building and deploying ML models in a production setup. Ability to quickly adapt and take care of data preprocessing, feature engineering, model engineering as needed.Preferred: Experience working with Python deep learning libraries like any or more than one of these - PyTorch, Tensorflow, Keras or equivalent.Preferred: Prior experience working with LLMs, transformers. Must be able to work through all phases of the model development as needed.Experience integrating with various data stores, including:SQL/NoSQL databasesIn-memory stores like RedisData lakes (e.g., Delta Lake)Experience with Kafka streams, producers & consumers.Required: Experience with Databricks or similar data lake / data platform.Required: Java and Spring Boot experience with respect to data processing - near real time, batch based.Familiarity with notebook-based environments such as Jupyter Notebook.Adaptability: Must be open to learning new technologies and approaches.Initiative: Ability to take ownership of tasks, learn independently, and innovate.With technology landscape changing rapidly, ability and willingness to learn new technologies as needed and produce results on job. Preferred Skills:Ability to pivot from conventional approaches and develop creative solutions.Required Skills: • Python and PySpark. • Kafka and Kafka streams. • MySQL and MySQL Heat. • Azure Delta Lake. • ETL processes. • Kafka integrations using Spring Boot Java. • Data streaming with Spark.","company":"My3Tech","rawCompany":"my3tech","city":"Millbrae","state":"CA","isRemote":false,"isActive":false,"createdAt":"2026-08-25T04:02:35.795Z","occupations":[{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-2051.00","title":"Data Scientists","slug":"data-scientists"},{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"}],"industries":[{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Lead Data Engineer","description":"Lead Data Engineer - Job DescriptionResponsibilitiesDevelopment Tasks:Collect metrics based on user interactions.Visualize data for business teams.Develop and redesign data pipelines using Kafka streams.Implement solutions using Spring Boot Java and Databricks Spark streaming.Leadership Duties:Lead the measurement processes from requirements gathering to production delivery.Collaborate with other team leads, business partners, and product managers.Balance between hands-on engineering (50%) and team leadership (50%).Collaboration Structure:Onsite: Lead role (this resource)Nearshore: Senior developer.Offshore: Data engineer role.Required Skills & Experience:Hands-on code mindset with deep understanding in technologies / skillset and an ability to understand larger picture.Sound knowledge to understand Architectural Patterns, best practices and Non-Functional RequirementsOverall, 8-10 years of experience in heavy volume data processing, data platform, data lake, big data, data warehouse, or equivalent.5+ years of experience with strong proficiency in Python and Spark (must-have).3+ years of hands-on experience in ETL workflows using Spark and Python.4+ years of experience with large-scale data loads, feature extraction, and data processing pipelines in different modes – near real time, batch, realtime.Solid understanding of data quality, data accuracy concepts and practices.3+ years of solid experience in building and deploying ML models in a production setup. Ability to quickly adapt and take care of data preprocessing, feature engineering, model engineering as needed.Preferred: Experience working with Python deep learning libraries like any or more than one of these - PyTorch, Tensorflow, Keras or equivalent.Preferred: Prior experience working with LLMs, transformers. Must be able to work through all phases of the model development as needed.Experience integrating with various data stores, including:SQL/NoSQL databasesIn-memory stores like RedisData lakes (e.g., Delta Lake)Experience with Kafka streams, producers & consumers.Required: Experience with Databricks or similar data lake / data platform.Required: Java and Spring Boot experience with respect to data processing - near real time, batch based.Familiarity with notebook-based environments such as Jupyter Notebook.Adaptability: Must be open to learning new technologies and approaches.Initiative: Ability to take ownership of tasks, learn independently, and innovate.With technology landscape changing rapidly, ability and willingness to learn new technologies as needed and produce results on job. Preferred Skills:Ability to pivot from conventional approaches and develop creative solutions.Required Skills: • Python and PySpark. • Kafka and Kafka streams. • MySQL and MySQL Heat. • Azure Delta Lake. • ETL processes. • Kafka integrations using Spring Boot Java. • Data streaming with Spark.","datePosted":"2026-08-25T04:02:35.795Z","dateModified":"2026-08-25T04:02:35.795Z","hiringOrganization":{"@type":"Organization","name":"My3Tech","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Millbrae","addressRegion":"CA","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"3f365e9ecde376952e8d4d78"},"url":"https://jobsearcher.com/jobs/3f365e9ecde376952e8d4d78"}}