Senior Data Engineer

Key Responsibilities Design, build, and maintain scalable batch and streaming data pipelines for ingesting and processing large datasets Transform, model, and optimize data for analytics, reporting, and downstream applications Implement data validation, monitoring, and security controls to ensure data quality, reliability, and compliance Contribute to the design and evolution of the data platform with a focus on scalability, performance, and maintainability Collaborate with BI, analytics, AI, and product teams to deliver data solutions aligned with business needs Develop automated workflows and observability mechanisms to ensure pipeline reliability and system visibility Create and maintain documentation for pipelines, data models, and platform components Evaluate and improve tools, frameworks, and processes to enhance efficiency and maintainability Required QualificationsStrong experience with Databricks, Apache Spark, and PySpark for large-scale data processingExperience building and optimizing data pipelines at scale, including parallelization and performance tuningExperience with near real-time or streaming data systemsProficiency in Python and SQL for data engineering and transformation workflowsExperience with ETL/ELT processes and toolsHands-on experience with cloud data platforms (Azure, AWS, or GCP)Solid understanding of data modeling and dataset design for analytics and downstream applicationsExperience tuning queries and optimizing compute performanceKnowledge of data governance, security, and compliance practices Strong communication skills and ability to work cross-functionally Preferred QualificationsExperience with cloud platforms (Azure, AWS, or GCP)Experience with vector databases and embedding-based systemsExperience with streaming frameworks and data quality toolsFamiliarity with knowledge graphs and graph-based data modelingExperience with CI/CD pipelines and deployment automationFamiliarity with BI tools and machine learning pipelines Education & ExperienceBachelor's degree or equivalent experience.3–7 years of data engineering experience.US Persons only (Citizens/ Green card)