Data Engineer/Analyst

Role: Data Engineer/Analyst Location: Irving, TX Duration: 12+ Months Role Overview We are seeking a highly skilled Data Engineer to lead the architecture, development, and optimization of our end-to-end data pipelines. A primary focus of this role will be driving our on-premise to cloud migration strategy, ensuring a seamless transition of legacy systems into a modern GCP ecosystem. You will be responsible for deep-dive data analysis, building robust ETL/ELT processes, and delivering actionable insights through advanced reporting. Key Responsibilities Design and implement high-throughput streaming architectures using Kafka to capture and process event-driven data Lead the end-to-end migration of complex datasets and workloads from Hadoop, Hive, and Teradata to Google Cloud Platform (GCP) Build and maintain robust ETL/ELT pipelines using Python and PySpark, ensuring seamless integration of both batch and streaming data Conduct deep-dive SQL analysis to ensure data quality and consistency throughout the migration lifecycle and across JSON-based data structures Collaborate with stakeholders to deliver advanced ETL reporting, turning raw streams into actionable dashboards and performance metrics Perform complex SQL analysis to validate data integrity, identify patterns, and troubleshoot performance bottlenecks across distributed systems Develop automated ETL workflows using Python and PySpark, transforming raw data into structured formats for business intelligence and executive reporting. Exceptional collaboration skills, with the ability to mentor junior engineers and communicate technical concepts to non-technical stakeholders Technical Stack & Tools Cloud Platform: Google Cloud Platform (GCP) – specifically BigQuery, Dataflow, Cloud Functions, GCS, and Cloud Composer. Data Processing: Python (Expert), PySpark, Apache Spark. Streaming & Messaging: Apache Kafka (Real-time architecture design). Legacy Ecosystems: Hadoop (HDFS), Hive, and Teradata. Data Languages: Advanced SQL (Optimization, Window Functions, Performance Tuning). Data Formats: handling and parsing complex JSON and semi-structured data. Required Qualifications & Experience Bachelor's degree and five or more years of work experience Hands-on experience with Apache Kafka to design and implement real-time streaming architectures Proven experience leading large-scale migration strategies, successfully moving complex workloads and multi-terabyte datasets from Hadoop, Hive, and Teradata environments to Google Cloud Platform (GCP) Proven experience with GCP (BigQuery, Cloud Functions, Dataflow, and GCS) Mastery of Advanced SQL to perform deep-dive analysis, troubleshoot performance bottlenecks in distributed systems, and ensure 100% data integrity during migration phases Expert proficiency in Python and PySpark for building scalable ETL/ELT pipelines that seamlessly unify batch and streaming data streams Strong background in Hadoop, Hive, and Teradata to facilitate smooth legacy transitions. Expertise in handling and parsing JSON data for large-scale ingestion optimized schemas for downstream consumption and cloud-native storage Comprehensive understanding of end-to-end data lifecycles, from raw ingestion to the delivery of actionable dashboards and performance metrics Demonstrated ability to act as a bridge between technical and non-technical stakeholders, with a focus on mentoring junior talent and fostering a collaborative engineering culture Preferred Qualifications Relevant Cloud Certifications (e.g., GCP Professional Data Engineer or AWS Certified Data Engineer). Experience using AI-assisted development tools (e.g., GitHub Copilot, Gemini) to accelerate delivery cycles and optimize code performance .