GCP Cloud Data Engineer
GCP Data EngineerLooking for a Big Data Engineer on Google Cloud Platform to help design, develop and maintain data engineering solutions to migrate data hosted on our on-prem platform to Google cloud Platform (GCP).The engineer will design and program PB sized scalable data lake interfaces, micro services and web technologies that support ingesting and querying structured data.Architect and program sophisticated distributed systems and high performance compute and data pipelines.Enable mining and analyzing data to help AI leaders and researchers make data driven decisions for data collection, diversity, training, and evaluation.Build and implement support for versioned, traceable, and immutable datasets in a data lake in a distributed and scalable manner.Spend a majority of the time hands-on writing code and peer reviewing high performance, high quality, and well tested and well architected code.MUST HAVE SKILLS (Most Important):Certified Google Cloud Data Engineer/Architect Bachelor's degree with 3-5+ years of experience in Data Engineering/BI areas with at least 2 years data engineering on GCPExperience working in GCP based Big Data deployments (Batch/Real-Time) leveraging Big Query, Big Table, Google Cloud Storage, PubSub, Data Fusion, Dataflow, Dataproc, etc.Experience developing and deploying ETL / ELT processes and documentation including physical data model, source to target mappings, ETL / ELT packages (Matillion, Fivetran, Spark, Google Data Fusion, etc.)Demonstrated mastery in Google BigQueryDESIRED SKILLS:Strong knowledge of Hadoop, HDFS, Hive, Spark, Spark Streaming and Presto Strong knowledge on Google cloud storage Data lifecycle managementStrong knowledge on BIGQuery Slots managementCost optimization for Dataproc workload managementDemonstrated mastery in cloud database concepts and large-scale cloud data warehouse and lake implementations using Big Data Tools: BigQuery, Cloud Dataflow, Cloud Proc, Cloud Pub/Sub, Cloud Composer, Google Data Studio, Cloud functions, Google Cloud Storage Implement solutions for structured, semi-structured, and unstructured data sources, relational and non-relational databases.Advanced Java/Python coding skills Experience with CI-CD pipelines for promoting big data release deployments and designing log monitoring features ((e.g. JIRA, GitHub, Jenkins, Nexus, Artifactory)Experience in data visualization tools like Kibana, Grafana, Tableau and associated architectures.JOB DUTIES:Design and build data engineering solutions using Google Cloud Platform (GCP) services: BigQuery, DataFlow, Pub/Sub, BigTable, Data Fusion, DataProc.Data modeling and schema design that will range across multiple business domains within the cloud for large enterprise data warehouse and data lakes solutions.Extracting, Loading, Transforming, cleaning, and validating data using cloud ETL/ELT toolsDesigning data engineering pipelines and architectures for data processing Partner with multiple client stakeholders including partners, business users, BI and Analytics teams.Work with teams to conduct training workshops to identify data sources, flows, and requirements.