Hadoop Hive Python Developer
Role: Hadoop Hive Python DeveloperLocation: Charlotte, NC (Onsite)Fulltime - PermanentJob DescriptionMust Have Technical/Functional SkillsPrimary skills: Hadoop, Hive, Python, PySpark, Apache Kafka, Hadoop Ecosystem, Hive, Databricks Lakehouse Architecture, Delta Lake, Bronze/Silver/Gold Data Modeling, Big Data ETL Pipeline Development, SQL, Real-time Data Ingestion Frameworks, Data Governance & Cataloging, CI/CD Tools - Git, Jenkins, Bitbucket, Workflow Orchestration, and Cloud & On-Prem Big Data Platforms.Experience: Minimum 9+ yearsRoles & ResponsibilitiesSeeking a Senior Big Data Engineer with 9-14 years of experience specializing in Hadoop, Python, Hive PySpark, Kafka, and strong experience designing data solutions for large-scale financial systems.In addition, the candidate must possess advanced expertise in Databricks Lakehouse architecture, particularly around Bronze/Silver/Gold layer data modeling, Delta Lake optimizations, and building reliable, scalable pipelines for regulatory, risk, trading, and analytics workloads.This role focuses on delivering highly performant, well-governed data platforms that support the bank's mission-critical global markets functions.Key Responsibilities:Big Data Platform EngineeringDesign, develop, and optimize PySpark-based ETL pipelines running on on-prem Hadoop clusters and cloud environments.Build high-volume ingestion frameworks using Kafka for real-time and near-real-time trading and market data.Develop, tune, and manage Hadoop ecosystem components-HDFS, YARN, MapReduce, Tez, Oozie/Airflow.Build high-performance, optimized Hive data models for regulatory reporting, trade lifecycle, and market risk processing.Databricks Lakehouse & Delta FrameworkArchitect and implement Bronze/Silver/Gold layer modeling patterns within the Databricks Lakehouse.Apply Delta Lake best practices including:o optimized file managemento Z-Orderingo Delta Change Data Feed (CDF)o schema evolution & enforcemento ACID transaction handlingBuild reusable frameworks for ingestion, cleansing, transformation, and consumption of data across Lakehouse layers.Enable governance, lineage, and auditability using Unity Catalog or equivalent cataloging tools.Collaboration, Leadership & DeliveryCollaborate closely with quants, product owners, architects, risk tech, and business users.Participate in agile ceremonies - sprint planning, refinement, design reviews.Mentor junior engineers and contribute to building strong engineering practices across tech teams.Required Skills & Experience9-14 years of hands-on experience in Big Data engineering.Expert skills in:o PySpark - dataframe optimizations, partitioning, broadcast strategies, distributed computing.o Kafka - producer/consumer design, schema registry, streaming ETLs.o Hadoop ecosystem - HDFS, YARN, MapReduce/Tez, Oozie/Airflow.o Hive - advanced query tuning, TEZ optimization, partition/bucket management.• Extensive hands-on experience with Databricks Lakehouse, including:o Bronze/Silver/Gold layer modelingo Delta Lake optimizationso Data quality frameworks on Lakehouseo Structured & unstructured data handlingExperience in Global Markets, Risk, Treasury, Trade Surveillance, or Regulatory Reporting.Strong SQL knowledge with experience working on massive datasets (TB/PB scale).Experience with CI/CD practices - Git, Jenkins, Bitbucket, build pipelines.