Bigdata Pyspark Lead
Job Title – Pyspark LeadLocation – Irving, TX – Onsite Hire Type – FulltimeKey ResponsibilitiesLead the design and development of scalable data pipelines using PySpark and distributed computing frameworks.Build and optimize ETL/ELT workflows for large-scale banking datasets.Collaborate with business stakeholders, data architects, analysts, and application teams to understand requirements.Ensure high performance, fault tolerance, and scalability of big data applications.Work with structured and unstructured datasets from multiple banking systems.Implement data quality checks, governance, and security standards.Mentor and guide junior developers and data engineers.Participate in architecture discussions and recommend best practices.Integrate data solutions with cloud platforms and enterprise data lakes.Support production deployments, troubleshooting, and performance tuning.Coordinate with offshore/onshore teams and manage delivery timelines.Required SkillsTechnical SkillsStrong hands-on experience in PySpark and Apache SparkExpertise in Python programmingExperience with Hadoop ecosystem components:HDFSHiveSqoopKafkaStrong SQL and data warehousing conceptsExperience with ETL pipeline developmentKnowledge of Airflow/Oozie for workflow orchestrationExperience with cloud platforms such as:AWSAzureGCPFamiliarity with Delta Lake, Databricks, or SnowflakeVersion control using GitCI/CD pipeline understandingBanking Domain SkillsGood understanding of banking data domains:Retail BankingCorporate BankingRisk & ComplianceAML/KYCPayments/TransactionsUnderstanding of regulatory and compliance requirements in banking environments.Preferred QualificationsBachelor's/Master's degree in Computer Science, Engineering, or related fieldSpark/Databricks/AWS certifications preferredExperience working in Agile/Scrum environmentsExposure to real-time streaming and event-driven architecturesSoft SkillsStrong communication and stakeholder managementLeadership and team mentoring abilitiesProblem-solving and analytical mindsetAbility to work in a fast-paced banking environment