Big data Pyspark Developer

Job Title: Big data Pyspark DeveloperWork Location : Irving, Texas/ Tampa, FLJob SummaryExp 6 Years Must have good technical experience and should be able to provide technical solutions for multiple modules in parallel on need basis and bring the task to closure on timeUnix SQL and Shell Scripting experience is a must haveExpertise in Designing and developing scalable Apache spark ETL based Data processing pipelinesStrong commandline knowledge in UnixLinux with Shell scripting using Bash Kornshell or Perl and File processing using awk scriptsExpertise in SQL querying and complex joinsImplementing comprehensive Spark based Data validation frameworks transforming large volumes of Financial data within the Project lifecycleExpertise with complex Data workflows with Apache AirFlow managing task dependencies SLAs etc to ensure timely data delivery and corresponding automated validation controlsStrong Analytical skills and expertise on SparkSQL for Data analysis and validation ensuring the delivery of clean queryready datasets for business consumptionExpertise in Data quality checks and monitoringQuality Engineering team where 70percent of effort will be for developing automation frameworks for testing Remaining 30percent effort will be on manual testing until its fully automatedHandson with Automation Framework Design for ETL and APISME in Data Analysis Database testing Messaging queuesExperience with coding standards code reviews source management build processes CICD pipelineSkillsMandatory Skills : Apache Spark, Big Data Hadoop Ecosystem, Python, Python for DATA, SparkSQL