Middleware Architect

Job Title: Middleware Architect/ Engineer Location: Irving, TX- HybridDuration: Long TermJob Description: Primary Responsibilities:* Design and build large scale data processing system (real-time and batch) to address growing AI/Client and Data needs of a Fortune 500 company* Build a product to process large amount data/events for AI/Client and Data consumption* Automate test coverage (90+%) for data pipelines. Best practices and frameworks for unit, functional and integration tests.* Automate CI and deployment processes and best practices for the production data pipelines.* Build AI/Client model based alert mechanism and anomaly detection system for the product. The goal is have a self-annealing productRequired Skills/Experience* 7+ years of overall experience in software development with 5 or more years of relevant experience in designing, developing, deploying and operating large data processing data pipelines at scale.* 3 or more years experience with Apache Spark for Streaming and batch process* Good knowledge on Apache Kafka* Strong background in programming (Scala/Java)* Experience on building reusable data frameworks/modules* Experience on Airflow scheduler* Experience with Containers, Kubernetes and scaling elastically* Strong background in algorithms and data structures* Strong analytical and problem solving skills* Strong bent towards engineering solutions which increase productivity of data consumers* Strong bent toward completely automated code deployment/testing (DevOps, CI/CD)* Passion for data engineering and for enabling others by making their data easier to access.* Some experience with working with and operating workflow or orchestration frameworks, including open source tools like Activiti, Spring Boot, Airflow and Luigi or commercial enterprise tools.* Excellent communication (writing, conversation, presentation) skills, consensus builder* Demonstrated ability to tackle tough coding challenges independently and work closely with others on a highly productive coding teamMust have Skills: Apache Spark, Apache Kafka, Scala/Java, NoSQL Databases, Elasticsearch & Kibana, Kubernetes, Docker ContainersNice to have Skills:* Knowledge of API Development* Apache Flink experience* Cloud experience* DevOps skills* Any other streaming technologies/tools experience