JOBSEARCHER

Data Engineer

ARCHIVED
IflowMalvern, PAL5 SeniorSeptember 19th, 2026

We can't find an active application page for this role right now. It may reopen or be listed elsewhere. Use Next Steps to search for an active apply link and similar live jobs.

Job Title: Data Engineer
Location: Malvern, PA
Duration: 6 Months
Experience Required: 10 & Above
Job Description: Technical skill sets :Python, Pyspark, AWS Lambda, S3, AWS glue, Kinesis, SQL, Apache Flink, Confluent Kafka, AI-LLM, Gen-AI, Java is optional
Responsibilities:
  • 1. Advanced Architecture & System DesignA Tech Lead is primarily responsible for the overall platform vision and ensuring systems do not break under scale.
  • Distributed Computing: Mastery of frameworks like Apache Spark or Ray for massive-scale parallel data processing.
  • Streaming & Event-Driven Architecture: Deep understanding of real-time pipeline design using Kafka, Kinesis, or Flink.
  • Cloud Infrastructure: Expertise in at least one major public cloud (AWS), specifically understanding storage/compute decoupling and cost optimization.2. Core Programming & Database Management
  • Leads set coding standards and review code, requiring complete fluency in the fundamentals.
SQL: Advanced mastery for metrics computation, window functions, and query performance tuning across relational and columnar databases (e.g., Snowflake, Redshift, BigQuery).Scripting Languages: High proficiency in Python or Scala for writing reusable pipeline code and interacting with APIs. Data Storage: Deep familiarity with both columnar/analytical stores and NoSQL databases (e.g., DynamoDb, Cassandra).3. Pipeline Orchestration & DevOpsEnsuring pipelines run smoothly, idempotently, and securely in production. Workflow Orchestration: Ability to architect Directed Acyclic Graphs (DAGs) in tools like Apache Airflow or Prefect. CI/CD & Infrastructure as Code (IaC): Applying software engineering principles to data by using Docker, Kubernetes, and Terraform. Data Governance & Security: Implementing Role-Based Access Control (RBAC), data masking, and compliance frameworks.4. Leadership & Soft SkillsTech leads also mentor junior engineers, estimate project timelines, and translate ambiguous business needs into concrete technical specifications. Mentorship & Code Review: Fostering a collaborative development environment and enforcing style guidelines. System Observability: Building logging, monitoring, and alerting mechanisms so the team knows exactly when and why pipelines fail. Role Descriptions: Python| Pyspark| AWS Lambda| S3|AWS glue| Kinesis|SQL| Apache Flink| Confluent Kafka| Java is optional Essential Skills: Python| Pyspark| AWS Lambda| S3|AWS glue| Kinesis|SQL| Apache Flink| Confluent Kafka| Java is optional Desirable Skills: Keyword:
Skills: Digital : Python~Digital : Amazon Web Service(AWS) Cloud Computing~Digital : Kafka~Advanced Java Concepts~Digital : PySpark