Data Engineer
Senior Streaming Data EngineerReal-Time Data Pipelines · Kafka / MSK / Kinesis · AWSTHE ROLEDesign, build, and operate real-time streaming pipelines on Apache Kafka, AWS MSK, and Kinesis — covering topic/partition design, schema handling, and producer/consumer development.Keep clusters healthy: tune for throughput, latency, and fault tolerance; manage scaling, replication, retention, and security (SSL/TLS, SASL, ACLs).Architect event-driven, microservices-based streaming systems that hold up under high-volume data and recover cleanly from faults.Write real-time processing/ETL apps (Spark Streaming, Kafka Streams, Flink, Lambda) with aggregations, windowing, and stateful and complex event processing.Connect streaming to data lakes, warehouses, and databases (Spark, Redshift, S3) with quality checks, lineage, and monitoring.Manage infrastructure as code (Terraform/CloudFormation) and automate deployment via CI/CD.Mentor junior engineers, partner with architects/scientists/analysts, and communicate clearly to technical and non-technical audiences.REQUIRED QUALIFICATIONS7+ years in data engineering with a strong concentration in streaming data.Bachelor's in Computer Science, Engineering, or a related STEM field.Deep production experience with Apache Kafka and AWS MSK (cluster management, tuning, Kafka Connect, schema registry, consumer groups) and Amazon Kinesis.Proven track record in real-time processing and stream analytics.Languages: Python, Java, Scala. Serialization: Avro, Protobuf, JSON with schema evolution.Frameworks: Spark Streaming, Flink, Lambda. Data: SQL/NoSQL, warehousing, time-series, Hadoop ecosystem.Cloud & DevOps: AWS (MSK, Kinesis, Lambda, EMR, Glue, S3, Redshift), Docker, Kubernetes, Terraform/CloudFormation, CI/CD, GitHub, Agile.Observability: CloudWatch, Prometheus, Grafana, Confluent Control Center.Leadership: team management, strategic planning, risk mitigation, capacity planning.Certification: AWS Data Engineer Associate.NICE TO HAVEApache Flink, ksqlDB, Apache Pulsar, Apache NiFi, or Airflow; event sourcing / CQRS patterns; multi-cloud streaming.Background in financial services, IoT, or other real-time-heavy industries.Certs: Confluent Kafka (Admin/Developer), AWS Solutions Architect / Developer / Security / Big Data, Databricks Data Engineer Pro, Java/Scala/Python certifications.Applications from the following states are not acceptedCaliforniaWashingtonMassachusettsWashingtonOregonConnecticut