- UpvoteDownvoteShare Job
- Suggest Revision
Experience using big data technologies (Snowflake, Airflow, Kubernetes, Docker, Helm, Spark, pySpark) Experience designing and building highly available, distributed systems of data extraction, ingestion, normalization and processing of large data sets in real time as well as batch, that will be used across engineering teams using orchestration frameworks like Airflow, KubeFlow or other pipeline tools.
ExpandApply NowActive JobUpdated Today - UpvoteDownvoteShare Job
- Suggest Revision
Our platform is ever evolving, but currently is a combination of SQL, Kafka, Flume, Spark, Scala, Java, Python, NoSQL (HBase, Cassandra and ScyllaDB), MPP RDBMS, Postgres, Hadoop, AWS, AirFlow, Docker, Kubernetes, and Elastic.
ExpandApply NowActive JobUpdated Today - UpvoteDownvoteShare Job
- Suggest Revision
Experience with building stream-processing systems, using solutions such as Kafka, Storm or Spark-Streaming. Queer-Inclusive Benefits: Industry-leading gender-affirming offerings with up to 90% cost coverage, access to Included Health, monthly stipends for HRT, and more.
ExpandApply NowActive JobUpdated Today - UpvoteDownvoteShare Job
- Suggest Revision
Experience working with Airflow as workflow management tools and build operators to connect, extract and ingest data as needed. Experience in Real Time Data Ingestion using GCP PubSub, Kafka, Spark or similar.
Full-timeExpandApply NowActive JobUpdated 4 days ago - UpvoteDownvoteShare Job
- Suggest Revision
Includes Snowflake, Apache Airflow, AWS cloud infrastructure (e.g., Kinesis, Kubernetes/EKS, Lambda, Aurora RDS PostgreSQL), Python and Typescript. Experience building and managing production-grade data pipelines using tools such as Airflow, dbt, DataHub, MLFlow.
Full-timeExpandApply NowActive JobUpdated Today - UpvoteDownvoteShare Job
- Suggest Revision
We use Apache Airflow, Python, and Apache Spark for ETL. For streaming data, we use Apache Kafka managed by a vendor and use Kafka Connect, Kafka Streams, and Spark Streaming for stream processing.
Full-timeExpandApply NowActive JobUpdated 14 days ago - UpvoteDownvoteShare Job
- Suggest Revision
Design efficient data synchronization processes between on-premises Oracle databases and Snowflake using Kafka and CDC extraction tools. ETL Tools:Expertisein Airflow for data pipeline orchestration.
ExpandApply NowActive JobUpdated 4 days ago - UpvoteDownvoteShare Job
- Suggest Revision
Experience with multiple cloud-based data platforms (e.g., Microsoft Azure, Google Cloud, SAP Datasphere or SAP Datawarehouse cloud) and an expert ability to use various tools and frameworks that support data engineering tasks, such as Databricks, Snowflake, Spark, Kafka, Airflow, Azure, GCP, etc.
ExpandApply NowActive JobUpdated 5 days ago - UpvoteDownvoteShare Job
- Suggest Revision
You are a driven data engineer with experience working with big data technologies such as Docker, ECS, S3, Redshift, Kafka, and RDS. You're an Apache Airflow practitioner with experience managing cross-DAG dependencies in Airflow.
ExpandApply NowActive JobUpdated 14 days ago - UpvoteDownvoteShare Job
- Suggest Revision
Experience with containerization and container orchestration technologies (e.g., Docker, Kubernetes, Airflow) and their application to machine learning workflows. Experience with Messaging/Streaming Technologies (e.g., AWS SQS, Kinesis/Kafka), Relational and NoSQL databases (e.g., DynamoDB, EKS, Graph database), and Microservices Architecture.
ExpandApply NowActive JobUpdated 6 days ago - UpvoteDownvoteShare Job
- Suggest Revision
Big Data Ecosystem | 2+ years of hands-on, professional experience with tools and platforms like Spark Streaming, EMR, Kafka. Experience with any of ETL schedulers such as Airflow or similar frameworks.
ExpandApply NowActive JobUpdated 14 days ago - UpvoteDownvoteShare Job
- Suggest Revision
Familiar with cloud services like AWS, Azure, and workflow orchestration tools (e.g., Airflow). Experience in building real-time applications, preferably in Spark and streaming platforms like Kafka and Kinesis.
Full-timeExpandUpdated Today - UpvoteDownvoteShare Job
- Suggest Revision
Hands-on exp w/ Programming using Python & PySpark; Hadoop; HDFS, map-reduce, YARN, AWS EMR; Hive, HBase, parquet, ORC, Spark SQL, Sqoop, Apache Hudi; orchestrating ETL pipelines involving data sourcing, transformations & publishing using Apache Airflow; performance tuning apps in distributed computing enviros designing & developing data pipeline apps w/ Apache Kafka.
ExpandApply NowActive JobUpdated 14 days ago - UpvoteDownvoteShare Job
- Suggest Revision
Strong experience working with data solutions - ideally looking for candidates that have experience with Spark, Kafka and/or Airflow. 3+ years of experience with back-end software development using Python, Django and/or Flask.
Full-timeExpandApply NowActive JobUpdated 2 months ago - UpvoteDownvoteShare Job
- Suggest Revision
Familiar with Snowflake, Datadog, Kafka, and Airflow - advantage. Familiar with Snowflake, Datadog, Kafka, and Airflow - advantage. Listen to our team's playlist on Startup for Startup podcast.
ExpandApply NowActive JobUpdated 14 days ago
airflow kafka jobs in Chicago, IL
FEATURED BLOG POSTS
16 Tech Jobs You can Get Without a College Degree
You might think that if you don’t have a computer science, information technology, or related degree, then there’s absolutely no way you can break into the technology field and score a high-paying tech job. But this is a misconception. There are actually tons of tech jobs out there that don’t require a college degree. Instead, employers are more interested in the skills that you can offer. So, read on to learn more about how to land tech jobs without a degree.
What Are SMART Goals?
When it comes to achieving our goals, there’s a lot of noise to work through. A study by the University of Scranton has found that only 8% of people who set New Year’s resolutions actually achieve them. Our busy lives might be one reason for this. Another, even more important reason, is our approach to goal setting. Being too vague, too ambitious or simply unclear on the timeframe can set us up for failure.
How Long Do Background Checks Take for Pre-Employment
Many jobs require a background check. For some companies, this is a matter of company policy. In other jobs, such as finance, childcare, and security, the government often requires background checks. While there are varying levels of intensity, for most people, a background check is nothing to worry about. In fact, the biggest question is often how long does a background check take? Here’s what job seekers and applicants need to know about background checks before they apply to new jobs.
The Best Remote Jobs: Where & How to Find Them
The Covid-19 pandemic hasn’t been easy on any of us, but if there’s one silver lining, it’s the fact that remote work has grown in popularity because of it. Companies that previously weren’t open to their employees working remotely were suddenly forced into allowing it. Since then, they've realized that much of their workforce is happier and more productive. Naturally, this has led to more remote job openings, which is great if you’re interested in this type of position. Read on to learn more about the best remote jobs and where to find them.
In-House vs Outsourcing Recruiting: Which is Better?
When looking at in-house vs outsourcing recruiting, it is important to nail down the benefits for each and whether those benefits outweigh the risks that follow.
How to Write a Short Bio
First impressions can be tricky. When meeting someone in person, it’s likely you have an elevator speech. These short, practiced introductions can help you share more about who you are, what you do for work and other facts about you. While this works well in a live, in-person context, there are many cases where professionals “meet” someone via written form - like social media, a resume, or published work. In these cases, your bio works as your first impression. So, here’s a step-by-step guide on how to write a short bio.
How to Effectively Recruit Employees
Today, hiring and retaining talent looks much different than it did only two or three years ago. Financial instability and the strain on our mental health brought on by the pandemic has made everyone more wary and selective of their workplace and employer. Whereas in the past people might have prioritized promotions and financial reward, today they look at other factors such as workplace flexibility, personal fulfillment and values alignment.