JOBSEARCHER

Airflow Architect – Big Data

Airflow Architect – Big Data Systems Austin, TX Onsite Fulltime/Permanent Job Summary Need an experienced Airflow Architect to design, architect, and lead enterprise-scale data orchestration solutions for modern big data platforms. The ideal candidate will have deep expertise in Apache Airflow, Big Data technologies, cloud-native architectures, and data engineering best practices to build scalable, reliable, and high-performance data pipelines.
Key Responsibilities:
  • Architect and implement large-scale data orchestration frameworks using Apache Airflow.
  • Design, develop, and optimize ETL/ELT pipelines for batch and near real-time processing.
  • Build scalable workflows integrating Hadoop, Spark, Kafka, Databricks, data lakes, and data warehouses.
  • Define Airflow deployment strategies, DAG standards, security, monitoring, and governance.
  • Implement workflow scheduling, dependency management, error handling, retries, alerting, and SLA monitoring.
  • Lead cloud-based data platform implementations on AWS
  • Optimize pipeline performance, scalability, reliability, and cost efficiency.
  • Establish CI/CD processes and DevOps practices for Airflow environments.
  • Collaborate with business stakeholders, data engineers, architects, and platform teams to translate requirements into technical solutions.
  • Mentor engineering teams on Airflow architecture, data orchestration patterns, and best practices.
Required Skills: Airflow & Data Engineering
  • Expert knowledge of Apache Airflow DAG design, scheduling, operators, sensors, and executors.
  • Strong experience in workflow orchestration, metadata management, and pipeline monitoring.
  • Hands-on expertise in ETL/ELT development and data pipeline automation. Big Data Technologies
  • Apache Spark / PySpark
  • Hadoop ecosystem
  • Apache Kafka
  • Data Warehousing concepts
  • Data Lakes and Lakehouse architectures
  • NoSQL databases Programming
  • Python (advanced)
  • SQL (advanced)
  • Shell scripting
  • Core Java (preferred) Cloud Platforms
  • AWS (EMR, Glue, S3, Redshift)
Preferred Qualifications:
  • 10+ years of experience in data engineering and big data platforms.
  • 3+ years of architecture experience designing enterprise data solutions.
  • Experience with Kubernetes, Docker, and cloud-native deployments.
  • Knowledge of Data Mesh, Data Fabric, and Modern Data Platforms.
  • Experience leading architecture reviews and technology strategy discussions. Success Metrics
  • Pipeline reliability and SLA adherence
  • Workflow performance optimization
  • Reduced operational incidents
  • Improved data quality and observability
  • Scalable and reusable orchestration frameworks