Data Engineer- Multiple data orgs Experience- Only W2
Position:
Data Engineer- Multiple data orgs Experience- Only W2Prior experience across multiple data orgs (WhatsApp, Instagram, Reality Labs, Infrastructure, Finance Technology)
Duration: Contract
Location: : REMOTEJob Description
The Data Engineer owns the data lifecycle end to end, from raw log ingestion to trusted dashboard metrics. You will develop, evaluate, test, and maintain scalable data architectures and pipelines, ensuring data remains accurate, timely, secure, reliable, and cost-effective. You will implement the strategies, policies, and practices that govern, protect, deliver, and enhance the value of organizational data assets. You will also leverage AI assistants and intelligent agents to improve productivity and automate workflows across the data engineering lifecycle.
Key Responsibilities:
- Manage data engineering projects through the complete lifecycle, from requirements gathering and design to deployment and production support.
- Identify and shape business initiatives from a data engineering perspective, collaborating with Product Managers, data scientists, and business stakeholders.
- Design, build, test, and maintain highly scalable data management systems and production pipelines using Python, SQL, Apache Spark, and Presto.
- Design, implement, automate, and maintain large-scale enterprise ETL/ELT processes across data ingestion, transformation, storage, and serving layers.
- Orchestrate pipelines with scheduling, retries, backfills, dependency management, and SLAs; deliver changes through code reviews and CI/CD practices.
- Ensure data systems meet business requirements, privacy policies, security standards, and data governance requirements, including access control and retention.
- Monitor data quality, observability, performance, cost, and reliability across the data lifecycle.
- Participate in on-call rotations, incident response, root cause analysis, and critical production issue investigations.
- Develop high-performance algorithms, prototypes, predictive models, proofs of concept, and AI agents to automate operational tasks.
- Build trusted datasets, semantic layers, and dashboards that enable stakeholders to access and analyze business metrics independently.
- Use AI-powered development and analytics tools to accelerate coding, testing, documentation, data discovery, troubleshooting, and operational reporting. Required Technical SkillsCore Languages and Frameworks
- Python
- SQL
- Apache Spark
- Apache Airflow
- PrestoDB or Trino
- Bash and shell scripting
- ETL/ELT development and data modeling
- Distributed data processing and large-scale data architecture Data Lifecycle and Platform Technologies Lifecycle Stage Capabilities Relevant Technologies Capture Logging, ingestion, data collection, and scraping Apache Kafka, Fluentd, Logstash, Apache NiFi, Scrapy, Playwright Move Pub/sub, streaming, and event processing Apache Kafka, Apache Pulsar, Amazon Kinesis Refine Data enrichment, cleansing, transformation, and aggregation Apache Spark, Apache Hive, PrestoDB, Trino, dbt Store and Secure Data lakes, warehouses, storage, and access control Amazon S3, Google Cloud Storage, Azure Data Lake, Delta Lake, Apache Iceberg, cloud IAMServe Semantic layers, BI, visualization, and self-service analytics Looker, Tableau, Power BI, Apache Superset, Metabase Orchestration Scheduling, retries, backfills, SLAs, and CI/CD Apache Airflow, Dagster, Prefect, Jenkins, GitHub Actions Configuration Feature flags, rollouts, and experimentation LaunchDarkly, Unleash, cloud-native configuration services Observability Monitoring, alerting, data quality, and incident management Prometheus, Grafana, Datadog, Open Telemetry, Great Expectations, Monte Carlo Development Source control, code review, and development environments Git, GitHub, GitLab, Bitbucket, VS Code Investigation Data discovery, lineage, and troubleshooting DataHub, Open Metadata, Apache Atlas, dbt documentation Equivalent technologies and experience with other enterprise data platforms are acceptable. AI Tooling and Automation Candidates are expected to use AI assistants regularly and develop intelligent agents that reduce manual work and improve operational efficiency. Tool Category Expected UseAI assistants and knowledge tools Knowledge retrieval, document summarization, technical research, and repetitive task automationAI coding assistants Pipeline development, code review, test generation, workflow scripting, documentation, and compliance support Data quality agents Automated data validation, anomaly detection, quality monitoring, and reporting Analytics assistants Data discovery, business question answering, exploratory analysis, and insight generation Autonomous agents Operational task triage, incident investigation, access request workflows, and repetitive process automation Familiarity with tools such as Claude, ChatGPT, GitHub Copilot, and other enterprise AI platforms is desirable. Professional Skills and Qualifications
- 5–10 years of professional data engineering experience, including designing and supporting production data pipelines at petabyte scale.
- Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field, or equivalent practical experience.
- Strong understanding of distributed systems, data architecture, data modeling, and large-scale data processing.
- Experience building and maintaining reliable, fault-tolerant, and cost-efficient production pipelines.
- Knowledge of data governance, privacy, security, access management, and retention policies.
- Ability to work collaboratively and independently with minimal supervision.
- Excellent written, verbal, presentation, and stakeholder communication skills.
- Experience collaborating with data architects, data modelers, Product Managers, data scientists, and IT teams.
- Proficiency with productivity and collaboration tools, including Google Workspace, Microsoft 365, documentation platforms, and project management tools.
- Relevant process or service management certifications, such as Six Sigma, CBPP, BPM, ISO 20000, ITIL, or CMMI, are desirable.
Nice to Have:
- Experience supporting large-scale data platforms across multiple business domains, such as social media, messaging, infrastructure, finance, or enterprise analytics.
- Proven track record of automating privacy, security, and governance compliance across large dataset and pipeline environments.
- Experience building autonomous agents to triage operational tasks, investigate incidents, or manage data access requests at scale.
- Experience developing and sharing reusable AI workflows, agent templates, and engineering best practices across teams.
- Familiarity with high-performance computing, distributed services, RPC frameworks, build systems, and large-scale software infrastructure.
- Experience mentoring junior engineers and establishing code review, testing, and development standards.
- AI-related certifications or demonstrated expertise in applying AI tools to software and data engineering workflows.