Data Engineer

Candidates should be eligible to work as a W2 employee (no 1099 or C2C) for any employer in the United States without needing Visa sponsorship. About the Role:Involves modernizing company's data ingestion architecture, replacing a portfolio of custom, failure-prone ingestion jobs with a managed data movement platform (Fivetran) feeding Azure Data Lake Storage and Databricks. We are hiring a hands-on Technical Lead to own the design, build, and delivery of this program end-to-end — not a role that hands off implementation to others, but one that writes connector code, configures the platform, builds observability dashboards, and is personally accountable for pipelines running reliably in production.This person will be the single technical owner bridging the data engineering, platform (Azure/Databricks), and analytics stakeholder groups, translating the target architecture into a working, monitored, production system — while building the internal capability for the company team to operate and extend it after go-live.Responsibilities: Architecture & DeliveryOwn the end-to-end technical design of the ingestion platform: Fivetran connector configuration, Managed Data Lake Service / ADLS landing pattern, Unity Catalog governance, and Databricks transformation layers.Lead discovery: inventory existing custom ingestion jobs and source systems, classify each as pre-built connector / custom SDK build / retain-as-is, and produce a Monthly Active Rows (MAR)-based cost model.Personally build and configure Fivetran connectors for standard sources, and design/develop custom connectors via the Fivetran Connector SDK (Python) for proprietary or league/stats data feeds.Design and implement the Databricks silver/gold transformation layer (Lakeflow/DLT or dbt) on top of the new ingestion layer.Decommission legacy custom jobs safely, with parallel-run validation to confirm data parity before cutover.Observability & ReliabilityBuild the pipeline observability layer: Fivetran alert routing, the Platform Connector-fed monitoring dashboard, and Databricks Lakehouse Monitoring / system-table based data quality checks.Define and implement SLAs for data freshness and pipeline health; establish on-call runbooks for pipeline failures.Act as the technical escalation point for pipeline incidents during and after migration.Cost & GovernanceOwn ongoing MAR governance — monitoring connector-level consumption, tuning sync scope, and flagging cost anomalies before they hit the monthly bill.Partner with security/network teams on requirements such as Azure Private Link and data residency where applicable.Leadership & EnablementMentor and pair with data engineers on the team, building internal Fivetran/Databricks capabilityCommunicate technical tradeoffs and status clearly to non-technical stakeholders (analytics, business ops, baseball operations) and to executive sponsors.Document architecture decisions, runbooks, and onboarding guides for future source additions.Required Skills:5+ years in data engineering, with at least 2 years leading or architecting production data pipelines — not just writing individual jobs.Hands-on production experience with Fivetran (or a comparable managed ELT platform — Airbyte, Matillion, Stitch) including custom connector development.Deep working knowledge of Databricks: Unity Catalog, Delta Lake, Lakeflow/DLT or equivalent, SQL warehouses, and cluster/job configuration.Strong experience with Azure Data Lake Storage (Gen2) and the broader Azure data ecosystem.Solid understanding of change data capture (CDC) concepts, incremental processing, and schema evolution — and the judgment to explain these tradeoffs to non-engineers.Proficiency in Python and SQL; comfortable building and debugging custom API/SDK-based connectors.Track record of decommissioning legacy systems safely (parallel-run validation, cutover planning) without data loss or downtime.Excellent stakeholder communication — able to run discovery workshops and translate business pain points into technical scope.Preferred Skills:Experience with sports, media, or ticketing/CRM data domains (Salesforce, ticketing platforms, POS/concessions systems).Familiarity with dbt for transformation layers.Experience standing up data observability/monitoring practices (SLAs, alerting, on-call) from scratch.Prior consulting or client-facing delivery experience, comfortable owning a statement-of-work-style engagement.Databricks or Azure certifications (e.g., Databricks Certified Data Engineer, Azure Data Engineer Associate).