JOBSEARCHER

Sr. Data Engineer – Data & Intelligence

Sr. Data Engineer – Data & Intelligence (RDMP)
Location: Frisco, TX ( onsite)
Must have Must-have skills: Snowflake, Databricks, PySpark, Python, advanced SQL, Azure, and CI/CD.
Nice to have: Exp with telecom/finance/revenue/billing data
Summary: Senior Data Engineer / Technical Lead who can architect and lead enterprise finance-data pipelines using Snowflake + Databricks + Python/PySpark + SQL, with strong Azure/cloud, streaming, CI/CD, data quality, and finance/revenue data experience. Additional skills: Snowflake + Databricks + Python/PySpark + SQL, with strong Azure/cloud, CI/CDCommentsLooking for -Senior/Lead Data Engineer with Snowflake + Databricks + PySpark + Azure, who has built high-volume enterprise data pipelines and preferably worked with telecom/finance/revenue/billing data________________________________________DATA PIPELINE DEVELOPMENT
  • Architect, design, and oversee development of enterprise-scale ELT/ETL pipelines for finance and revenue data (billing, revenue, GL, opex).
  • Define and enforce standards for batch, incremental, and streaming ingestion patterns (CDC, watermarking, event-driven ingestion).
  • Ensure idempotent, fault-tolerant, and highly scalable pipeline design across platforms.
  • Establish frameworks for error handling, retry strategies, dead-letter queue patterns, and operational resiliency.
  • Provide technical leadership for multi-source, high-volume data integration pipelines.________________________________________PLATFORM & TOOLING
  • Lead architecture and adoption of Snowflake and Databricks platforms for large-scale data processing and analytics.
  • Define best practices for:o Snowflake (Snowpipe, streams, tasks, query optimization, cost efficiency)o Databricks (PySpark, Delta Live Tables, Unity Catalog, job optimization)o dbt (modular design, testing frameworks, CI/CD integration, reusable components)
  • Establish and govern orchestration frameworks using Airflow / Azure Data Factory, including DAG standards, dependency design, and monitoring.
  • Evaluate and drive tooling strategy and platform standardization across teams.________________________________________CLOUD INFRASTRUCTURE
  • Architect and optimize cloud-native data platforms on Azure (ADLS Gen2, Event Hub, ADF, Key Vault) or AWS equivalents.
  • Define standards for infrastructure-as-code (Terraform, Bicep) and environment provisioning.
  • Drive cost optimization strategies (compute sizing, storage design, partitioning, workload isolation).
  • Ensure platforms are scalable, secure, and production-ready.________________________________________LANGUAGES & FRAMEWORKS
  • Provide deep technical leadership in:o Advanced SQL (query tuning, execution optimization, complex transformations)o Python / PySpark for distributed data processing
  • Guide teams on best practices, reusable frameworks, and performance optimization.
  • Oversee development standards for Spark, Scala (where applicable), and automation scripting.________________________________________STREAMING & REAL-TIME
  • Architect real-time and near real-time data processing solutions using Kafka / Event Hub and Spark Structured Streaming.
  • Define patterns for stateful processing, watermarking, checkpointing, and fault tolerance.
  • Lead implementation of real-time finance/revenue use cases such as reconciliation, anomaly detection signals, and operational reporting.________________________________________DATA QUALITY & TESTING
  • Establish enterprise frameworks for data quality, validation, and observability.
  • Define standards for:o Automated testing (unit, integration, regression)o Data validation (completeness, accuracy, consistency)o Data quality tools (dbt tests, Great Expectations, custom frameworks)
  • Ensure SLA monitoring, alerting, and data freshness tracking across all pipelines.
  • Drive proactive data quality and governance practices across teams.________________________________________DATA MODELING SUPPORT
  • Interpret and implement architect-defined enterprise data models (star, snowflake, data vault).
  • Provide guidance on:o SCD (Type 1/2) strategieso Partitioning, clustering, and performance optimization
  • Collaborate with architects to evolve scalable and reusable data models.
  • Support semantic layer enablement for analytics and reporting.________________________________________DEVOPS & ENGINEERING PRACTICES
  • Define and enforce CI/CD standards for data engineering (GitHub Actions, Azure DevOps).
  • Establish code quality, versioning, and deployment best practices (branching strategies, PR reviews, release pipelines).
  • Standardize environment promotion (dev → QA → prod) and release management.
  • Drive adoption of engineering excellence practices including reusable frameworks and templates.________________________________________SECURITY & GOVERNANCE
  • Lead implementation of enterprise-grade security and governance controls:o RBAC, row/column-level securityo PII and CPNI compliance (TISS-310)
  • Define standards for secrets management and secure pipeline design.
  • Ensure data lineage, auditability, and compliance readiness across platforms.________________________________________FINANCE DOMAIN KNOWLEDGE
  • Deep understanding of finance and revenue data domains, including:o Billing and revenue systemso GL structures and financial reportingo Revenue recognition and reconciliationo Period-end close cycles
  • Guide engineering teams on accurate implementation of finance logic.
  • Ensure high data integrity standards for regulated financial data.________________________________________SOFT SKILLS & COLLABORATION
  • Act as a technical leader and escalation point across engineering teams.
  • Partner with architects, product managers, analysts, and business stakeholders.
  • Drive cross-team alignment and solution consistency.
  • Communicate complex technical topics clearly to both technical and non-technical audiences.
  • Lead incident reviews and ensure continuous improvement.