Senior Big Data Engineer
Job Title: Senior Big Data Engineer
Location: Charlotte, NC (Hybrid 3days a week)
Duration: 12+ Months
Job Description:
We are looking for a Senior Data Engineer
Resource should have minimum 10 years of hands-on experience in building production data systems.
Resource is expected to design, build, and maintain scalable data platforms while leading technical decisions.
Key technical skills - required:
Machine learning data pipelines
Generative AI and LLM data infrastructure
Programming
Python (advanced)
SQL (expert level)
Scala or Java (often preferred for big data)
Data Engineering skills required
ETL/ELT pipeline development
Data modelling (star schema, dimensional modelling)
Data warehousing (Snowflake, Big-Query, Redshift, Synapse)
Data lakes and lake-house architectures
Big Data Technologies skills required
Apache Spark (essential)
Hadoop ecosystem
Kafka (streaming)
Apache Airflow or similar workflow orchestration tools
Cloud Platforms skills required
AWS (S3, Glue, EMR, Lambda, Redshift)
Azure (ADF, Synapse, Databricks)
Google Cloud (Big-Query, Dataflow, Data-proc)
Databases skills required
PostgreSQL, MySQL
NoSQL databases (MongoDB, Cassandra, DynamoDB)
DevOps & Engineering Practices
Git
CI/CD pipelines
Docker
Kubernetes
Infrastructure as Code (Terraform is a plus)
Data Quality & Governance
Data validation and testing
Metadata management
Security and access control
Monitoring and observability
Leadership Skills
System design and architecture
Mentoring junior engineers
Stakeholder communication
Project estimation and technical planning
Nice-to-Have Skills
Databricks
DBT (Data Build Tool)
Delta Lake or Apache Iceberg