Senior Data Engineer / Databricks Engineer
Job Title: Senior Data Engineer / Databricks Engineer
Location: Remote 100%Duration : long Term
Employment: W2/C2CVisa: USC, GC, H4-EAD and H1BJob Summary:· 10+ years of total experience in Data Engineering or Big Data Development.· 7+ years of hands-on experience with Databricks and Apache Spark.· Proficiency in AWS cloud services: S3, Glue, Lambda, EMR, Redshift, CloudWatch, and IAM.· Strong programming skills in Python, PySpark, and optionally Scala.· Solid understanding of Data Lakes, Lakehouse, and Delta Lake concepts.· Experience in SQL development and performance tuning.· Familiarity with Airflow, dbt, or similar orchestration tools.· Experience with CI/CD tools such as Jenkins, GitHub Actions, or AWS Code Pipeline.· Knowledge of data security, governance, and compliance frameworks.
Roles & Responsibilities:
- Develop and maintain scalable data pipelines using Apache Spark on Databricks.· Build end-to-end ETL/ELT pipelines on AWS using S3, Glue, Lambda, EMR, and Step Functions.· Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality data solutions.· Design and implement data models, schemas, and Lakehouse architecture in Databricks.· Optimize and tune Spark jobs for performance and cost efficiency.· Integrate data from multiple structured and unstructured data sources.· Monitor and manage data workflows, ensuring data quality, consistency, and security.· Follow CI/CD, Git version control, and DevOps best practices for data applications.· Write clean, reusable, and well-documented code using Python, PySpark, and Scala.