Data Engineer (Databricks)
Data Engineer (Databricks)
Location: Princeton, NJ (Hybrid)
Duration: Contract
Job Description:
We are seeking an experienced Data Engineer with strong Databricks expertise to design, develop, and optimize modern cloud-based data platforms. The ideal candidate will have hands-on experience building scalable data pipelines, data lakes, and analytics solutions using Databricks and Azure cloud technologies.
Key Responsibilities:
Design, develop, and maintain ETL/ELT pipelines using Databricks, PySpark, Spark SQL, and Python.
Build and optimize data ingestion frameworks for structured, semi-structured, and unstructured data sources.
Develop and manage Delta Lake architectures and data models to support enterprise analytics.
Integrate data from multiple sources including APIs, databases, and cloud storage platforms.
Collaborate with business stakeholders, data analysts, and engineering teams to deliver data-driven solutions.
Ensure data quality, governance, security, and performance optimization across data platforms.
Support CI/CD implementation, code reviews, deployment automation, and production support.
Participate in Agile ceremonies and contribute to continuous improvement initiatives.
Required Skills:
5+ years of experience in Data Engineering.
Strong hands-on experience with:
- Databricks
- PySparkSpark SQLDelta LakePythonSQLExperience with Azure cloud technologies:
- Azure Data Factory (ADF)
Azure Data Lake Storage (ADLS):
- Azure Synapse Analytics
- Experience building large-scale data pipelines and data lake solutions.
- Knowledge of Git, CI/CD pipelines, and DevOps best practices.
- Strong analytical, troubleshooting, and communication skills.
Preferred Skills:
Banking, Financial Services, or Insurance (BFSI) domain experience.
Knowledge of data governance, data warehousing, and master data management concepts.
Experience with streaming technologies such as Apache Kafka or Spark Streaming.
Familiarity with cloud-native architecture and modern data engineering practices.