Data Engineer / BI Analyst

Job Title: Data Engineer / BI Analyst Location: NJ (Hybrid – Morristown, NJ)Type: Full-timeSalary: up to $105k annually plus benefits, 401k, PTO, etc (Based on Experience)** US Citizen or Permanent Green Card holder only ***** Qualified candidates should have some experience writing PySpark jobs on Azure Databricks to ingest and transform data across a medallion lakehouse ***We are seeking a skilled and motivated Data Engineer to join our team, with deep hands-on experience building and optimizing data pipelines and lakehouse solutions in Databricks. In this role, you will collaborate with cross-functional teams to design, develop, and operate scalable, reliable data products that drive business value.Key Responsibilities:Design, build, and maintain batch and streaming data pipelines using Databricks (Spark, Delta Lake, Jobs/Workflows).Partner with data scientists, analysts, and application teams to deliver trusted, well-modeled data sets and features in the Databricks Lakehouse.Optimize Spark jobs (partitioning, caching, join strategies) and Databricks cluster configurations for performance, scalability, and cost.Implement data quality checks, observability, governance, and security controls (e.g., Unity Catalog, access policies) within Databricks.Troubleshoot and resolve pipeline failures, data issues, and production incidents; perform root-cause analysis and implement preventative improvements.Qualifications:Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience).Proven experience delivering production data engineering solutions in Databricks, including Apache Spark and Delta Lake on the Microsoft Azure cloud platform.Strong proficiency in SQL and Python (or Scala), including building ETL/ELT pipelines and writing efficient Spark code.Experience with data modeling and lakehouse/warehouse patterns (e.g., medallion architecture) and distributed data processing concepts.Excellent problem-solving skills and attention to detail.Strong communication and teamwork abilities.Preferred Skills:Hands-on experience with Databricks ecosystem capabilities such as Unity Catalog, Delta Live Tables, MLflow, and/or Databricks SQL.Experience implementing CI/CD and automated testing for Databricks notebooks/repos and deployment of Jobs/Workflows.Knowledge of data security, privacy, and compliance controls in cloud lakehouse environments (including role-based access and auditing).