Data Engineer Databricks

APN Consulting, Inc. is a progressive IT staffing and services company offering innovative business solutions to improve client business outcomes. We focus on high impact technology solutions in ServiceNow, Fullstack, Cloud & Data, and AI / ML. Due to our globally expanding service offerings we are seeking top-talent to join our teams and grow with us.Position: Data Engineer DatabricksLocation: Oldwick, NJ (Hybrid)Duration: FTEOverviewThe Data Engineer is responsible for designing, building, and optimizing scalable data solutions to support a wide range of business needs. This role requires a strong ability to work both independently and collaboratively in a fast-paced, agile environment. The ideal candidate will engage with cross-functional teams to gather data requirements, propose enhancements to existing data pipelines and structures, and ensure the reliability and efficiency of data processes.ResponsibilitiesAssist with leading the team's transition to the Databricks platform and utilize the newer features of Delta Live Tables, Workflows etcDesign and develop data pipelines that extract data from Oracle, load it into the data lake, transform it into the desired format, and load it into Databricks data lakehouseOptimize data pipelines and data processing workflows for performance, scalability, and efficiencyImplement data quality checks and validations within data pipelines to ensure the accuracy, consistency, and completeness of dataHelp create and maintain documentation for data mappings, data definitions, architecture and data flow diagramsBuild proof-of-concepts to determine viability of possible new processes and technologiesDeploy and manage code in non-prod and prod environmentsInvestigate and troubleshoot data related issues and fix or provide solutions to fix defectsIdentify and resolve performance bottlenecks, which could include suggesting ways to optimize and performance tune databases and queries to enhance query performanceQualificationsBachelor's Degree in Computer Science, Data Science, Software Engineering, Information Systems, or related quantitative field4 plus years of experience working as a Data Engineer, ETL Engineer, Data/ETL Architect or similar rolesMust hold a current/active Databricks Data Engineer/Analyst certificationSkills4+ years of solid continuous experience in Python3+ years working with Databricks with knowledge and expertise of data structures, data storage and change data capture gained from prior production implementations of data pipelines, optimizations, and best practices3+ years of experience in Kimball dimensional modeling (star-schema comprising of facts, type1 and type2 dimensions, aggregates, etc.) with solid understanding of ELT/ETL3+ years of solid experience writing SQL and PL/SQL code2+ years of experience with Airflow3+ years of experience working with relational databases (Oracle preferred)2+ years of experience working with NoSQL databases: MongoDB, Cosmos DB, DocumentDB or similar2+ years of cloud experience (Azure preferred)Experience with CI/CD utilizing git/Azure DevOpsExperience with storage formats including Parquet/Arrow/AvroEffectively collaborate with team members while being able to work independently with minimal supervisionMust have a creative mindset, knack to solve complex problems, passion to work with data, and a positive attitudeAbility to collaborate within and across teams of different technical knowledge to support delivery and educate end users on data productsExpert problem-solving skills, including debugging skills, allowing the determination of sources of issues in unfamiliar code or systemsPluses, but not required: Any work experience in the following:ETL / ELT tools: Spark, Kafka, Azure Data Factory (ADF)Languages: R, Java, ScalaDatabases: Redis, ElasticsearchWe are committed to fostering a diverse, inclusive, and equitable workplace where individuals from all backgrounds feel valued and empowered to contribute their unique perspectives. We strongly encourage applications from candidates of all genders, races, ethnicities, abilities, and experiences to join our team and help us build a culture of belonging.