Databricks Engineer

Location: -Remote

Duration: -6+ MonthsHere are some bullets on what they called out in my first conversation with them:

• Databricks experience required; they are building out data lakes in commercial, technical operations, etc.- would ideally like someone with Commercial experience and other areas
• Work is split between monitoring databricks pipelines, remedying issues, performing upgrades, etc. as well as some development work as needed
• Tie-ins to IDMC and PowerCenter
• Consulting firm experience could be good (e.g. Cognizant, TATA, Mindtree, etc. as long as have had both technical and functional experience)
• Does not need to lead the Databricks function, but has to be able to communicate with non-technical audiences and fellow team members -- strong comm skills required.

Day-to-Day Responsibilities: -Job Responsibilities:

• Design and develop scalable ETL/ELT pipelines for data extraction, ingestion, transformation, and processing using Apache Spark, Spark-Scala, PySpark, Spark SQL, and Databricks.
• Design and orchestrate Azure Data Factory (ADF) pipelines and cloud solutions to migrate and process data from on-premises RDBMS platforms to Azure Data Lake, Delta Lake, and Snowflake.
• Develop Python-based automation solutions for batch processing, data movement, workflow automation, monitoring, and alerting.
• Develop scalable real-time and streaming pipelines using Databricks Lakeflow Declarative Pipelines, Structured Streaming, PySpark, and Delta Lake.
• Develop and automate Snowflake data transformation pipelines using dbt, enabling scalable, reusable, version-controlled, and testable SQL workflows.
• Implement data profiling, data quality, validation, and standardization based on Enterprise Data Management (EDM) standards.
• Implement data governance using Databricks Unity Catalog, including catalogs, schemas, tables, external locations, access controls, permissions, and data lineage.
• Design databases, Data Lakes, and Data Warehouse solutions using Azure, Databricks, Delta Lake, Snowflake, and Microsoft SQL Server.
• Apply statistical and machine-learning techniques for predictive analytics and develop Business Intelligence dashboards using Power BI and Tableau.
• Collaborate with business stakeholders, architects, analysts, and application teams to understand data requirements, data sources, transformations, and data movement across enterprise systems.