Senior Data Engineer

Senior individual contributor responsible for the design, implementation, and continuous optimization of the enterprise Databricks platform. This role serves as the technical authority for all aspects of the Databricks environment — including workspace governance, Unity Catalog, cluster and compute strategy, data pipeline architecture, and cost management. The Architect works in close partnership with data engineering, analytics, and infrastructure teams, and operates within a broader multi-platform data ecosystem that includes Ab Initio and Fivetran. A strong background in Unix/Linux systems administration and scripting is essential, as the role requires deep engagement with the underlying compute infrastructure supporting the platform.Required Qualifications:• 7+ years of experience in data engineering or data platform roles, with a minimum of 4 years hands-on Databricks implementation experience.• Demonstrated expertise with Databricks platform capabilities: Unity Catalog, Delta Lake, Databricks Workflows, Delta Live Tables, and SQL Warehouses.• Strong Unix/Linux proficiency — shell scripting, process management, file system operations, cron scheduling, and environment configuration.• Proficiency in Python and PySpark for distributed data processing, pipeline development, and platform automation.• Experience with cloud infrastructure (AWS, Azure, or GCP), including compute, storage, networking, and IAM/security constructs.• Demonstrated ability to design for scale, cost efficiency, and operational reliability in an enterprise data environment.• Demonstrated experience designing automation frameworks for data platform operations — including job orchestration, monitoring, alerting, and pipeline self-healing.• Familiarity with AI/ML concepts and tooling within the Databricks ecosystem, including MLflow, AutoML, and Model Serving; exposure to generative AI or LLM-integrated workflows is a plus.• Experience with Oracle database environments, including SQL development, schema design, and integration patterns for data extraction and pipeline sourcing.• Proficiency in Git-based version control — branching strategies, pull request workflows, repository management, and CI/CD pipeline integration for data platform code.• Experience working within ITSM and project delivery frameworks such as ServiceNow and Jira.• Strong written and verbal communication skills, with the ability to convey complex architectural concepts to both technical and non-technical audiences.Prefered Qualifications:• Hands-on experience with MLflow experiment tracking, model registry, and deployment patterns within Databricks.• Exposure to generative AI frameworks (LangChain, LlamaIndex) or experience building LLM-integrated data pipelines and retrieval-augmented generation (RAG) workflows.• Experience with workflow automation tools such as Apache Airflow, Databricks Workflows, or comparable orchestration platforms at enterprise scale.• Experience integrating Databricks with ETL/ELT platforms including Fivetran, or Ab Initio; hands-on Ab Initio development or administration experience is a strong plus.• Familiarity with enterprise data governance frameworks and catalog tools (e.g., Collibra, Alation, or Unity Catalog advanced features).• Experience supporting Databricks in regulated industries (financial services, insurance) with associated audit and compliance requirements.• Working knowledge of Infrastructure-as-Code tooling (Terraform, Ansible) for platform provisioning and configuration management.• Background in disaster recovery design and resiliency planning for cloud-hosted data platforms.Core Technical CompetenciesPlatform & Data Engineering Infrastructure & SystemsDatabricks (Unity Catalog, DLT, Workflows) Unix/Linux Administration & ScriptingDelta Lake / Delta Live Tables Bash / Shell ScriptingApache Spark / PySpark EC2 / VM Compute ManagementMLflow, AutoML & Model Serving Python Automation & OrchestrationGenerative AI / LLM-Integrated Workflows Git Version Control & Repository ManagementFivetran, Ab Initio Integration Oracle Database / SQL DevelopmentCloud Storage (S3, ADLS, GCS) Secrets Management & Security HardeningSQL / SparkSQL ServiceNowAbout US Tech Solutions:US Tech Solutions is a global staff augmentation firm providing a wide range of talent on-demand and total workforce solutions. To know more about US Tech Solutions, please visit www.ustechsolutions.com.US Tech Solutions is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, colour, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.Recruiter Details:Name: Shiva EppalapelliEmail: shiva.e@ustechsolutionsinc.com