JOBSEARCHER

Data Engineer

Role: Data Engineer Location: St. Louis, MOHybrid- 3 daysSkill Requirement1. Design and architect scalable, secure, and high-performance AWS and Databricks solutions, leveraging AWS Networking, Kinesis, Elastic Cache, Redshift, EKS, Cognito, and PaaS services2. Develop and maintain robust ETL/ELT pipelines using PySpark and Python within Databricks, including notebooks, jobs, Delta Lake tables, and Unity Catalog for governance3. Implement medallion architecture (bronze, silver, gold layers) and optimize Spark jobs for performance, cost, scalability, and reliability, addressing partitioning, skew, caching, and adaptive query execution4. Provision and manage cloud infrastructure using Terraform for Databricks workspaces, clusters, jobs, storage (ADLS/S3), networking, IAM roles and permissions, and related resources on Azure and AWS5. Write efficient SQL queries for data transformation, validation, and analytics within Databricks, ensuring data quality, lineage, security, and compliance6. Implement and maintain CI/CD pipelines using Jenkins and GitHub Actions for automated testing and deployment of notebooks, jobs, Delta Live Tables, and Terraform configurations7. Integrate data from diverse sources (databases, APIs, streaming, files) into cloud storage and processing layers8. Optimize cloud costs through auto-scaling clusters, spot instances, job scheduling, and efficient resource usage9. Monitor pipeline performance, troubleshoot failures, and implement alerting and observability using Databricks tools, cloud monitoring services, or third-party solutionsRequired Skills:1. Advanced proficiency in AWS Networking and VPC design2. Expertise in AWS Kinesis for real-time data streaming and analytics3. Deep experience with AWS Elastic Cache for distributed caching4. Extensive experience with AWS Redshift for data warehousing5. Strong knowledge of AWS PaaS services including Lambda, S3, and Glue6. Hands-on experience with AWS EKS for container orchestration7. Proficiency in AWS Cognito for identity and access management8. Expertise in Databricks, including Delta Lake, Unity Catalog, and Delta Live Tables9. Strong Python and PySpark skills for distributed data processing10. Advanced SQL skills for complex querying and optimization11. Proficiency in Terraform for infrastructure automation and management12. Experience with CI/CD tools such as Jenkins and GitHub ActionsPreferred Skills:1. Knowledge of Azure Data Lake, Data Factory, Synapse, and Key Vault2. Understanding of big data modeling (star/snowflake, dimensional) and lakehouse architecture3. Experience with performance tuning in Spark and Databricks environments4. Familiarity with data governance, lineage, and compliance in cloud environments