Data Engineer

Hi ProfessionalsRole: Data EngineerLocation: RemoteDuration ContractJob Summary:The Data Engineer is responsible for designing, building, and maintaining scalable data pipelines and architectures to support data analytics and reporting needs. The role involves working closely with data scientists, analysts, and software developers to ensure efficient data flow, high data quality, and optimal data management practices. The ideal candidate will have experience with big data technologies, cloud platforms, and data integration processes.Key Responsibilities:Data Pipeline Development: Design, build, and manage robust data pipelines to collect, transform, and store large datasets from various sources.Data Integration: Integrate structured and unstructured data from internal and external sources, ensuring data is ready for analysis and reporting.Database Management: Develop and maintain databases, data warehouses, and data lakes using SQL and NoSQL technologies.Data Transformation: Clean, transform, and aggregate data using ETL (Extract, Transform, Load) tools and processes.Automation: Implement automation processes for repetitive data tasks and streamline data workflow processes.Data Quality: Ensure data integrity and consistency across all data pipelines, identifying and resolving data-related issues as they arise.Collaboration: Work closely with data scientists, analysts, and business teams to understand data requirements and deliver solutions that meet business objectives.Performance Optimization: Optimize the performance of data architectures to ensure scalability and efficient processing.Cloud Platform Management: Develop and manage cloud-based data platforms (e.g., AWS, Azure, Google Cloud) and ensure secure, reliable data storage.Monitoring: Monitor and troubleshoot data pipelines and systems for any performance issues, ensuring high availability and minimal downtime.Documentation: Create and maintain comprehensive technical documentation of all data engineering processes and systems.Required Skills & Qualifications:Education: Bachelor’s degree in Computer Science, Information Technology, or a related field. Master’s degree is a plus.Experience:9-12 years of experience in data engineering or related roles.Experience with big data technologies like Hadoop, Spark, and Kafka.Technical Skills:Proficiency in SQL, Python, Java, or Scala.Experience with ETL tools such as Apache Airflow, Talend, or SSIS.Familiarity with cloud platforms (AWS, Azure, Google Cloud) and their data services (e.g., Redshift, BigQuery, Databricks).Experience with data warehousing and database management systems (e.g., Snowflake, PostgreSQL, MongoDB).Knowledge of data modeling, schema design, and data governance.Experience with containerization technologies (e.g., Docker, Kubernetes) is a plus.Analytical Skills: Strong analytical thinking and problem-solving skills with the ability to work with large datasets.Communication Skills: Ability to communicate complex data concepts clearly and concisely with both technical and non-technical stakeholders.Attention to Detail: A keen eye for identifying and fixing data discrepancies and ensuring high data accuracy.Teamwork: Ability to work collaboratively in cross-functional teams.Share Quality Resumes @ Richard.Brown@crescentsofttech.com