Data & Software Engineer

The Data & Software Engineer works with a small team to build complex data flows for a custom application. Successful candidate will have advanced Python programming skills, familiarity with Java, an understanding of data security, privacy, governance and compliance principles and a demonstrated history of building production data pipelines and ETL workflows at scale.ResponsibilitiesWork with stakeholders to understand data requirements, assess feasibility, and design appropriate solutions with minimal oversightLeverage strong problem-solving and debugging skills for data quality issues, pipeline failures, and performance bottlenecksLeverage a background in large-scale data migration or platform modernization effortsContribute to data engineering documentation, best practices, and design patterns RequirementsTS/SCI FSP ClearanceMinimum of 5 years' experience with:Apache Spark & PySparkAdvanced Python skills (including Pandas & NumPy)Docker, PodmanAWS S3, Lambda & Step functionsApache Iceberg, Airflow, etc.SQL (with Trino)NoSQL, DynamoDBUnity Catalog OSS, Apache PolarisApache SupersetTerraform or CloudFormationOpenLineageH3, PostGISCandidate must have experience in:Building end-to-end data pipelines leveraging PythonUsing orchestration tools to deploy data pipelines, including configuring and updating Spark JobsContainerizing and deploying applications in cloud environments like AWSWorking with MySQL and PostgreSQL including performance tuning, schema design, and query optimization for complex, analytical workloadsLeveraging industry standard tools for code control (Git, IaaC control, etc.)Working with data catalogs, tracking data lineageand handling a variety of data formats, including GeospatialUsing Bash scripting for automation and data processing tasksIntegrating Al/ML services and models About Us For more than 20 years, NewGen Technologies has solved our clients' toughest IT challenges with integrity, security, and outstanding service by delivering both technology and talent. We have helped secure borders, have used artificial intelligence (AI) to fight terror, aided the identification of criminals, and have helped to prevent crime through the introduction of biometrics. Our team of Highly Cleared Specialists have hard-to-find skills and expertise in a wide spectrum of technologies to provide solutions that transform business processes and solve problems of national significance. #CJ