{"schemaVersion":"jobsearcher.job.v1","id":"d3ddb4b2bc57323b78469433","url":"https://jobsearcher.com/jobs/d3ddb4b2bc57323b78469433","canonicalUrl":"https://jobsearcher.com/jobs/d3ddb4b2bc57323b78469433","title":"Data Engineer","description":"Data Surge is looking for like-minded individuals who are passionate about being at the forefront of data technology. The Data Engineer will be joining an upcoming program supporting US Dept of Justice and is responsible for conversion of an Informatica legacy platform into AWS/Databricks using Python/PySpark.\n\n*Responsibilities:*\n\n* Converts existing Informatica mappings and mapplets into well-commented Python, preserving business logic, data quality checks, cleansing logic, aggregation, and address validation/standardization.\n* Performs data profiling and documents the data quality assessment ahead of migration (Task 7).\n* Executes bulk data transfer from HDFS to AWS S3/Lakehouse storage and converts to Delta/Parquet format.\n* Builds and tunes the pipeline scheduling, incremental sync, and reconciliation/validation logic needed to hit the SOW's 100 percent migration success standard (Task 7).\n* Replicates existing role-based access controls in the new environment and documents the access procedures.\n* Supports Unity Catalog structure, master data model, and workflow/approval configuration during system development.\n\n*Qualifications:*\n\n* Active public trust or higher-level clearance\n\nApplicable experience with the following technologies:\n\n* AWS GovCloud (Amazon Government Cloud)\n* Databricks; Delta Lake storage format; Unity Catalog (or equivalent governance/catalog layer)\n* Python 3+; PySpark\n* ANSI SQL (minimum)\n* Cloudera Hive; HDFS\n* Informatica (mappings & mapplets) — being converted, not retained\n* Parquet or similar high-performance columnar format\n* Change Data Capture (CDC), since the legacy system stays live during the build\n* Azure DevOps (government-hosted, server-based instance)\n* Okta, EntraID, Kerberos, Active Directory, DOJLogin\n* FedRAMP High, FISMA High, FIPS 199, NIST 800-53\n* Encryption at rest and in transit (FIPS-compliant); column-level encryption/masking\n* Splunk (audit log ingestion target)\n* Power BI; ESRI\n* VS Code, PyCharm, Posit (or similar) — native developer tool integration required\n* Section 508 (36 CFR Part 1194); WCAG 2.1 AA\n* Role-based access design: Administrator, Power User, User, Reader, service identity, auditor tiers\n\nPay: $126,902.13 - $152,828.38 per year\n\nWork Location: In person","company":"Data Surge","rawCompany":"data surge","city":"Washington","state":"DC","isRemote":false,"isActive":false,"createdAt":"2026-08-23T10:09:45.846Z","occupations":[{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-1243.00","title":"Database Architects","slug":"database-architects"},{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"518210","title":"Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services","slug":"computing-infrastructure-providers-data-processing-web-hosting-and-related-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Data Engineer","description":"Data Surge is looking for like-minded individuals who are passionate about being at the forefront of data technology. The Data Engineer will be joining an upcoming program supporting US Dept of Justice and is responsible for conversion of an Informatica legacy platform into AWS/Databricks using Python/PySpark.\n\n*Responsibilities:*\n\n* Converts existing Informatica mappings and mapplets into well-commented Python, preserving business logic, data quality checks, cleansing logic, aggregation, and address validation/standardization.\n* Performs data profiling and documents the data quality assessment ahead of migration (Task 7).\n* Executes bulk data transfer from HDFS to AWS S3/Lakehouse storage and converts to Delta/Parquet format.\n* Builds and tunes the pipeline scheduling, incremental sync, and reconciliation/validation logic needed to hit the SOW's 100 percent migration success standard (Task 7).\n* Replicates existing role-based access controls in the new environment and documents the access procedures.\n* Supports Unity Catalog structure, master data model, and workflow/approval configuration during system development.\n\n*Qualifications:*\n\n* Active public trust or higher-level clearance\n\nApplicable experience with the following technologies:\n\n* AWS GovCloud (Amazon Government Cloud)\n* Databricks; Delta Lake storage format; Unity Catalog (or equivalent governance/catalog layer)\n* Python 3+; PySpark\n* ANSI SQL (minimum)\n* Cloudera Hive; HDFS\n* Informatica (mappings & mapplets) — being converted, not retained\n* Parquet or similar high-performance columnar format\n* Change Data Capture (CDC), since the legacy system stays live during the build\n* Azure DevOps (government-hosted, server-based instance)\n* Okta, EntraID, Kerberos, Active Directory, DOJLogin\n* FedRAMP High, FISMA High, FIPS 199, NIST 800-53\n* Encryption at rest and in transit (FIPS-compliant); column-level encryption/masking\n* Splunk (audit log ingestion target)\n* Power BI; ESRI\n* VS Code, PyCharm, Posit (or similar) — native developer tool integration required\n* Section 508 (36 CFR Part 1194); WCAG 2.1 AA\n* Role-based access design: Administrator, Power User, User, Reader, service identity, auditor tiers\n\nPay: $126,902.13 - $152,828.38 per year\n\nWork Location: In person","datePosted":"2026-08-23T10:09:45.846Z","dateModified":"2026-08-23T10:09:45.846Z","hiringOrganization":{"@type":"Organization","name":"Data Surge","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Washington","addressRegion":"DC","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"d3ddb4b2bc57323b78469433"},"url":"https://jobsearcher.com/jobs/d3ddb4b2bc57323b78469433"}}