{"schemaVersion":"jobsearcher.job.v1","id":"39c47f6e601fa7d692b7da2d","url":"https://jobsearcher.com/jobs/39c47f6e601fa7d692b7da2d","canonicalUrl":"https://jobsearcher.com/jobs/39c47f6e601fa7d692b7da2d","title":"Principal Engineer, High Performance Data & Algorithm Infrastructure","description":"Principal Engineer, High-Performance Data Pipeline & InfrastructureLocation: San Diego, CAJob Type: Full-TimeSalary Range: $258,000 - $275,000Position OverviewWe are looking for a Principal Engineer to architect, build, and own theend-to-end data pipeline that drives our high-throughput diagnosticinstrument platform — from real-time image acquisition on theinstrument, through GPU-accelerated signal processing, to offloadingfor secondary and tertiary analysis on local HPC clusters and cloudinfrastructure.This is a technical leadership role for an engineer who can design anddeliver industrial-grade data processing infrastructure that operatesreliably at sustained high throughput. You will be responsible for thefull data path: acquiring raw image data from sensors, processing itthrough GPU pipelines, orchestrating job distribution across local HPCand cloud compute, and ensuring the entire system handles errors,backpressure, and recovery gracefully. The scope spans instrument-embedded software, on-premises Linux HPC infrastructure, and cloud-based compute and storage.The central challenge of this role is not raw compute optimization —GPU and CPU resources will have adequate headroom. The challenge isbuilding a pipeline architecture that is robust, scalable, and evolvableas instrument throughput increases with each generation, the numberof instruments grows, and data volumes scale accordingly. You willdesign systems that keep a complex multi-stage pipeline runningcontinuously and reliably in a production lab environment, and thatCan Be Evolved Without Wholesale Re-architecture As Requirementsintensify.Key ResponsibilitiesEnd-to-End Data Pipeline ArchitectureOwn the architecture of the complete data path from image acquisition to final processed outputDesign pipeline stages with clear interfaces, flow control, and backpressure mechanismsEnsure the pipeline sustains continuous high-throughput operation across extended instrument runsDefine data formats, handoff protocols, and buffering strategies between pipeline stagesArchitect for graceful degradation — the system must handle transient failures without data loss or pipeline stallsEstablish performance budgets and SLAs for each pipeline stage and monitor adherenceImage Acquisition & On-Instrument ProcessingDevelop and optimize real-time image acquisition from high-speed sensors on the instrumentImplement low-latency, high-bandwidth data capture with minimal frame lossDesign on-instrument preprocessing stages that reduce data volume before offloadManage memory and storage constraints within the instrument compute environmentEnsure deterministic, repeatable performance under sustained acquisition loadsGPU-Accelerated Signal & Image ProcessingDevelop and maintain GPU compute pipelines using CUDA for signal and image processingImplement DSP algorithms including frequency-domain analysis, deconvolution, filtering, and detectionManage host-to-GPU data transfers and ensure efficient use of GPU resourcesProfile GPU workloads to identify issues and validate performance headroomBalance numerical accuracy against throughput requirementsJob Orchestration & Distributed ProcessingDesign and implement job queuing, scheduling, and orchestration across instrument, local HPC, and cloud computeBuild robust work distribution that maximizes resource utilization across heterogeneous computeImplement backpressure handling so upstream stages throttle gracefully when downstream is saturatedDesign comprehensive error handling, retry logic, and dead-letter strategies for failed jobsEnsure jobs are idempotent and recoverable — partial failures must not corrupt the pipelineImplement priority scheduling to balance real-time instrument processing with batch reprocessingMonitor queue depths, processing latencies, and resource utilization with actionable alertingLinux Systems & PerformanceConfigure and tune Linux systems for reliable, high-throughput operation across instrument and HPC nodesTune kernel parameters (scheduler, NUMA, IRQs, huge pages) as needed for stable pipeline performanceUnderstand and manage DMA paths, PCIe topology, and device-to- memory data movementProfile and diagnose system-level issues using perf, ftrace, eBPF, and similar toolsEnsure system configurations are reproducible and documented across instrument and HPC environmentsHPC Compute Platform & Algorithm Infrastructure (co- owned with DevOps)Co-design the HPC compute platform architecture with DevOps — define computational requirements, job flow, and data access patterns while DevOps provisions and manages the infrastructureDefine how algorithms are deployed, versioned, and rolled into production on the HPC platform — support safe side-by-side execution of new and existing algorithm versionsDesign compute allocation strategies that balance real-time instrument processing, batch algorithm development/validation, and historical data reprocessingDesign the data handoff between instrument-side processing andHPC/cloud compute — formats, staging, transfer protocolsDefine storage tiering requirements for the processing pipeline — what data stays hot for active processing, what moves to warm for algorithm development access, and what archives to coldSpecify when and how workloads should burst from local HPC to cloud (AWS) based on pipeline load and priorityOptimize data movement across high-speed networks (RDMA,InfiniBand, high-speed Ethernet) between instrument, HPC, and storageDesign for scalability — the architecture must accommodate increasing instrument throughput, additional instruments, and growing algorithm complexityReliability & ObservabilityInstrument every pipeline stage with metrics, logging, and tracingBuild real-time dashboards showing pipeline health, throughput, latency, and queue stateDesign automated recovery mechanisms for common failure modesImplement data integrity checks and validation at pipeline stage boundariesSupport root-cause analysis and post-mortem investigation for pipeline incidentsEstablish runbooks and operational procedures for pipeline operationsQualificationsEducation:BS/MS in Computer Science, Electrical Engineering, or related field.PhD preferred.Required:Experience & Technical Leadership12+ years of professional software engineering experience inperformance-critical systemsTrack record of architecting and delivering complex, multi-stage data processing pipelinesDemonstrated technical leadership — ability to drive architecture decisions and mentor engineersExperience operating systems at industrial-grade reliability and throughput requirementsSystems Programming & GPU ComputingExpert-level C/C++ and systems programming on LinuxSolid experience with CUDA programming and GPU pipeline development (required)Strong understanding of computer architecture: CPU caches,NUMA, memory hierarchies, PCIe, DMAExperience with Python for tooling, orchestration, and pipeline glueExperience with performance profiling and diagnostics tools (perf, ftrace, Nsight, or similar)Pipeline & OrchestrationExperience designing multi-stage data pipelines with flow control, buffering, and backpressure managementStrong understanding of error handling, retry strategies, and fault recovery in performance-critical systemsExperience with job scheduling and work distribution across heterogeneous compute resourcesFamiliarity with workflow orchestration frameworks (Airflow, Celery, custom solutions, or similar) is a plusSignal Processing & AlgorithmsPractical experience implementing DSP or image processing algorithms in production systemsFamiliarity with frequency-domain analysis, filtering, and detection algorithmsAbility to reason about numerical accuracy and throughput tradeoffsData Movement, Storage & NetworkingExperience optimizing data transfer across high-speed networks (RDMA, InfiniBand, high-speed Ethernet)Understanding of shared storage architectures, tiered storagestrategies, and high- throughput data stagingExperience defining compute platform requirements and collaborating effectively with infrastructure teamsFamiliarity with algorithm deployment and versioning in production compute environmentsPreferred:Experience with high-throughput diagnostic instrument, imaging, or scientific instrument data pipelinesExperience scaling a data pipeline through multiple hardware or throughput generationsExperience with GPUDirect RDMA or other hardware offload technologiesFamiliarity with real-time or low-latency Linux variantsBackground in scientific computing, computational physics, or bioinformaticsExperience designing systems that span embedded instrument software and datacenter infrastructureWhat Success Looks LikeThe end-to-end pipeline from image acquisition to processed output runs continuously and reliably at target throughputBackpressure and error handling work transparently — operators are not firefighting pipeline stallsJob orchestration seamlessly distributes work across local and cloud compute based on load and priorityPipeline performance is predictable, measurable, and well understood with clear per-stage metricsNew instrument generations with higher data rates can be accommodated through evolution, not redesignAdding instruments to the lab scales the pipeline without disproportionate complexity or operational burdenAlgorithm developers can deploy, test, and validate new algorithms on the HPC platform without disrupting production processingStorage tiering keeps the right data accessible at the right cost as volumes growCompensation Range: $260K - $270K","company":"Foresite Labs","rawCompany":"foresite labs","city":"San Diego","state":"CA","isRemote":false,"isActive":false,"createdAt":"2026-04-12T19:37:33.542Z","occupations":[{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"},{"code":"17-2061.00","title":"Computer Hardware Engineers","slug":"computer-hardware-engineers"},{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"518210","title":"Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services","slug":"computing-infrastructure-providers-data-processing-web-hosting-and-related-services"},{"code":"541715","title":"Research and Development in the Physical, Engineering, and Life Sciences (except Nanotechnology and Biotechnology)","slug":"research-and-development-in-the-physical-engineering-and-life-sciences-except-nanotechnology-and-biotechnology"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Principal Engineer, High Performance Data & Algorithm Infrastructure","description":"Principal Engineer, High-Performance Data Pipeline & InfrastructureLocation: San Diego, CAJob Type: Full-TimeSalary Range: $258,000 - $275,000Position OverviewWe are looking for a Principal Engineer to architect, build, and own theend-to-end data pipeline that drives our high-throughput diagnosticinstrument platform — from real-time image acquisition on theinstrument, through GPU-accelerated signal processing, to offloadingfor secondary and tertiary analysis on local HPC clusters and cloudinfrastructure.This is a technical leadership role for an engineer who can design anddeliver industrial-grade data processing infrastructure that operatesreliably at sustained high throughput. You will be responsible for thefull data path: acquiring raw image data from sensors, processing itthrough GPU pipelines, orchestrating job distribution across local HPCand cloud compute, and ensuring the entire system handles errors,backpressure, and recovery gracefully. The scope spans instrument-embedded software, on-premises Linux HPC infrastructure, and cloud-based compute and storage.The central challenge of this role is not raw compute optimization —GPU and CPU resources will have adequate headroom. The challenge isbuilding a pipeline architecture that is robust, scalable, and evolvableas instrument throughput increases with each generation, the numberof instruments grows, and data volumes scale accordingly. You willdesign systems that keep a complex multi-stage pipeline runningcontinuously and reliably in a production lab environment, and thatCan Be Evolved Without Wholesale Re-architecture As Requirementsintensify.Key ResponsibilitiesEnd-to-End Data Pipeline ArchitectureOwn the architecture of the complete data path from image acquisition to final processed outputDesign pipeline stages with clear interfaces, flow control, and backpressure mechanismsEnsure the pipeline sustains continuous high-throughput operation across extended instrument runsDefine data formats, handoff protocols, and buffering strategies between pipeline stagesArchitect for graceful degradation — the system must handle transient failures without data loss or pipeline stallsEstablish performance budgets and SLAs for each pipeline stage and monitor adherenceImage Acquisition & On-Instrument ProcessingDevelop and optimize real-time image acquisition from high-speed sensors on the instrumentImplement low-latency, high-bandwidth data capture with minimal frame lossDesign on-instrument preprocessing stages that reduce data volume before offloadManage memory and storage constraints within the instrument compute environmentEnsure deterministic, repeatable performance under sustained acquisition loadsGPU-Accelerated Signal & Image ProcessingDevelop and maintain GPU compute pipelines using CUDA for signal and image processingImplement DSP algorithms including frequency-domain analysis, deconvolution, filtering, and detectionManage host-to-GPU data transfers and ensure efficient use of GPU resourcesProfile GPU workloads to identify issues and validate performance headroomBalance numerical accuracy against throughput requirementsJob Orchestration & Distributed ProcessingDesign and implement job queuing, scheduling, and orchestration across instrument, local HPC, and cloud computeBuild robust work distribution that maximizes resource utilization across heterogeneous computeImplement backpressure handling so upstream stages throttle gracefully when downstream is saturatedDesign comprehensive error handling, retry logic, and dead-letter strategies for failed jobsEnsure jobs are idempotent and recoverable — partial failures must not corrupt the pipelineImplement priority scheduling to balance real-time instrument processing with batch reprocessingMonitor queue depths, processing latencies, and resource utilization with actionable alertingLinux Systems & PerformanceConfigure and tune Linux systems for reliable, high-throughput operation across instrument and HPC nodesTune kernel parameters (scheduler, NUMA, IRQs, huge pages) as needed for stable pipeline performanceUnderstand and manage DMA paths, PCIe topology, and device-to- memory data movementProfile and diagnose system-level issues using perf, ftrace, eBPF, and similar toolsEnsure system configurations are reproducible and documented across instrument and HPC environmentsHPC Compute Platform & Algorithm Infrastructure (co- owned with DevOps)Co-design the HPC compute platform architecture with DevOps — define computational requirements, job flow, and data access patterns while DevOps provisions and manages the infrastructureDefine how algorithms are deployed, versioned, and rolled into production on the HPC platform — support safe side-by-side execution of new and existing algorithm versionsDesign compute allocation strategies that balance real-time instrument processing, batch algorithm development/validation, and historical data reprocessingDesign the data handoff between instrument-side processing andHPC/cloud compute — formats, staging, transfer protocolsDefine storage tiering requirements for the processing pipeline — what data stays hot for active processing, what moves to warm for algorithm development access, and what archives to coldSpecify when and how workloads should burst from local HPC to cloud (AWS) based on pipeline load and priorityOptimize data movement across high-speed networks (RDMA,InfiniBand, high-speed Ethernet) between instrument, HPC, and storageDesign for scalability — the architecture must accommodate increasing instrument throughput, additional instruments, and growing algorithm complexityReliability & ObservabilityInstrument every pipeline stage with metrics, logging, and tracingBuild real-time dashboards showing pipeline health, throughput, latency, and queue stateDesign automated recovery mechanisms for common failure modesImplement data integrity checks and validation at pipeline stage boundariesSupport root-cause analysis and post-mortem investigation for pipeline incidentsEstablish runbooks and operational procedures for pipeline operationsQualificationsEducation:BS/MS in Computer Science, Electrical Engineering, or related field.PhD preferred.Required:Experience & Technical Leadership12+ years of professional software engineering experience inperformance-critical systemsTrack record of architecting and delivering complex, multi-stage data processing pipelinesDemonstrated technical leadership — ability to drive architecture decisions and mentor engineersExperience operating systems at industrial-grade reliability and throughput requirementsSystems Programming & GPU ComputingExpert-level C/C++ and systems programming on LinuxSolid experience with CUDA programming and GPU pipeline development (required)Strong understanding of computer architecture: CPU caches,NUMA, memory hierarchies, PCIe, DMAExperience with Python for tooling, orchestration, and pipeline glueExperience with performance profiling and diagnostics tools (perf, ftrace, Nsight, or similar)Pipeline & OrchestrationExperience designing multi-stage data pipelines with flow control, buffering, and backpressure managementStrong understanding of error handling, retry strategies, and fault recovery in performance-critical systemsExperience with job scheduling and work distribution across heterogeneous compute resourcesFamiliarity with workflow orchestration frameworks (Airflow, Celery, custom solutions, or similar) is a plusSignal Processing & AlgorithmsPractical experience implementing DSP or image processing algorithms in production systemsFamiliarity with frequency-domain analysis, filtering, and detection algorithmsAbility to reason about numerical accuracy and throughput tradeoffsData Movement, Storage & NetworkingExperience optimizing data transfer across high-speed networks (RDMA, InfiniBand, high-speed Ethernet)Understanding of shared storage architectures, tiered storagestrategies, and high- throughput data stagingExperience defining compute platform requirements and collaborating effectively with infrastructure teamsFamiliarity with algorithm deployment and versioning in production compute environmentsPreferred:Experience with high-throughput diagnostic instrument, imaging, or scientific instrument data pipelinesExperience scaling a data pipeline through multiple hardware or throughput generationsExperience with GPUDirect RDMA or other hardware offload technologiesFamiliarity with real-time or low-latency Linux variantsBackground in scientific computing, computational physics, or bioinformaticsExperience designing systems that span embedded instrument software and datacenter infrastructureWhat Success Looks LikeThe end-to-end pipeline from image acquisition to processed output runs continuously and reliably at target throughputBackpressure and error handling work transparently — operators are not firefighting pipeline stallsJob orchestration seamlessly distributes work across local and cloud compute based on load and priorityPipeline performance is predictable, measurable, and well understood with clear per-stage metricsNew instrument generations with higher data rates can be accommodated through evolution, not redesignAdding instruments to the lab scales the pipeline without disproportionate complexity or operational burdenAlgorithm developers can deploy, test, and validate new algorithms on the HPC platform without disrupting production processingStorage tiering keeps the right data accessible at the right cost as volumes growCompensation Range: $260K - $270K","datePosted":"2026-04-12T19:37:33.542Z","dateModified":"2026-04-12T19:37:33.542Z","hiringOrganization":{"@type":"Organization","name":"Foresite Labs","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"San Diego","addressRegion":"CA","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"39c47f6e601fa7d692b7da2d"},"url":"https://jobsearcher.com/jobs/39c47f6e601fa7d692b7da2d"}}