Data Platform Engineer – Data Engineering / ML Data / Video Data / Python / GDPR / ASPICE
Data Platform Engineer – Data Engineering / ML Data / Video Data / Python / GDPR / ASPICEWe are currently partnered with a global technology and engineering organisation developing advanced data platforms and machine learning technologies for large-scale visual and video datasets. As part of their continued investment in machine learning and data infrastructure, they are looking to hire a Data Platform Engineer – Video Data & ML Dataset Management to own the end-to-end platform for managing a large and growing collection of video data used for machine learning.Key responsibilitiesDesign and build automated ingestion pipelines for large video and media filesImplement validation, metadata extraction, deduplication, registration, and quality checks during ingestionDesign and maintain a robust relational metadata catalogue covering videos, people, recording campaigns, cameras, vehicles, and dataset usageEnsure strong data integrity through appropriate database relationships, constraints, validation, and lifecycle managementImplement GDPR and consent-management capabilities for video datasetsLink individuals to recorded video data and track consent status, expiry, retention, and relevant ownership informationAutomate notifications and workflows relating to consent and compliance requirementsMaintain appropriate audit trails and traceability for sensitive dataEstablish dataset versioning and reproducible train/test/validation tracking linked to ML experimentsKey requirementsStrong professional experience in data engineering and building production-quality data pipelinesExperience with data ingestion, ETL, workflow orchestration, and automationStrong relational database design and data modelling skillsUnderstanding of database constraints, relationships, data integrity, and lifecycle managementStrong Python programming skills for data pipelines, automation, and API developmentExperience handling large media or binary datasetsExperience with containerisation and secure deployment practicesExperience with dataset versioning and reproducibility for machine-learning workflowsExperience with data cataloguing and metadata-management platformsExperience integrating low-code platforms or internal tooling for data entry, workflow management, and visualisationPractical experience with video and media processingKeywordsData Platform Engineer / Data Engineer / Data Engineering / Data Platform / Data Pipelines / Data Ingestion / ETL / Workflow Orchestration / Python / Relational Databases / Database Design / Data Modelling / Data Integrity / Metadata Management / Data Cataloguing / Object Storage / File Storage / Large Media Data / Video Data / Video Processing / Machine Learning Data / ML Datasets / Dataset Management / Dataset Versioning / Dataset Reproducibility / ML Experiment Tracking / Visual Data / Data Governance / GDPR / Data Protection / PII / Consent Management / Data Retention / Audit Trails / Compliance / Data Dashboards / Reporting / APIs / API Development / Secure Deployment / Containerisation / Infrastructure Automation / ASPICE / ISO / Codebeamer / ALM / Requirements Management / Traceability / Automotive Data / Vehicle Testing / In-Car Testing / Data Collection / Data Infrastructure / ML Infrastructure / Data Lifecycle / Data Management / Data Automation / Data Platform Architecture / Data Operations / Engineering ToolsBy applying to this role, you understand that we may collect your personal data and process it in line with our privacy policy.