Platform Engineer

The foundational role for the entire streaming layer. Owns Confluent Kafka, Schema Registry, Kafka Connect, and all integration paths from Confluent (AWS) to BigQuery and GCS (GCP). This role must be filled at Senior level, so that the platform can be built. Responsible for the cross-cloud streaming boundary that neither vendor fully owns.Technical skills and proficiency requirementsKafka internals - Consumer group protocol, ISR, controller election, log compaction — production experience required, not tutorial knowledge - ExpertConfluent Cloud - CKU model, cluster types, RBAC, metrics API, managed connectors, Confluent CLI - ExpertSchema Registry - Avro schema design, backward/forward compatibility, breaking change detection, Schema Registry ACLs - ExpertKafka Connect - Debezium CDC configuration, BQ Sink connector, GCS Sink connector, SMT (Single Message Transforms) - AdvancedAWS networking - VPC, security groups, IAM roles for Confluent, AWS Secrets Manager, Direct Connect/VPN concepts - IntermediateGCP networking - Private Service Connect, Private Google Access, VPC peering, VPC-SC access levels for cross-cloud Ips - IntermediatePython - confluent-kafka-python library, custom producer/consumer patterns, offset management, Schema Registry client - AdvancedTerraform - Confluent Terraform provider, GCP provider for network resources, AWS provider for SA key storage - IntermediateDual-cloud KMS - Confluent BYOK via AWS KMS, GCP Cloud KMS — separate key estates, rotation governance - IntermediateKey responsibilitiesKafka cluster ownership - Confluent Cloud provisioning, CKU sizing, network peering to GCP VPC, billing monitoringTopic design - Naming conventions, partition count, retention policy, compaction strategy per topicSchema Registry - Compatibility rule governance, schema evolution reviews, Avro/Protobuf standards enforcementIntegration path ownership - Kafka Connect BQ Sink (Option 2), GCS Sink (Option 3), Dataflow KafkaIO auth (Option 4)SA key governance - GCP service account keys stored in Confluent — rotation schedule, exposure monitoringConsumer group monitoring - Lag alerting per topic, rebalancing tuning, offset managementCross-cloud ops - Joint ownership of AWS-GCP boundary with Platform Infra Engineer — runbook, incident responseEgress monitoring - AWS CloudWatch bytes-out alerting, Confluent metrics API, daily egress cost reviewPlease provide years of experience:Kafka streamingPythonConfluentStreamingAWSPlatform EngineeringData GovernanceCloud integration