Technical Lead

Overview

Key Responsibilities

• Design, implement, and maintain ingestion adapters for diverse real-time data sources, including protocol decoding, structural validation, and early rejection of malformed or incomplete messages.
• Convert decoded data into a stable canonical event envelope containing timestamps, source identity, normalized measurements, quality metadata, and complete lineage information.
• Develop time-synchronization, state-buffering, interpolation, and reference-frame normalization services to align observations across multiple streams.
• Own the central replayable event transport layer using Kafka (KRaft mode) so that all downstream components operate from the identical event history.
• Build Apache Flink stream-processing applications for event-time windowed correlation, hypothesis management, belief combination, and multi-stream reasoning while preserving uncertainty and conflicts.
• Implement the complete persistence strategy: Redis for hot operational and belief state, PostgreSQL for curated reference data and rules, ClickHouse for long-term history, lineage tables, and replay data.
• Guarantee deterministic recovery and hot-state rebuild from checkpoints plus replay without silently resetting or losing internal identities.
• Design and enforce explicit identity lifecycle rules (observation → internal track → hypothesis → fused object) with full audit trail for all merges, splits, and relinks.
• Collaborate with ML/AI engineers to treat model outputs strictly as evidential input (never as final truth) while preserving complete source lineage for every operational conclusion.
• Support first-class handling of partial or unresolved observations with proper uncertainty representation.

Mandatory Technical Skills

• 5+ years of production experience building real-time streaming data platforms.
• Deep expertise in Apache Kafka (KRaft mode) and Apache Flink (stateful event-time processing, windowed operations, exactly-once semantics).
• Strong Python skills, including high-performance data parsing and binary protocol handling (ONNX integration experience is a plus).
• Hands-on production experience with Redis, PostgreSQL, and ClickHouse (or equivalent OLAP store).
• Proven track record designing systems with strong data lineage, auditability, and full replay capabilities.
• Solid understanding of temporal data handling, uncertainty management, and event-time versus processing-time semantics.