Kafka Architect

We are seeking an Expert Kafka professional with overall experience of 12+ Years.
7+ years of deep expertise in designing and implementing enterprise-scale event streaming solutions using Confluent Kafka. The ideal candidate will possess extensive experience in large-scale Confluent Kafka infrastructure (5+ years) and designing complex Kafka integration solutions (5+ years) that power real-time data pipelines, event-driven architectures, and mission-critical business applications. This senior role requires a technical professional who can design highly available, scalable, and secure Kafka ecosystems, establish streaming best practices, and drive the adoption of event-driven patterns across the organization. The Expert Kafka professional will serve as the subject matter expert for all Kafka-related initiatives, mentoring engineering teams, optimizing platform performance, and ensuring the reliability of streaming data infrastructure that supports business-critical operations.
Roles & Responsibilities
ROLES AND RESPONSIBILITIES
Confluent Kafka Infrastructure Architecture (5+ Years):
• Architect and design enterprise-scale Confluent Kafka infrastructure supporting high-throughput, low-latency event streaming requirements
• Design Kafka cluster topologies including multi-region, multi-datacenter, and hybrid cloud deployments • Plan and implement Kafka broker sizing, partition strategies, replication factors, and cluster scaling patterns
• Architect Confluent Platform components: Confluent Server, ZooKeeper/KRaft, Schema Registry, Kafka Connect, ksqlDB, Confluent Control Center
• Design high-availability and disaster recovery strategies for Kafka clusters including backup, replication, and failover mechanisms
• Implement Kafka security architectures including SSL/TLS encryption, SASL authentication, and ACL-based authorization
• Optimize Kafka infrastructure for performance, throughput, and cost-efficiency across cloud and on-premises environments
• Evaluate and select deployment options: Confluent Cloud, Confluent Platform on-premises, or self-managed Kafka on cloud infrastructure
Kafka Integration Solutions Design (5+ Years):
• Design end-to-end event streaming solutions using Kafka for real-time data integration and event-driven architectures
• Architect Kafka Connect solutions for integrating with databases, cloud storage, SaaS applications, and legacy systems
• Design stream processing applications using Kafka Streams, ksqlDB, or Apache Flink for real-time data transformation
• Implement event sourcing and CQRS (Command Query Responsibility Segregation) patterns using Kafka as the event store
• Design microservices integration patterns using Kafka for asynchronous communication and event-driven choreography
• Architect data pipeline solutions for ETL/ELT, data synchronization, and real-time analytics use cases
• Design schema management strategies using Confluent Schema Registry with Avro, Protobuf, and JSON Schema
• Implement exactly-once processing semantics and transactional messaging patterns for critical business applications
Performance Optimization & Tuning:
• Optimize Kafka cluster performance: throughput tuning, latency reduction, and resource utilization
• Tune Kafka producer and consumer configurations for optimal performance and reliability
• Design and implement partitioning strategies for balanced load distribution and parallel processing
• Monitor and optimize Kafka consumer lag, rebalance operations, and consumer group management
• Implement Kafka compaction strategies and log retention policies for storage optimization
• Tune JVM settings, garbage collection, and OS-level configurations for Kafka brokers
• Conduct performance benchmarking and capacity planning for future growth
Security & Governance:
• Architect comprehensive Kafka security models including authentication, authorization, and encryption • Implement SSL/TLS encryption for data in transit and encryption at rest for sensitive data
• Design SASL-based authentication mechanisms: SASL/PLAIN, SASL/SCRAM, SASL/GSSAPI (Kerberos), SASL/OAUTHBEARER
• Implement fine-grained ACLs and role-based access control for Kafka resources
• Design data governance frameworks for event streaming including data lineage, quality, and compliance
• Ensure compliance with regulatory requirements: GDPR, PCI-DSS, HIPAA, SOX for streaming data
• Implement audit logging and monitoring for security events and access patterns
Platform Operations & Reliability:
• Design monitoring and observability solutions for Kafka infrastructure using Confluent Control Center, Prometheus, Grafana
• Implement alerting mechanisms for cluster health, consumer lag, and critical metrics
• Create runbooks and operational procedures for Kafka cluster maintenance and incident response
• Design automated failover and self-healing mechanisms for Kafka infrastructure
• Plan and execute Kafka upgrades, migrations, and platform maintenance with minimal downtime
• Implement disaster recovery strategies including MirrorMaker 2.0 for cross-cluster replication
• Conduct chaos engineering and resilience testing for Kafka infrastructure
Integration Architecture:
• Design integration patterns between Kafka and enterprise systems: databases, data lakes, data warehouses
• Architect Kafka-to-Kafka replication strategies for multi-region and multi-cloud scenarios
• Design integration with cloud-native services: AWS Kinesis, Azure Event Hubs, Google Pub/Sub
• Implement CDC (Change Data Capture) patterns using Debezium or similar tools with Kafka
• Design API integration patterns using Kafka as the messaging backbone
• Architect hybrid cloud and multi-cloud Kafka deployments with data sovereignty considerations
Development & Best Practices:
• Establish Kafka development standards, coding guidelines, and best practices for engineering teams
• Design and review Kafka producer and consumer applications for performance and reliability
• Implement error handling, dead letter queues, and retry mechanisms for robust event processing
• Create reusable Kafka components, libraries, and templates for common patterns
• Establish testing strategies for Kafka applications including unit, integration, and chaos testing
• Mentor development teams on Kafka patterns, anti-patterns, and optimization techniques
• Conduct code reviews and architecture reviews for Kafka-based solutions
Stakeholder Collaboration:
• Collaborate with data engineers, software engineers, and architects to design streaming solutions
• Work with security and compliance teams to ensure Kafka implementations meet regulatory requirements
• Partner with infrastructure and DevOps teams for Kafka deployment automation and CI/CD integration • Engage with business stakeholders to understand streaming requirements and translate to technical solutions
• Present Kafka architecture designs, roadmaps, and recommendations to technical and executive audiences
• Contribute to internal communities of practice and knowledge sharing sessions
Innovation & Strategy:
• Research and evaluate emerging technologies in the event streaming and real-time analytics space
• Develop Kafka platform roadmaps and migration strategies for legacy integration systems
• Explore new Confluent Platform features and capabilities for potential adoption
• Drive adoption of event-driven architecture patterns and streaming best practices
• Identify opportunities for automation, optimization, and cost reduction in Kafka infrastructure
• Contribute to open-source Kafka community and stay current with Kafka ecosystem developments

Work arrangement
No