Senior Data Engineer (Real-Time Data Processing & Streaming)

We are looking for an experienced Senior Data Engineer to join Endava and contribute to the design, development, and maintenance of scalable, high-performance data processing solutions.
In this role, you will focus on real-time data processing, streaming technologies, and event-driven architectures, working with complex data pipelines that ingest, transform, enrich, and distribute data from connected physical devices to downstream applications and storage systems.
You will collaborate closely with Tech Leads, software engineers, and cross-functional teams to build reliable, efficient, and maintainable data processing solutions in a cloud-native environment.
Key Responsibilities:
Design, develop, and maintain scalable real-time and near-real-time data processing pipelines.
Implement streaming data processing solutions using technologies such as Apache Kafka, Apache Flink, and Apache Spark.
Develop and maintain Java-based services and integration components for processing data received from physical devices.
Implement complex data transformations, mappings, validations, and enrichments to support business and technical requirements.
Build and maintain event-driven integrations using Kafka, Kafka Streams, Kafka Connect, and messaging protocols such as AMQP.
Ensure reliable data delivery to downstream consumers, applications, and storage systems.
Implement and maintain MongoDB collections, data structures, and related application-level integrations, following architectural guidelines.
Develop, deploy, and maintain cloud-native services primarily on AWS, leveraging Kubernetes and managed cloud services.
Contribute to the modernization and migration of legacy data processing components toward Kafka-based streaming architectures.
Collaborate with technical leads and engineering teams to translate architectural designs into robust, production-ready implementations.
Troubleshoot performance, reliability, and data consistency issues across distributed data processing systems.
Contribute to code quality, automated testing, CI/CD practices, and continuous improvement of engineering processes.
Required Skills and Experience:
Strong professional experience in Data Engineering, with a focus on streaming data processing and distributed systems.
Advanced programming skills in Java (11+), including experience with the Spring framework.
Hands-on experience with Apache Kafka, including Kafka Streams, Kafka Connect, and the development of Kafka-based integrations.
Practical experience with streaming and distributed processing frameworks such as Apache Flink, Apache Spark, or Spark Streaming.
Solid understanding of event-driven architectures, asynchronous communication patterns, and messaging systems, including AMQP.
Experience designing and implementing data transformation, enrichment, and integration pipelines.
Experience developing and integrating microservices and distributed applications.
Hands-on experience with AWS cloud services, particularly S3, SQS, and EKS.
Experience deploying and operating containerized applications using Docker and Kubernetes.
Working knowledge of MongoDB, including collections, document structures, and Java-based data access.
Familiarity with Redis and its use in distributed applications.
Experience with Git-based development workflows and CI/CD pipelines, preferably using GitLab and GitLab CI/CD.
Strong troubleshooting and problem-solving skills, particularly in distributed, high-throughput data processing environments.
Ability to work collaboratively with technical leads and cross-functional engineering teams, while taking ownership of implementation and delivery.
Nice to Have:
Experience with Apache Pulsar or Apache Storm, particularly in legacy system maintenance or modernization initiatives.
Familiarity with Microsoft Azure services and hybrid or multi-cloud environments.
Experience processing telemetry, IoT, or other event data generated by physical devices.
Knowledge of SMTP-based integrations.
Experience migrating legacy messaging and stream processing solutions to modern Kafka-based architectures.
Understanding of performance optimization, fault tolerance, monitoring, and observability in distributed streaming systems.

Employment type
Full-time

Work arrangement
Hybrid