Streaming Data Platform Engineer

Streaming Data Platform Engineer
W2 Contract
Pay Rate: $55 - $65 per hour
Location: Cupertino, CA - Remote Role
Job Summary:
We are seeking an experienced software engineer to help design and build large-scale, cloud-native data platform capabilities with a strong emphasis on real-time streaming. This is a platform engineering role, not a conventional ETL or analytics development position. The engineer will create reusable services, SDKs, frameworks, and self-service capabilities that enable internal consumers - including machine learning practitioners - to process, manage, and access high-volume data reliably.
Duties and Responsibilities:
Design, build, and operate distributed streaming systems using Apache Flink and/or Apache Spark Structured Streaming for high-throughput, low-latency workloads.
Develop production-grade platform services, APIs, reusable frameworks, and SDKs using Python and/or Java (Spring Boot).
Contribute to control-plane and data-plane architecture, including workload orchestration, configuration, lifecycle management, and operational reliability.
Enable scalable data management for ML consumers: dataset organization, efficient access and retrieval, indexing patterns, and repeatable training-data workflows.
Engineer observability, fault tolerance, checkpointing, recovery, scaling, performance optimization, and automated testing for distributed processing systems.
Partner with platform teams, application engineers, and ML practitioners to translate use cases into self-service capabilities and well-supported developer interfaces.
Apply security and governance practices, including authentication, authorization, data protection, and appropriate handling of sensitive datasets.
Requirements and Qualifications:
Strong hands-on real-time streaming experience with Apache Flink or Apache Spark Structured Streaming in production; able to explain design tradeoffs and incidents.
Deep understanding of distributed data processing, partitioning, state management, event-time processing, backpressure, and failure recovery.
Strong software engineering proficiency in Python or Java, including API/service design, automated tests, and maintainable code.
Experience building data platform capabilities, internal developer platforms, reusable libraries/SDKs, or large-scale backend systems - beyond individual pipelines.
Experience with microservices, containers, cloud-native deployment, CI/CD, monitoring, and production operations.
Ability to work across requirements ambiguity, evaluate architecture options, and communicate tradeoffs with engineering stakeholders.
Hands-on knowledge of both Flink and Spark, plus Databricks or comparable managed distributed-data platforms.
Experience with Java/Spring Boot and Python in the same platform ecosystem.
Experience supporting ML practitioners through training-data access, feature/dataset management, data quality, indexing, or governed data interfaces.
Familiarity with Kafka or comparable event-streaming infrastructure; schema evolution and exactly-once/at-least-once delivery tradeoffs.
Experience with Kubernetes, service orchestration, cloud object storage, data access governance, and RBAC/ABAC patterns.
Bayside Solutions, Inc. is not able to sponsor any candidates at this time. Additionally, candidates for this position must qualify as a W2 candidate.
Bayside Solutions, Inc. may collect your personal information during the position application process. Please reference Bayside Solutions, Inc.'s CCPA Privacy Policy at www.baysidesolutions.com.
Desired Skills and Experience
Real-time streaming, Apache Flink, Apache Spark Structured Streaming, Python, Java, Spring Boot, distributed data processing, cloud-native platform engineering, backend development, API and microservices design, reusable frameworks and SDKs, internal developer platforms, self-service tooling, control-plane and data-plane architecture, workload and service orchestration, configuration and lifecycle management, partitioning, state management, event-time processing, backpressure, fault tolerance, checkpointing and recovery, scalability, performance optimization, automated testing, maintainable code, containers, Kubernetes, CI/CD, observability and monitoring, production operations, Databricks, Kafka, schema evolution, exactly-once and at-least-once delivery, cloud object storage, ML training-data workflows, dataset and feature management, data quality, indexing and retrieval, governed data access, authentication and authorization, RBAC/ABAC, sensitive data protection, architectural tradeoff analysis, requirements clarification, cross-functional collaboration, technical communication

Work arrangement
Yes