Data Engineer – Unified Data Layer (UDL) (f/m/div.)

Platform & Pipeline Engineering
Develop, automate, and maintain resilient data pipelines connecting on-premises
laboratory equipment, high throughput test benches, and edge systems to Microsoft Azure.
Investigate, unpack, and parse complex, non-standard, binary, proprietary, and undocumented file formats where standard connectors do not exist.
Implement parsers and normalization routines capable of ingesting diverse technical data streams, for example:
High-frequency sensor streams, time series, and bus telemetry.
Multi-dimensional scientific arrays, spatial grids, and spectral data.
- Hierarchical, nested, and unstructured binary logs or custom ASCII exports.
Event logs, operational metadata, and intermittent experimental batch runs.
Clean, validate, and transform fragmented or schema-less raw data into optimized, partitioned Lakehouse tables (Delta Lake/Parquet) ready for analytics and ML model training.
Deploy, monitor, and maintain pipeline resources on Azure using Terraform, Databricks, and containerized workflows following established team templates.
Collaboration & Lab Support
Work directly alongside experimental scientists, lab engineers, and technicians to decipher raw measurement files, clarify domain-specific semantics, and automate recurring data uploads.
Collaborate closely with solution architects and senior engineers to implement data ingestion patterns, cataloguing schemas, and governance standards.
Maintain clear technical documentation, code comments, and onboarding walkthroughs to help research teams connect their equipment seamlessly.
Support proof-of-concept initiatives exploring metadata integration with corporate AI assistants, data discovery catalogs, and modern data tools.
Education
BSc, MSc or Ph.D. in Data Engineering, Computer Science, Software Engineering, Applied Physics, Electrical Engineering, or a related field.
Experience
3+ years of professional experience in data engineering, building automated and reliable data pipelines.
1–2 years of practical experience handling heterogeneous, non-standard, or binary data sources (e.g., physical sensor logs, laboratory exports, hardware telemetry, or raw byte streams).
1+ years of hands-on experience utilizing Apache Spark (preferably Databricks) and Microsoft Azure in production environments.
Demonstrated experience deciphering, parsing, or converting non-standard, legacy, or custom data formats into structured data structures.
Technical Skills
Strong proficiency in Python for data manipulation (including binary data handling using modules like struct, byte manipulation, NumPy, Pandas, and PySpark) and Bash.
Solid understanding of modern Lakehouse concepts (Delta Lake), distributed data processing, SQL, and NoSQL/document datastores.
Proficient with Git, GNU/Linux environments, and working knowledge of Infrastructure as Code (Terraform).
Practical experience with standard transport mechanisms (e.g., REST APIs, SFTP, Kafka/Event Hubs, or IoT/streaming protocols).
Mindset & Soft Skills
Strong analytical mindset with a natural curiosity to open unfamiliar files, inspect headers, and solve data decoding puzzles.
Friendly, empathetic communication style with the ability to bridge conversations between laboratory technicians, domain scientists, and software teams.
C2 CEFR level in English, written and spoken.
Nice to Have
Experience with scientific/hierarchical file formats (e.g., HDF5, NetCDF, Parquet, Protobuf).
Familiarity with industrial telemetry or time-series historian protocols (e.g., OPC UA, MQTT).
Exposure to Azure analytics tooling (Azure Data Factory, Azure Event Hubs, Azure Data Explorer).
Experience with vectorial databases.
Languages
Fluent in Portuguese.
B2 CEFR level in English, written and spoken.
B2 CEFR level in German, written and spoken is a plus

Skills
Python, REST API, Git, Linux, Kafka, Azure, Terraform, SQL, Databricks, Pandas

Employment type
Full-time

Work arrangement
Hybrid