Databricks Data Engineer / Business Analyst – Data Contracts & OCDS

Position Summary
Scicom Infrastructure Services is seeking a Data Engineer / Business Analyst to support a large-scale Databricks implementation. This hybrid role will bridge business, contracting, data-governance, and engineering teams to define business requirements, construct data contracts, map source data, and develop reliable Databricks data pipelines.
The successful candidate must have hands-on experience with the Open Contracting Data Standard (OCDS) and understand how contracting and procurement information is structured across planning, tender, award, contract, and implementation stages. OCDS provides a standardized model for publishing and analyzing data throughout the public-contracting process and uses defined schemas, codelists, releases, records, and packages.
This individual will work closely with business stakeholders, procurement subject-matter experts, data architects, Databricks engineers, governance teams, and program leadership to translate complex business and contracting requirements into enforceable technical specifications and production-ready data products.
Key Responsibilities
Data Contracts and Business Analysis
Lead requirements-gathering sessions with procurement, contracting, program, analytics, governance, and technical stakeholders.
Define, construct, document, and maintain data contracts between data producers and consumers.
Establish data-contract requirements covering:
Dataset purpose and ownership
Source and target systems
Schemas, fields, and data types
Required and optional attributes
Business definitions and transformation rules
Data-quality expectations
Validation and reconciliation rules
Refresh frequency and delivery schedules
Versioning and schema-evolution requirements
Security classifications and access controls
Service-level expectations
Issue ownership and change-management procedures
Translate business requirements into user stories, acceptance criteria, process flows, data mappings, interface specifications, and technical requirements.
Conduct source-system analysis, data profiling, gap assessments, and source-to-target mapping.
Identify differences between existing procurement data and required OCDS structures.
Facilitate agreement among data owners, producers, consumers, architects, and governance teams.
Maintain traceability from business requirements through data models, engineering implementation, testing, and acceptance.
Evaluate requested changes for downstream effects on data products, reports, integrations, and analytical use cases.
Support backlog refinement, sprint planning, demonstrations, testing, and stakeholder acceptance.
OCDS Responsibilities
Apply the Open Contracting Data Standard to procurement and public-contracting datasets.
Map source-system data to appropriate OCDS fields and structures.
Work with data across the contracting lifecycle, including:
Planning
Tender
Award
Contract
Implementation
Develop and maintain mappings for OCDS releases, records, release packages, record packages, identifiers, organizations, parties, items, milestones, documents, transactions, amendments, and related contracting elements.
Interpret and apply OCDS schemas, codelists, validation rules, and implementation guidance.
Determine whether standard OCDS fields meet project requirements or whether documented extensions are necessary.
Support the construction of complete contracting records from multiple transactional releases.
Establish rules for handling amendments, updates, cancellations, corrections, and historical changes.
Validate transformed data against applicable OCDS JSON schemas.
Identify missing, invalid, inconsistent, or nonconforming procurement data and work with stakeholders to resolve deficiencies.
Document assumptions, business rules, mappings, extensions, and exceptions.
Support the publication, exchange, analysis, or internal use of standardized contracting data.
Databricks Data Engineering
Design, develop, test, and maintain data pipelines using Databricks, Apache Spark, PySpark, Python, and SQL.
Build ingestion and transformation pipelines for structured and semi-structured procurement data.
Process JSON, CSV, XML, Parquet, relational database, API, and file-based data sources.
Implement Bronze, Silver, and Gold data layers using medallion architecture.
Build normalized, dimensional, analytical, and OCDS-aligned data models.
Use Delta Lake capabilities for schema enforcement, schema evolution, versioning, auditability, and reliable processing.
Develop reusable frameworks for mapping source procurement data into OCDS-compatible outputs.
Implement batch and incremental ingestion patterns.
Use Databricks Workflows, notebooks, jobs, Auto Loader, Delta Live Tables or Lakeflow capabilities, as appropriate.
Develop REST API integrations for source ingestion and downstream data delivery.
Implement automated data-quality, reconciliation, completeness, and conformity checks.
Support Unity Catalog implementation for metadata, ownership, lineage, access control, and data discovery.
Optimize Spark jobs, SQL queries, clusters, partitioning, file sizes, and data layouts.
Participate in code reviews, automated testing, CI/CD, deployment, monitoring, and production support.
Investigate pipeline failures, data discrepancies, and performance issues.
Data Quality and Governance
Define measurable quality rules for accuracy, completeness, validity, timeliness, consistency, and uniqueness.
Develop validation controls for required OCDS fields, identifiers, dates, amounts, currencies, organizations, classifications, and contracting relationships.
Create dashboards or reports that show data-contract compliance and data-quality results.
Establish processes for detecting and managing schema drift.
Document data lineage from original procurement systems through Databricks transformations and downstream products.
Work with governance teams to assign data owners, stewards, classifications, retention requirements, and access policies.
Ensure sensitive procurement and supplier information is handled according to security and privacy requirements.
Support auditability through documented rules, version-controlled mappings, validation results, and change histories.
Required Qualifications
Bachelor’s degree in computer science, information systems, data analytics, business analysis, engineering, public administration, supply-chain management, or a related field.
At least five years of combined data-engineering, data-analysis, business-analysis, or data-integration experience.
Hands-on experience implementing or working with the Open Contracting Data Standard.
Demonstrated experience constructing, documenting, negotiating, or maintaining data contracts.
Experience mapping procurement or contracting data to OCDS schemas.
Strong knowledge of OCDS releases, records, schemas, codelists, identifiers, contracting stages, and validation practices.
At least three years of hands-on experience with Databricks.
Strong experience with:
Apache Spark
PySpark
Python
SQL
Delta Lake
ETL and ELT pipelines
Data modeling
JSON and JSON Schema
REST APIs
Data-quality validation
Source-to-target mapping
Experience gathering requirements and translating them into implementable engineering specifications.
Ability to communicate effectively with technical and nontechnical stakeholders.
Experience writing user stories, acceptance criteria, business rules, data dictionaries, interface specifications, and process documentation.
Strong analytical, troubleshooting, facilitation, and documentation skills.
Experience working within Agile delivery teams.
Preferred Qualifications
Experience with government procurement, public-sector contracting, grants, acquisition, supplier, or financial data.
Experience implementing OCDS extensions or tailoring OCDS for specific organizational requirements.
Familiarity with procurement classifications, organizational identifiers, tender processes, awards, amendments, milestones, transactions, and contract implementation.
Databricks Certified Data Engineer Associate or Professional certification.
Experience with Unity Catalog, Delta Live Tables, Lakeflow, Databricks Workflows, or Structured Streaming.
Experience with Microsoft Azure, AWS, or Google Cloud.
Experience with Azure Data Factory, ADLS Gen2, AWS Glue, Amazon S3, Snowflake, dbt, Kafka, Airflow, or Power BI.
Experience using JSON Schema validation tools and automated data-quality frameworks.
Knowledge of metadata management, master-data manageme

Employment type
Consultant

Work arrangement
Yes