Platform Data Engineering Lead

Company Overview
Boston Insights is an innovative startup creating competitive advantage for pharmaceutical companies by unlocking their clinical supply chain data and enabling end-to-end visibility. We augment risk resiliency and agility to ensure uninterrupted supply of investigational drugs to patients on-time. Our mission is to transform how pharmaceutical companies manage their clinical supply chains through cutting-edge data solutions.
Key Responsibilities
Data Integration, Pipeline Development & Maintenance
Lead the integration strategy for ingesting and harmonizing data from external sources, including AWS-based data lakes/warehouses (such as S3, Redshift) and SAP systems.
Design, build, and maintain scalable data pipelines using Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake / MS Fabric OneLake.
Design reusable metadata-driven ingestion framework for rapid on-boarding of new datasets.
Hands-on experience in metadata driven pipelines with event-based ingestion.
Automate ETL processes for data extraction, transformation, and loading across hybrid and multi-cloud environments.
Build and maintain real-time and batch data integration workflows between Azure, AWS, and on-premises sources.
Data Architecture & Infrastructure
Design and implement data lake and data warehouse solutions on Azure platform.
Establish data governance frameworks and ensure data quality across all pipelines.
Implement security best practices for handling sensitive pharmaceutical data.
Create and maintain data documentation and lineage tracking.
Data Governance & Quality
Define and enforce data governance frameworks: data cataloging, lineage, quality, privacy, and compliance.
Implement robust data validation, cleansing, and monitoring systems to ensure accuracy and reliability.
Support security standards through effective data management practices.
Design and develop Microsoft Fabric-based AI agents leveraging OneLake, Fabric Data Agents, Semantic Models, Lakehouses, and enterprise data governance capabilities to support metadata discovery, lineage interpretation, policy validation, anomaly triage, and data stewardship workflows.
Automation & Tooling
Develop data automation tools and reusable components using Python, PySpark (and other relevant frameworks/languages).
Enable end-to-end process automation for data ingestion, processing, and reporting.
Implement CI/CD processes for data solutions, including testing, monitoring, and alerting.
Analytics & Reporting Support
Collaborate with data scientists and analysts to support advanced analytics.
Build data models that enable risk assessment and supply chain optimization.
Develop APIs and data services to support front-end applications.
Create monitoring and alerting systems for data pipeline health.
Microsoft Fabric AI Agent Development
Develop AI agents using Microsoft Fabric infrastructure and services, including Fabric Data Agents, Lakehouses, Semantic Models, OneLake, Notebooks, and Data Factory.
Build agents that support data cataloging, metadata enrichment, lineage interpretation, and data stewardship workflows.
Collaboration & Support
Partner with supply chain, analytics, and business stakeholders to understand business requirements and translate them into scalable technical solutions.
Collaborate with SAP functional and technical teams to optimize data extraction and synchronization.
Ideal Candidate Profile
The strongest candidate combines hands-on Azure data engineering expertise with the judgment to build governed, automated, and scalable data solutions for complex pharmaceutical supply chain environments.
Required Technical Experience
5+ years of professional data engineering experience.
Data Integration: Proven track record integrating data from AWS services (S3, Redshift, Glue, etc.) into Azure or other cloud environments.
Azure Data Services: Expert-level knowledge of Azure Data Factory, Azure Synapse Analytics, Data Bricks, Azure Data Lake Storage, Azure SQL Database, and Apache Spark.
Database Technologies: Strong knowledge of both relational (SQL Server, PostgreSQL) and NoSQL (Cosmos DB) databases.
Programming Languages: Proficiency in Python, SQL, PySpark, and PowerShell for data automation and wrangling.
Microsoft Fabric AI Agent Development
Hands-on experience developing AI agents using Microsoft Fabric infrastructure and services, including Fabric Data Agents, Lakehouses, Semantic Models, OneLake, Notebooks, and Data Factory.
Building agents that support data cataloging, metadata enrichment, lineage interpretation, and data stewardship workflows.
Designing agent workflows, retrieval patterns, governance controls, and security guardrails for enterprise pharmaceutical and supply chain data.
Integrating Fabric-based agents with data platforms, APIs, notebooks, pipelines, and automation frameworks to streamline data quality reviews, compliance checks, and anomaly detection.
Evaluating agent responses for accuracy, traceability, governance compliance, security, and business relevance.
Version Control: Proficiency with Git and Azure DevOps.
Hands-on experience with SAP data models and integrating SAP data with Azure data lake.
Preferred Technical Skills
Experience with API-based data integration for cloud and enterprise applications.
Experience with Infrastructure as Code (ARM templates, Terraform).
Familiarity with data quality tools, metadata management, and automated data lineage tracking.
Knowledge of containerization (Docker, Kubernetes) for data automation workflows.
Knowledge of machine learning pipelines and MLOps practices.
Experience with data visualization tool, Power BI.
Professional Skills
Strong problem-solving and analytical thinking abilities.
Excellent communication skills with ability to explain technical concepts to non-technical stakeholders.
Experience with Agile development methodologies.
Attention to detail and commitment to data quality.
The Opportunity We Offer
Competitive salary commensurate with experience.
Professional development training and certifications.
Application Instructions
Please Submit Your Resume And a Cover Letter Highlighting
Experience in Azure, AWS data integration, and SAP data extraction.
Examples of data automation tools or frameworks you have developed.
Experience developing Microsoft Fabric-based agents, Data Agents, Copilot-enabled workflows, or AI-assisted solutions that improve data governance, lineage visibility, data quality, compliance monitoring, and operational efficiency.
Join us to unlock new possibilities in pharmaceutical supply chain data through advanced engineering and multi-cloud innovation!
Boston Insights is transforming pharmaceutical supply chains through innovative data solutions. Join us in ensuring that life-saving investigational drugs reach patients on-time, every time.

Work arrangement
Yes