Databricks Architect and Engineering Lead

About AbarVa
AbarVa is building Nexus, an AI-native enterprise execution platform that helps organizations turn strategic decisions into governed action, measurable outcomes, and sustained business value.
We are building Nexus alongside complex enterprise Data & AI transformations, particularly in regulated industries.
The Opportunity
We are looking for an experienced Databricks Architect and Engineering Lead to lead the architecture and engineering execution of large-scale data modernization and migration programs.
This is a hands-on architecture and engineering leadership role. We are looking for someone who has spent significant time personally building modern data platforms, has deep Databricks expertise, and can lead engineering teams through complex enterprise migrations.
This is not a strategy-only or architecture-only position. The successful candidate must be able to design, build, review code, troubleshoot complex engineering issues, establish engineering standards, and lead teams through production delivery.
Required Experience
10+ years of hands-on data engineering experience
5+ years of hands-on Databricks experience
Current Databricks certification is mandatory
Significant experience as a Databricks Architect, Data Engineering Lead, Principal Data Engineer or equivalent
Deep hands-on expertise with Databricks, Apache Spark, PySpark and SQL
Strong experience with Delta Lake, Unity Catalog, Databricks Workflows and modern Databricks ingestion and transformation patterns
Demonstrated experience designing and implementing enterprise-scale Bronze, Silver and Gold architectures
Proven experience leading large-scale migrations from legacy data platforms to Databricks
Experience assessing existing data workloads, ETL pipelines, data models, reports, KPIs and downstream dependencies
Strong experience designing metadata-driven ingestion and transformation frameworks
Strong knowledge of data quality, reconciliation, lineage, observability, security and governance
Strong cloud platform experience; AWS experience is highly desirable
Proven ability to lead and mentor teams of data engineers while remaining technically hands-on
Strong communication skills and experience working directly with senior client and technology stakeholders
What You’ll Do
Own the architecture and engineering execution of large-scale Databricks modernization and migration programs
Assess complex legacy data environments and develop practical modernization strategies
Determine which workloads should be migrated, refactored, redesigned, consolidated or retired
Design scalable enterprise Bronze / Silver / Gold architectures
Lead migration of legacy data warehouses, ETL platforms, analytics environments and reporting ecosystems to Databricks
Architect reusable, metadata-driven ingestion, transformation and orchestration frameworks
Lead enterprise data modeling and development of reusable data products
Establish engineering standards for performance, security, governance, lineage, observability and data quality
Lead reconciliation and parallel-run activities to prove Databricks outputs against trusted legacy platforms
Define migration waves, cutover plans and production stabilization approaches
Conduct architecture, design and code reviews
Troubleshoot complex engineering, scalability and performance issues
Lead and mentor distributed engineering teams
Partner directly with business and technology leaders to translate business outcomes into scalable data solutions
Introduce modern AI-assisted engineering approaches to improve development productivity while maintaining appropriate governance and human review
Help develop reusable AbarVa engineering methods, accelerators and intellectual property
Large-Scale Migration Experience
Candidates should have personally led significant migrations involving platforms such as:
Netezza
Teradata
Oracle / Exadata
SQL Server
Hadoop ecosystems
Traditional enterprise data warehouses
Legacy ETL and integration platforms
Externally hosted analytics platforms
Enterprise reporting and BI environments
Candidates should be able to clearly explain how they:
Assessed the existing data landscape
Identified workload and reporting dependencies
Rationalized legacy pipelines and reports
Designed the target Databricks architecture
Sequenced migration waves
Reconciled legacy and Databricks results
Managed parallel operations
Executed production cutover
Stabilized and optimized the new environment
Architecture & Leadership Expectations
This role requires someone who can lead from the front technically.
You will be expected to:
Own critical architecture decisions
Lead engineering teams through complex delivery programs
Establish architecture and engineering standards
Mentor architects and data engineers
Conduct detailed design and code reviews
Make difficult architecture and engineering trade-offs
Work directly with senior client technology and business leaders
Translate complex technical decisions into clear business implications
Take accountability from architecture through successful production implementation
We are not looking for someone who only manages engineers or produces architecture diagrams. You must be comfortable getting deeply hands-on when required.
Industry Experience
Experience in healthcare, financial services, insurance or another highly regulated industry is a significant advantage.
Healthcare experience is particularly valuable, including exposure to:
Health-plan / payer data
Claims
Membership, enrollment and eligibility
Medicare and Medicaid
Clinical and provider data
Epic / Epic Clarity
Healthcare financial and operational analytics
PHI and other regulated data environments
Deep healthcare expertise is not mandatory, but experience designing platforms where security, privacy, governance, auditability and data quality are critical is strongly preferred.
Work Authorization
Candidates must be legally authorized to work in the United States on a W-2 basis without current or future visa sponsorship.
AbarVa is not able to provide employment visa sponsorship for this role.
What Makes This Role Different
You will not simply create an architecture and hand it to another team.
You will be expected to lead discovery, architecture, engineering, migration, reconciliation, cutover and production delivery.
We are looking for a Databricks architect, hands-on engineer and engineering leader who has successfully delivered large-scale modernization programs and can build and lead high-performing teams.

Work arrangement
Yes