SENIOR DATA ARCHITECT - Databricks

Job Description ? Senior Databricks Architect
Position
Senior Databricks Architect
Location
Offshore ? Bangalore
Domain
Retail / Grocery
Experience
10+ years overall IT experience, with 5+ years in Databricks / Cloud Data Engineering / Data Architecture
Role Overview
We are looking for an experienced?Senior Databricks Architect?to lead the architecture and technical execution of a large-scale?Talend-to-Databricks migration?for a leading Retail/Grocery client.
The architect will be responsible for designing the target?Databricks Lakehouse architecture, defining migration patterns, guiding the data engineering team, and ensuring successful modernization of existing Talend-based ETL/ELT workloads onto Databricks.
The role requires strong hands-on expertise in?Databricks, Apache Spark, SQL, cloud data platforms, data architecture, ETL modernization, and migration strategy, along with a good understanding of?Retail/Grocery business processes and data domains.
Key Responsibilities
Architecture & Solution Design
Lead the end-to-end architecture for migration of Talend ETL/ELT workloads to Databricks.
Define the target-state?Databricks Lakehouse architecture, including data ingestion, transformation, storage, orchestration, governance, and consumption layers.
Design scalable, secure, highly available, and cost-optimized data solutions.
Define architecture standards, design patterns, coding standards, and best practices for Databricks development.
Establish migration frameworks and reusable patterns for converting Talend jobs into Databricks/Spark-based pipelines.
Evaluate existing Talend jobs and determine appropriate migration strategies, including re-platforming, re-engineering, consolidation, or retirement.
Provide technical leadership for complex data pipelines and integration scenarios.
Talend to Databricks Migration
Analyze existing Talend workflows, jobs, mappings, dependencies, schedules, and data transformations.
Develop migration strategies for batch and incremental data processing workloads.
Translate Talend transformations and business rules into?PySpark/SQL/Databricks?implementations.
Identify opportunities to simplify and optimize legacy ETL processes during migration.
Define data reconciliation and validation strategies to ensure parity between Talend and Databricks outputs.
Establish migration sequencing based on business criticality, dependencies, complexity, and risk.
Support migration of high-volume and business-critical data pipelines.
Databricks & Data Engineering
Design and implement solutions using?Databricks, Apache Spark, Delta Lake, PySpark, and SQL.
Design?Medallion Architecture (Bronze, Silver, Gold)?and appropriate data processing patterns.
Implement incremental processing, CDC, SCD Type 1/Type 2, data quality, error handling, and audit frameworks.
Optimize Spark workloads, Delta tables, SQL queries, partitioning, clustering, and job execution.
Leverage?Delta Live Tables / Lakeflow capabilities, Workflows, Unity Catalog, and other Databricks platform services where appropriate.
Design data pipelines integrating structured and semi-structured data from databases, files, APIs, and enterprise applications.
Establish monitoring, logging, alerting, operational support, and performance-management patterns.
Cloud & Integration
Strong experience with at least one major cloud platform, preferably?Azure or AWS.
Design integration between Databricks and cloud storage, databases, messaging platforms, APIs, and enterprise applications.
Experience with technologies such as?ADLS/S3, Azure Data Factory, Kafka, REST APIs, relational databases, and cloud-native services?is highly desirable.
Define secure connectivity, networking, secrets management, and access-control patterns.
Retail / Grocery Domain
Work closely with business and technology stakeholders to understand Retail/Grocery data requirements.
Experience with retail data domains such as:
Product / Item
Store / Location
Customer / Loyalty
Sales / Transactions
Pricing
Promotions
Inventory
Supply Chain
Vendors / Suppliers
Orders
Forecasting
Merchandising
Understand retail-specific challenges such as high-volume transaction processing, store-level data, product hierarchies, promotions, pricing, inventory movements, and customer analytics.
Databricks

Work arrangement
No