Azure Databricks Architect

Azure Databricks & Agentic AI Architect
Who we are: -
At CitiusTech, we constantly strive to solve the industry's greatest challenges with technology, creativity, and agility. With over 8,500 healthcare technology professionals worldwide, CitiusTech powers healthcare digital innovation, business transformation, and industry-wide convergence for over 140 organizations through next-generation technologies, solutions, and products. We aim to accelerate the transition to a human-first, sustainable, and digital healthcare ecosystem with the world's leading Healthcare and life sciences organizations and our partners.
Here is an opportunity for you to make a difference and collaborate with global leaders to shape the future of healthcare and positively impact human lives.
Our vision: -
To inspire new possibilities for the health ecosystem with technology and human ingenuity.
Role Summary:
We are seeking a visionary Azure Databricks & Agentic AI Architect to design and implement next-generation AI-powered data platforms. This role combines deep expertise in Azure Databricks, Lakehouse Architecture, Data Engineering, and Generative AI to build intelligent, autonomous, and self-optimizing data ecosystems.
The ideal candidate will lead the adoption of Agentic AI within Data Engineering and AI-DLC, enabling autonomous data ingestion, transformation, quality management, lineage discovery, observability, optimization, testing, and governance.
Key Responsibilities
Agentic Data Engineering Leadership
· Design and implement AI-powered Data Engineering platforms leveraging Azure Databricks and Lakehouse architecture.
· Define autonomous workflows using AI Agents for:
· Data ingestion
· Data mapping
· Schema evolution
· Data quality remediation
· Metadata Enrichment
· Pipeline optimization
· Root cause analysis
· Establish frameworks for Human-in-the-Loop (HITL) decision-making and governance.
AI-Driven Data Lifecycle (AI-DLC)
· Lead architecture for AI-enabled Data Development Lifecycle across:
· Requirement analysis
· Data modeling
· Pipeline generation
· Automated testing
· Code review
· Documentation
· Deployment
· Monitoring
· Implement AI copilots to accelerate developer productivity.
· Enable automated lineage creation and intelligent impact analysis.
Lakehouse & Data Platform Architecture
· Design scalable Lakehouse platforms using:
· Azure Databricks
· Delta Lake
· Unity Catalog
· ADLS Gen2
· Databricks Workflows
· Delta Live Tables
Enterprise GenAI Integration
· Architect RAG-based solutions using enterprise data assets.
· Design agent orchestration frameworks using:
· Azure OpenAI
· LangGraph
· Semantic Kernel
· AutoGen
· MCP-enabled architectures
· Build domain-specific AI agents supporting Data Engineering and Analytics teams.
AI Governance & Responsible AI
· Define guardrails for enterprise GenAI adoption.
· Implement:
· Prompt governance
· Observability
· Cost monitoring
· Auditability
· Explainability
· Security controls
· Establish governance models for autonomous AI agents.
AI-Powered Platform Optimization
· Design self-healing data pipelines.
· Implement AI-driven:
· Incident triage
· Failure prediction
· Capacity planning
· Cost optimization
· SLA monitoring
· Enable intelligent workload placement and model routing.
Technical Skills
· Data Platform
· Azure Databricks
· Delta Lake
· Unity Catalog
· Azure Data Factory
AI & Agentic Frameworks
· Azure OpenAI
· Knowledge Graph
· RAG Architecture
· LangChain
· LangGraph
· MCP Protocol
· Vector Databases
· AI Agent Orchestration
Data Engineering
· PySpark
· Spark SQL
· Python
· SQL
· ELT/ETL Modernization
DevOps & AI-DLC
· Azure DevOps
· GitHub Actions
· CI/CD
· MLOps
· LLMOps
· Evaluation Frameworks
· AI Testing Frameworks
Leadership Expectations
· Drive AI-First Data Engineering transformation.
· Define enterprise patterns, accelerators, and reusable AI agents.
· Mentor architects, data engineers, and AI engineers.
· Lead executive conversations on AI adoption, ROI, and transformation roadmaps.
Preferred Certifications
· Databricks Certified Data Engineer Professional
· Databricks Certified Solution Architect
· Microsoft Azure Solution Architect (AZ-305)
· Azure Data Engineer (DP-203)
· Microsoft Applied Skills – Azure OpenAI
· Generative AI / Agentic AI Certifications
Success Metrics
· 30-50% Data Engineering productivity improvement.
· Reduction in manual pipeline development effort.
· Improved data quality and governance compliance.
· Measurable ROI from Agentic AI adoption.
· Expansion of reusable AI agents across programs.
Ideal Candidate Profile
A forward-looking architect with 15+ years of Data & Analytics experience, strong Azure Databricks expertise, and hands-on experience building Agentic AI platforms, AI-powered SDLC frameworks, and autonomous data engineering ecosystems. The candidate should be comfortable leading enterprise-scale AI transformation initiatives and engaging CXO stakeholders on AI strategy and business value.
Experience: - 16-20 Years
Location: - Chicago, IL(Hybrid- 3 days/week in office)
Educational Qualifications: -
Engineering Degree – BE/ME/BTech/MTech/BSc/MSc.
· Technical certification in multiple technologies is desirable.
Skills: -
Mandatory skills
· Azure, DataBricks, Agentic AI, ETL, SQL, Data Engineering
Life at CitiusTech
We focus on building highly motivated engineering teams and thought leaders with an entrepreneurial mindset, centered on our core values of Passion, Respect, Openness, Unity, and Depth (PROUD) of knowledge. Our success lies in creating a fun, transparent, non-hierarchical, diverse work culture that focuses on continuous learning and work-life balance.
Rated by our employees as the ‘Great Place to Work for’ according to the Great Place to Work survey. We offer you a comprehensive set of benefits to ensure that you have a long and rewarding career with us.
Our EVP
Be You Be Awesome is our EVP and it reflects our continuing efforts to create CitiusTech as a great place to work where our employees can thrive, both personally and professionally. It encompasses the unique benefits and opportunities we offer to support your growth, well-being, and success throughout your journey with us and beyond. Together with our clients, we are solving some of the greatest healthcare challenges and positively impacting human lives. Welcome to the world of Faster Growth, Higher Learning, and Stronger Impact.

Work arrangement
Yes