Databricks SME

Job Description
CSCI Consulting is looking for a Databricks Subject Matter Expert (SME) to support our Department of War client. In this role, the consultant will serve as the technical lead for Databricks-based data modernization initiatives, providing architecture, engineering, governance, analytics, and cloud data platform expertise to support mission-critical data, reporting, AI/ML, and decision-support capabilities.
Responsibilities
Serve as the primary Databricks SME supporting enterprise data and analytics initiatives
Design, develop, and optimize scalable data platforms using the Databricks Lakehouse architecture
Lead the implementation of data engineering, data integration, and data transformation solutions using Databricks and Apache Spark
Design and implement Medallion Architecture frameworks, including bronze, silver, and gold data layers
Develop and maintain ETL/ELT pipelines for structured and unstructured data sources
Configure and optimize Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables
Support cloud-native data solutions within Azure, AWS, or multi-cloud environments
Collaborate with business stakeholders, architects, developers, and data teams to translate requirements into scalable solutions
Support data governance, metadata management, data quality, and master data management initiatives
Optimize platform performance, scalability, availability, and cost efficiency
Develop technical documentation, architecture diagrams, data-flow diagrams, and operational procedures
Support enterprise analytics, business intelligence, artificial intelligence, and machine learning initiatives
Implement and support DevSecOps, CI/CD, and Infrastructure-as-Code practices
Integrate Databricks solutions with enterprise applications, APIs, databases, and cloud services
Support cybersecurity, compliance, audit, and data protection requirements
Provide technical mentorship and guidance to engineers, analysts, and project teams
Participate in Agile ceremonies, technical reviews, solution demonstrations, and program planning activities
Minimum Requirements
7+ years of experience supporting enterprise data engineering, analytics, or data platform initiatives
5+ years of hands-on experience with Databricks and Apache Spark
Experience designing and implementing enterprise data lake, data warehouse, or Lakehouse architecture
Strong expertise with Databricks Lakehouse Platform and Delta Lake
Advanced proficiency with SQL, Python, PySpark, and distributed data processing
Experience building and maintaining ETL/ELT pipelines and data integration solutions
Experience supporting Azure, AWS, or other cloud-based data platforms
Knowledge of data governance, data quality, metadata management, and data security best practices
Experience troubleshooting performance, scalability, and reliability issues in large-scale data environments
Experience developing technical documentation and communicating complex technical concepts to both technical and non-technical audiences
Ability to obtain and maintain the security clearance required by the program
U.S. Citizenship required where mandated by the applicable government contract
Able to obtain security clearance
Creativity and adaptability in problem-solving
Ability to work with clients to understand their needs
Strong organizational and time-management skills
Excellent written and verbal communication skills
Professional presence
Preferred Skills
Databricks Certified Data Engineer Associate, Professional Data Engineer, or related Databricks certification
Experience supporting Department of War, DoD, Intelligence Community, or Federal data modernization programs
Experience with Unity Catalog, Delta Live Tables, MLflow, Databricks Workflows, and Databricks Asset Bundles
Experience implementing modern data architecture including Data Mesh, Data Fabric, and Lakehouse strategies
Experience supporting Artificial Intelligence (AI), Machine Learning (ML), and Generative AI initiatives
Experience integrating Databricks with SAP, Salesforce, ServiceNow, Oracle, or other enterprise platforms
Experience with Power BI, Tableau, Qlik, or other enterprise reporting and visualization platforms
Experience implementing DevSecOps, CI/CD pipelines, and Infrastructure-as-Code frameworks
Experience using Azure DevOps, GitHub, GitLab, Jenkins, or other enterprise development platforms
Knowledge of Kubernetes, containerization technologies, and cloud-native architecture
Experience with Terraform, Bicep, or other infrastructure automation tools
Experience supporting data governance and compliance requirements within regulated environments
Experience working within Agile or SAFe delivery frameworks
Ability to work in a team environment, as well as independently
Strong customer and vendor relationship skills