Live opening · Posted 18 hours ago
The key details from the original listing.
Live timing from JobBeeper.
Description supplied by the original job listing.
Company Description Codians.ai is a technology platform focused on IT outsourcing, talent hiring, and transparent reviews of IT companies and professionals. The company connects businesses with vetted IT experts, including developers, designers, and consultants, to support a wide range of project needs. Through a user-friendly interface and a large pool of trusted professionals, Codians.ai simplifies project outsourcing and hiring decisions. The platform emphasizes transparency, ratings, and feedback to help organizations find reliable partners while enabling IT professionals to showcase their skills and build their reputation. Codians.ai is committed to fostering an innovative, efficient, and quality-driven community in the IT industry.
Key Responsibilities
Design and build scalable batch and real-time data pipelines using Azure Databricks, PySpark, Python, SQL, and Delta Lake.
Implement Medallion Architecture (Bronze/Silver/Gold) for scalable and maintainable data platforms.
Develop governed data products using Databricks Unity Catalog.
Build real-time and streaming solutions using Databricks Structured Streaming and Azure Event Hubs.
Work extensively with ADLS Gen2, Azure Key Vault, Azure DevOps, and Git.
Configure and manage Databricks Workflows/Jobs for data pipeline orchestration.
Own CI/CD pipelines, deployment automation, release processes, monitoring, and production troubleshooting.
Optimize Databricks workloads and data pipelines for performance, scalability, reliability, and cost efficiency.
Build analytical datasets and dashboards using suitable BI and visualization tools.
Develop reusable data products capable of supporting analytics, machine learning, GenAI, RAG, and agentic applications.
Use AI coding assistants and AI agents for development, testing, documentation, debugging, troubleshooting, and engineering automation.
Implement appropriate cloud security, access-control, data-governance, and operational practices.
Collaborate with architects, analysts, business stakeholders, and engineering teams to deliver production-ready solutions.
Take ownership of production issues and ensure reliable operation of deployed data solutions.
Mandatory Skills & Experience
4–6 years of hands-on experience in Data Engineering, Data Platforms, or related roles.
Strong hands-on expertise in Azure Databricks.
Strong programming skills in Python and PySpark.
Strong SQL skills, including complex data transformation and optimization.
Hands-on experience with Delta Lake.
Strong understanding and implementation experience with Medallion Architecture – Bronze, Silver, and Gold layers.
Experience developing both batch and real-time/streaming data pipelines.
Hands-on experience with Databricks Unity Catalog.
Experience with Databricks Workflows/Jobs.
Hands-on experience with Structured Streaming.
Experience working with Azure Event Hubs.
Hands-on experience with ADLS Gen2.
Experience with Azure Key Vault and cloud security practices.
Experience with Azure DevOps and Git.
Good understanding of CI/CD, deployment automation, monitoring, and production operations.
Hands-on experience with at least one BI/dashboarding or data visualization tool.
Experience troubleshooting and optimizing data pipelines and Databricks workloads.
Strong problem-solving skills with an ownership-driven approach.
Comfortable using AI coding assistants/agents and willing to adopt emerging AI-native engineering practices.
Good to Have
Databricks and/or Microsoft Azure certifications.
Experience with Lakeflow / Delta Live Tables (DLT).
Knowledge of Terraform or other Infrastructure-as-Code (IaC) tools.
Experience with Apache Kafka or similar event-streaming platforms.
Experience working with Databricks on AWS or GCP in addition to Azure.
Exposure to Generative AI, RAG, LLM-based solutions, or agentic applications.
Experience developing data platforms that support ML/AI workloads.
Domain experience in Telecom, Media, Advertising, or AdTech.
Work arrangement
No
Recently tracked roles with full details and direct application links.