Databricks Engineer

Role: Databricks Engineer

Experience

4–8 years

Location

As per business requirement

Employment Type

Full-time

Job Summary

We are looking for a skilled Databricks Engineer with strong expertise in designing, developing, and optimizing modern data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate should have experience building scalable ETL/ELT pipelines, working with large-scale data, and leveraging Apache Spark to deliver high-performance data solutions.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks.
  
• Build ETL/ELT workflows for batch and streaming data processing.
  
• Develop solutions using PySpark, Spark SQL, and Delta Lake.
  
• Implement Medallion Architecture (Bronze, Silver, Gold) for data transformation.
  
• Integrate data from various sources including relational databases, APIs, cloud storage, and streaming platforms.
  
• Optimize Spark jobs for performance, scalability, and cost efficiency.
  
• Collaborate with Data Architects, Data Scientists, BI developers, and business stakeholders.
  
• Implement CI/CD pipelines and deployment automation for Databricks workloads.
  
• Ensure data quality, security, governance, and compliance.
  
• Monitor, troubleshoot, and optimize production data pipelines.
  
• Document technical solutions and follow engineering best practices.
  

Required Skills

Core Technologies

• Databricks Lakehouse Platform
  
• Apache Spark
  
• PySpark
  
• Spark SQL
  
• Delta Lake
  
• Python
  
• SQL
  

Cloud Platforms (one or more)

• Microsoft Azure (preferred)
  
• AWS
  
• Google Cloud Platform
  

Azure Technologies (Preferred)

• Azure Data Factory (ADF)
  
• Azure Data Lake Storage (ADLS Gen2)
  
• Azure Synapse Analytics
  
• Azure Key Vault
  
• Azure DevOps
  

Data Engineering

• Data Warehousing
  
• Data Modeling
  
• ETL/ELT Development
  
• Batch Processing
  
• Streaming (Kafka/Event Hubs)
  
• Data Lake Architecture
  

DevOps & Version Control

• Git
  
• Azure DevOps / GitHub
  
• CI/CD Pipelines
  

Preferred Qualifications

• Experience with Unity Catalog.
  
• Knowledge of Databricks Workflows and Jobs.
  
• Hands-on experience with Delta Live Tables (DLT).
  
• Exposure to MLflow is an added advantage.
  
• Experience with data governance and security best practices.
  
• Familiarity with Infrastructure as Code (Terraform) is a plus.
  

Educational Qualification

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
  

Preferred Certifications

• Databricks Certified Data Engineer Associate
  
• Databricks Certified Data Engineer Professional
  
• Microsoft Certified: Azure Data Engineer Associate (DP-203)
  
• Azure Fundamentals (AZ-900)
  

Good to Have

• Experience with real-time analytics.
  
• Knowledge of Lakehouse architecture.
  
• Experience with Agile/Scrum methodologies.
  
• Strong analytical and problem-solving skills.
  
• Excellent communication and stakeholder management abilities.
  

Mandatory Skills

• Databricks
  
• PySpark
  
• Spark SQL
  
• Delta Lake
  
• Python
  
• SQL
  
• Azure/AWS/GCP (at least one cloud platform)
  
• ETL/ELT Development
  
• Data Lake Architecture
  

Nice to Have

• Unity Catalog
  
• Delta Live Tables (DLT)
  
• MLflow
  
• Kafka/Event Hubs
  
• Azure Data Factory
  
• Terraform
  
• Azure DevOps/GitHub Actions