Databricks Data Engineer Contract

Databricks Data Engineer - ContractLocation: Hybrid in London, 2 days per weekDuration: 6 month contractRate: £550 to £600 per day, Inside IR35

We are working with a leading organization that plays a key role in the UK's energy sector, driving forward innovative data solutions to support the transition to renewable energy. As part of their strategic initiatives, they are preparing for an upcoming regulatory change that will require detailed reporting on complex data products.

This organization manages a vast amount of critical data, primarily from smart meters and power generators, which is essential for building impactful dashboards and driving operational efficiency. With these developments on the horizon, the team is looking for an experienced contractor to assist with a growing backlog, enhance data pipelines, and improve the overall quality and output of their data operations.

Key Responsibilities

• Build and maintain scalable data lakehouse architectures using Databricks.

• Design and develop ETL/ELT pipelines for large-scale data processing.

• Manage both structured and unstructured datasets, optimizing performance and reliability.

• Set up structured streaming pipelines (using Kafka).

• Support Power BI reporting by preparing and transforming datasets.

• Deploy infrastructure and manage environments using Terraform and CI/CD practices.

• Collaborate with the engineering team to resolve backlogs and support data product delivery.

Required Experience & Skills

• Extensive experience with Databricks in production environments.

• Strong analytical skills with large, complex datasets (structured and unstructured).

• Proven experience in building ETL/ELT pipelines.

• In-depth understanding of Azure services (Data Factory, Azure Functions, Synapse, etc.).

• Advanced SQL skills, including performance tuning and query optimization.

• Strong Python programming skills.

• Experience with big data tools such as Hadoop, Spark, and Kafka.

• Proficiency in CI/CD processes and version control.

• Solid experience with Terraform and Infrastructure as Code (IaC).

• Experience with cloud-based networking and distributed systems.