Position: Lead Data Engineer

Contract Type: Fixed term / Contract

Contract Duration: Start Date: 25 May 2026 – End Date: December 2026

Work Model: Hybrid (2-3 days a week)

Work Location: Sandton, Johannesburg, South Africa (Hybrid / Office-based as required)

Role Overview

We are seeking a Lead / Senior Data Engineer to design, build, and operate modern Databricks and Lakehouse data platforms that support advanced analytics, AI, and Generative AI use cases.

This role is a senior individual contributor position, operating within product-aligned, cross‑functional squads. The successful candidate will deliver high-quality, governed, scalable data assets consumed by analytics platforms, machine learning models, and Generative AI solutions, including LLM- and agent-based systems.

Key Responsibilities

1. Databricks & Data Platform Engineering

Design, build, and operate data solutions using Databricks, including:

• Delta Lake
• Databricks Jobs and Workflows
• Unity Catalog
• Notebooks and shared libraries
• Develop scalable, reliable Lakehouse architectures supporting analytics and AI workloads.

2. Data Enablement & Consumption

Enable data consumption for:

• Generative AI use cases (e.g. Retrieval-Augmented Generation, AI services, agent workflows)
• Analytics and reporting platforms
• Downstream operational and business systems
• Support feature-style and curated data access patterns required by AI and GenAI workloads.

3. Generative AI Data Enablement

Build and maintain data pipelines that feed Generative AI applications, including:

• Curated knowledge and reference datasets
• Structured and semi-structured data sources
• Metadata, lineage, and traceability for AI consumption
• Enable common GenAI data patterns such as:
• Retrieval Augmented Generation (RAG)
• Contextual and prompt data preparation
• Model input, output, and feedback data flows

4. Engineering Standards & Best Practices

Develop production-grade data pipelines using:

• Python
• SQL
• Apache Spark
• Implement automated testing, CI/CD, and deployment practices for data workloads.
• Ensure data solutions are:
• Observable
• Resilient
• Performant
• Cost-efficient
• Continuously improve data quality, reliability, and operational stability.

5. Collaboration & Ways of Working

• Act as a senior engineer within a cross-functional product squad.
• Collaborate closely with:
• Product Owners
• AI / Machine Learning Engineers
• Analytics teams
• Platform and security teams
• Provide engineering input into design discussions and delivery decisions.
• Support peer reviews and contribute to shared engineering standards.
• Provide mentorship and technical guidance, including involvement in AI Engineer development.

6. Risk, Governance & Run

• Ensure all data solutions comply with enterprise security, risk, and governance standards.
• Support the operational stability of data pipelines used by analytics and AI workloads.
• Participate in incident resolution and root cause analysis.
• Maintain appropriate technical documentation and runbooks.

Required Background & Experience:

• 10–15 years of industry experience in data engineering or related fields.
• 5+ years' operating as a Senior or Lead Data Engineer.
• Mandatory Technical Skills (with minimum experience)
• Databricks (hands-on): 2+ years
• Enterprise data lake / lakehouse architecture: 5+ years
• Python: 5+ years
• SQL: 5+ years
• Apache Spark: 5+ years
• Production-grade data platforms: 3+ years
• Enterprise or regulated environments: 5+ years

Mandatory Skills Summary:

• Databricks
• Data lake and lakehouse architecture
• Python
• SQL
• Apache Spark
• Production-grade data platforms
• Enterprise or regulated environments

Desirable / Beneficial Skills:

• Experience enabling AI, ML, or Generative AI use cases from a data engineering perspective

Familiarity with:

• RAG data patterns
• Feature-style or AI-serving datasets
• Vector-based or embedding-ready data workflows
• Experience working in Agile, product-aligned squads
• Exposure to cloud-native data platforms such as AWS or Azure

Desired Skills Summary:

• AI, ML, or Generative AI
• RAG data patterns
• Feature-style or AI-serving datasets
• Vector or embedding-ready data workflows
• Cloud-native data platforms (AWS or Azure)