Lead Data Engineer, Databricks (Remote, US)

About V4C.ai

V4C is a pure-play Databricks partner, and in under two years we have earned 850+ Databricks certifications across 325+ customers and been named Databricks' 2026 Partner of the Year (C&SI Emerging).

As a Lead Data Engineer, you will lead the build of production Lakehouse pipelines on customer engagements, working directly with customers and often Databricks' own field teams. Every technical hire here gets a certification roadmap built for them and taught in-house by Databricks-approved trainers, plus a dedicated Employee Success Manager whose only responsibility is your experience at V4C.

The Role

Lead Data Engineers are the technical backbone of V4C delivery. You lead the engineering workstream on an engagement — translating an architecture into working pipelines, setting the patterns the team follows, and owning the quality of what ships. You will work across batch and streaming, on messy enterprise source systems, under real deadlines, and in front of the customer explaining what you built and why. The difference from a Data Engineer is ownership: you are accountable for a workstream and the engineers on it.

Responsibilities

  • Lead engineering on customer engagements, from design through production handover
  • Build and optimize batch and streaming pipelines using PySpark, SQL, and Delta Lake on Databricks
  • Implement Medallion architecture across cloud ecosystems and set the layer conventions the team follows
  • Own performance and cost: benchmark Spark jobs, tune SQL, right-size compute, and explain the trade-offs to the customer
  • Implement data quality and validation as code, including Lakeflow expectations and automated testing in CI
  • Mentor mid-level and junior engineers through code review, pairing, and design feedback
  • Partner with the Data Architect on design decisions, and escalate when implementation reveals the design will not hold
  • 7+ years of data engineering plus experience leading projects or teams
  • Strong proficiency in Python, SQL, Apache Spark, and Delta Lake
  • Production experience building both batch and streaming pipelines at enterprise scale
  • Hands-on experience with AWS, Azure, or GCP and its data services
  • Experience with data modeling, Git, and CI/CD for data workloads
  • Demonstrated experience optimizing Spark performance and managing platform cost
  • Strong client communication skills, including the ability to explain technical trade-offs to a non-technical stakeholder

Preferences

  • Healthcare or life sciences experience preferred but not required
  • Existing Databricks Certified Data Engineer Associate or Professional certification
  • Hands-on experience with Unity Catalog, Lakeflow, or Databricks Asset Bundles
  • Experience with dbt, Airflow, or equivalent orchestration and transformation tooling
  • Experience migrating a legacy ETL estate to a modern cloud platform
  • Consulting or client-facing engineering background
  • Unlimited PTO
  • Health Care Plan (Medical, Dental & Vision)
  • Retirement Plan (401k, IRA)
  • Life Insurance (Basic, Voluntary & AD&D)
  • Family Leave (Maternity, Paternity)
  • Short Term & Long Term Disability
  • Training & Development
Back to blog

Other Jobs To Apply

Remote Account Executive, HubSpot Partner Sales

Remote Staff Backend Product Software Engineer, Commerce Platform

Remote Senior Digital Marketing Strategist -Startups/SMB (Remote US)

amazon flex

Regional Team Leader – South 1

Warehouse Assistant - Full-time

Remote Account Executive, HubSpot Partner Sales

AVP, Regional Credit Sales Manager Sam's Club (AZ, Phoenix Metro)

Remote Staff Backend Product Software Engineer, Commerce Platform

Remote Senior Digital Marketing Strategist -Startups/SMB (Remote US)

Remote Out of Office Position / Data Entry (Hiring Immediately)

amazon flex

Branded Content Writer, Freelance, Part-Time

Warehouse Assistant - Full-time

Remote Real Tool-Use Repair Clips Creator

AVP, Regional Credit Sales Manager Sam's Club (AZ, Phoenix Metro)

Remote Russian-English Healthcare Customer Service Specialist

Remote Out of Office Position / Data Entry (Hiring Immediately)

Branded Content Writer, Freelance, Part-Time

Staff iOS Engineer, Media Foundation - Remote

Remote Real Tool-Use Repair Clips Creator

Remote Russian-English Healthcare Customer Service Specialist

Remote Care Review Clinician

Staff iOS Engineer, Media Foundation - Remote

Sr. Project Management Analyst (Health & Casualty Claims implementations, Remote)

Remote Care Review Clinician

Remote Insurance Agent, West Coast

Sr. Project Management Analyst (Health & Casualty Claims implementations, Remote)

Remote Insurance Agent, West Coast

Medical Coder - AI Trainer (Remote)

Medical Coder - AI Trainer (Remote)

Remote Work From Home Administrative Assistant Admin - Part Time Panelists Needed

Remote Work From Home Administrative Assistant Admin - Part Time Panelists Needed

Remote Data Analyst

Remote Data Analyst

Senior Product Designer - Search & Findability (Remote)

Senior Product Designer - Search & Findability (Remote)

Remote Medical Assistant – CMA (Clinical & Admin)

Remote Medical Assistant – CMA (Clinical & Admin)

Remote Help Desk Specialist I - Secret Clearance

Remote Help Desk Specialist I - Secret Clearance

Solutions Architect, Enterprise (Retail, Restaurant & CPG) (Remote)

Sr. Project Manager - BESS

Sr. Project Manager - BESS

Amazon Customer Service

Amazon Customer Service

Remote Pharmacist — Diabetes Care Programs

Remote Pharmacist — Diabetes Care Programs

Sr Sales Manager

Sr Sales Manager

Common Interview Questions And Answers

1. HOW DO YOU PLAN YOUR DAY?

This is what this question poses: When do you focus and start working seriously? What are the hours you work optimally? Are you a night owl? A morning bird? Remote teams can be made up of people working on different shifts and around the world, so you won't necessarily be stuck in the 9-5 schedule if it's not for you...

2. HOW DO YOU USE THE DIFFERENT COMMUNICATION TOOLS IN DIFFERENT SITUATIONS?

When you're working on a remote team, there's no way to chat in the hallway between meetings or catch up on the latest project during an office carpool. Therefore, virtual communication will be absolutely essential to get your work done...

3. WHAT IS "WORKING REMOTE" REALLY FOR YOU?

Many people want to work remotely because of the flexibility it allows. You can work anywhere and at any time of the day...

4. WHAT DO YOU NEED IN YOUR PHYSICAL WORKSPACE TO SUCCEED IN YOUR WORK?

With this question, companies are looking to see what equipment they may need to provide you with and to verify how aware you are of what remote working could mean for you physically and logistically...

5. HOW DO YOU PROCESS INFORMATION?

Several years ago, I was working in a team to plan a big event. My supervisor made us all work as a team before the big day. One of our activities has been to find out how each of us processes information...

6. HOW DO YOU MANAGE THE CALENDAR AND THE PROGRAM? WHICH APPLICATIONS / SYSTEM DO YOU USE?

Or you may receive even more specific questions, such as: What's on your calendar? Do you plan blocks of time to do certain types of work? Do you have an open calendar that everyone can see?...

7. HOW DO YOU ORGANIZE FILES, LINKS, AND TABS ON YOUR COMPUTER?

Just like your schedule, how you track files and other information is very important. After all, everything is digital!...

8. HOW TO PRIORITIZE WORK?

The day I watched Marie Forleo's film separating the important from the urgent, my life changed. Not all remote jobs start fast, but most of them are...

9. HOW DO YOU PREPARE FOR A MEETING AND PREPARE A MEETING? WHAT DO YOU SEE HAPPENING DURING THE MEETING?

Just as communication is essential when working remotely, so is organization. Because you won't have those opportunities in the elevator or a casual conversation in the lunchroom, you should take advantage of the little time you have in a video or phone conference...

10. HOW DO YOU USE TECHNOLOGY ON A DAILY BASIS, IN YOUR WORK AND FOR YOUR PLEASURE?

This is a great question because it shows your comfort level with technology, which is very important for a remote worker because you will be working with technology over time...