How to build a data pipeline in DataLens
Everything you did by hand — load, clean, join, shape — assembled into something that runs the same way every time.
What you will do
You will assemble the steps of a data process into a pipeline, set the order they must run in, and run it as one unit rather than as a sequence you remember.
The value is repeatability. A pipeline runs the same way when you are on holiday.
When this is useful
- You are repeating the same load-clean-join sequence every week.
- A process works but only one person knows the order of the steps.
- New data arrives regularly and the downstream work should follow it.
- You need a process someone else can run and understand.
Before you start
- The steps working individually — Get the cleaning, the joins and the shaping right on their own first. A pipeline automates a working process; it does not fix a broken one.
- Knowing the dependencies — Which step must finish before which. This becomes the execution order.
Steps
Open Model → Pipeline Workbench
Select Model in the navigation, then Pipeline Workbench.
The Flow tab beside it shows how the pieces connect, which is useful for reading a pipeline someone else built.
The workbench canvas.
Add the steps
Assemble the pipeline from the datasets, views and transformations you have already built.
Each node is typed by what it is — Raw, Staged, View, CRM, Lake or Data Product — so a pipeline stays readable as it grows.
Nodes on the canvas representing each step.
Connect them in order
Connect the nodes so the flow of data is explicit: source, then staging, then the views built from it.
Nodes carry a stage — STAGE 0 for raw sources, STAGE 1 for what is derived from them.
Set the execution order
Use Execution Order to record what must run before what. Getting this wrong produces a pipeline that runs a join before the data it joins has been staged, which fails in a confusing way.
The execution order recorded on the pipeline.
Run it
Run the pipeline. Each step executes in the order you set.
Check the result against what you got when you did the steps by hand. A discrepancy is almost always an ordering problem.
The pipeline completing, with the outputs available as datasets.
Review the flow
Use the Flow tab to see the pipeline as a diagram. This is what you show someone who needs to understand the process without stepping through it.
What happens next
Pipeline output behaves like any dataset — chart it in Analyse, publish it in Deliver, or govern it in Govern.
Where the pipeline reads from a source that changes constantly, a CDC capture feeding it means it runs against current data rather than a snapshot.
Example
A weekly report took three hours of clicking: load two files, clean both, join them, aggregate, export. Assembled as a pipeline with the execution order set, it becomes one run — and the person who built it is no longer the only one who can do it.
Tips
- Get every step working alone before assembling. Debugging a pipeline is harder than debugging a step.
- Set the execution order explicitly rather than relying on how the canvas looks.
- Compare the first pipeline run against your manual result. It is the only real proof the pipeline does what you did.
- Use the Flow view when handing the pipeline over — it explains the process faster than a walkthrough.
Limitations
- A pipeline assembles steps available in DataLens. It does not execute arbitrary external code.
- Steps that fail stop the pipeline at that point; downstream steps do not run against incomplete input.
- A pipeline works on datasets inside DataLens. Refreshing a source is a separate concern — see CDC sync for keeping a source current.
Related how-to guides
Map source to target
Connect source columns to target columns, see what is unmapped, and export a source-to-target mapping.
Set up CDC sync
Keep a dataset current by capturing changes from a source table on a schedule, rather than reloading it whole.
Clean a messy dataset
Drop nulls, fill gaps, remove duplicates and cap outliers — with a live preview before anything is applied.
Trace where a column came from
Follow a field back from the dataset you are looking at to the source it came from, step by step.
Related questions
More of these on the DataLens FAQ page.
Can I automate a repeated data process in DataLens?
Yes. The Pipeline Workbench assembles the steps you have built — loads, cleaning, joins, shaping — into a pipeline with an explicit execution order, so the whole sequence runs as one unit instead of being repeated by hand.
What happens if a pipeline step fails?
The pipeline stops at the failing step, and steps that depend on it do not run. That is deliberate: running downstream work against incomplete input produces results that look fine and are wrong.
Try this in DataLens
DataLens is in private beta. Request access and work through this guide on your own data.
Request beta access