How to set up CDC sync in DataLens
Capture changes from a source table on a schedule instead of reloading the whole thing — with a free polling mode and a Pro log-based mode.
What you will do
You will create a change capture against a PostgreSQL or MySQL table, choose how changes are detected, set how often it runs, and end up with a dataset that keeps itself current.
This is the difference between a snapshot that ages and a dataset that tracks its source.
When this is useful
- A dataset needs to stay current and reloading the whole table is too slow or too expensive.
- You are building a pipeline that should run on new and changed rows rather than everything.
- Downstream reporting needs recent data without someone remembering to refresh it.
- You want to see how change data capture works on a real table before committing to a heavier tool.
Before you start
- A reachable PostgreSQL or MySQL database — Those are the two engines the capture form offers. MySQL is audit-poll only.
- The table name, schema-qualified — The field expects the form schema.table — for example public.orders.
- A watermark column — A column that reliably increases when a row changes, such as updated_at. Audit-poll mode uses it to find what is new.
- Database credentials — Host, database, user and password. A read-only user is sufficient and safer.
Steps
Open Data → Connections → CDC Sync
Select Data, then Connections, then the CDC Sync tab.
A New capture (PostgreSQL) panel, and below it a list of any captures already defined.
Name the capture and choose the engine
Give the capture a Name — something like "Orders CDC" that says which table it follows.
Choose the Engine: PostgreSQL, or MySQL (audit-poll). Selecting MySQL forces the mode to audit-poll, because log-based capture is PostgreSQL only.
Point at the table and the database
Enter the table in Table (schema.table) form, for example public.orders.
Fill in Host, Database, User and Password for the source database.
Set the watermark column
Enter the column the capture should use as its Watermark column — commonly updated_at.
Audit-poll mode finds changes by asking for rows whose watermark is newer than the last run. A column that does not change when a row is updated will silently miss those updates, so choose it carefully.
Choose the capture modePro
Audit-poll (Free) queries the table on a schedule using the watermark column. It works on both engines and needs no special database configuration.
Log-based (Pro) reads the database change log instead, which catches deletes and does not depend on a watermark column. It is available on PostgreSQL only.
Set the schedule and create it
Set the Every field to how many minutes should pass between runs.
Select Create. The capture appears in the list below with its mode, schedule, current lag, schema version, watermark and the status of its last run.
The capture listed with its mode, interval, lag and last run status.
Run it once and check
Select Run on the capture to trigger it immediately rather than waiting for the schedule.
Check the last run status and the lag figure. A capture that ran successfully but returned nothing usually means the watermark column is not moving.
An updated last run status and a lag value on the capture row.
What happens next
The captured data becomes a dataset you can use anywhere in the platform. Point a pipeline at it in Orchestrate so downstream work runs on current data.
Watch the lag figure over time — a lag that grows steadily means the schedule is not keeping up with the change rate.
Example
An orders table gets a few thousand updates a day. A capture named "Orders CDC" points at public.orders with updated_at as the watermark, audit-poll mode, every 15 minutes. The dataset stays within a quarter of an hour of the source, and the pipeline reading it no longer reloads two million rows to find four thousand changes.
Tips
- Test the underlying database connection on the Database tab first. It rules out credentials and networking before you debug the capture itself.
- Pick the schedule from how fast the data actually changes, not from how fresh you would like it to be. Polling a slow table every minute costs queries and gains nothing.
- Watch the lag figure. It is the honest measure of whether the capture is keeping up.
- If a capture succeeds but finds nothing, check the watermark column really updates on every write. Some tables only set it on insert.
Limitations
- Two engines: PostgreSQL and MySQL. MySQL is audit-poll only.
- Log-based capture is a Pro capability and works on PostgreSQL only.
- Audit-poll detects inserts and updates through the watermark column. It cannot see hard deletes, because a deleted row has no watermark left to read.
- Audit-poll depends entirely on the watermark column being maintained. If the application does not update it, the capture will report success and miss changes.
Related how-to guides
Connect a database
Connect PostgreSQL, MySQL, SQL Server, Redshift, BigQuery or Snowflake, browse the tables and load what you need.
Build a data pipeline
Turn a sequence of steps into a repeatable pipeline with a defined execution order.
Trace where a column came from
Follow a field back from the dataset you are looking at to the source it came from, step by step.
Set data quality rulesPro
Turn a one-off quality check into a standing rule, so the same problem is caught every time.
Related questions
More of these on the DataLens FAQ page.
What is the difference between audit-poll and log-based CDC in DataLens?
Audit-poll queries the source table on a schedule and uses a watermark column such as updated_at to find rows that changed since the last run. It is on the free tier and works with PostgreSQL and MySQL. Log-based capture reads the database change log instead, catches deletes, and does not depend on a watermark column — it is a Pro capability and is available on PostgreSQL only.
Will CDC sync capture deleted rows?
Audit-poll mode cannot, because it identifies changes by reading a watermark column and a deleted row is no longer there to read. Log-based capture, which is Pro and PostgreSQL only, reads the change log and does see deletes.
Try this in DataLens
DataLens is in private beta. Request access and work through this guide on your own data.
Request beta access