Sync Dataset

Sourcesible Product Documentation · Activation

Overview

Sync Dataset allows you to push data from a Sourcesible Dataset directly to a BigQuery table on a defined schedule. It sends structured dataset records from your connected warehouse including BigQuery, Azure Synapse, Redshift, and PostgreSQL sources enabling you to keep a BigQuery table continuously updated with the latest data for downstream analysis or activation.

Before creating a Sync Dataset, ensure you have at least one Dataset connected and a BigQuery Destination connected.

Creating a Sync Dataset

Open the Sync Wizard

In the left navigation, click Sync. Click Sync in the top-right corner and select Sync Dataset from the dropdown.

Select Dataset

On the Select Dataset screen, select the Dataset you want to sync by clicking its radio button, then click Next.

Select Destination

On the Select Destination screen, select the BigQuery Destination you want to push data to, then click Next.

Set Up Sync

Enter a Sync Name (a unique name for this sync you can update it anytime under Settings) and an optional Sync Description. Under Select BigQuery Dataset, select the BigQuery dataset in your warehouse where the data will be written. Under Select BigQuery table, select the specific BigQuery table to use for this sync.

Under How do you want to update the records in BigQuery?, select your update mode:

Under Data Mapping, configure which columns from your Dataset should map to which fields in the BigQuery table. In the left dropdown, select a source column. The destination field will auto-populate or can be selected manually from the right dropdown. Click Add Mapping Field to add additional column mappings.

Click Next.

Set Schedule

Choose your Schedule Type: Manual (trigger each sync manually, no automatic runs) or Interval (run automatically at a set interval enter a number in the Every field and select the interval unit). Click Finish.

Monitoring Sync Runs

After creation, click the Sync name from the Sync list page to open the detail page. The Overview tab shows the Sync Runs log with Sync Date & Time, Status, Sync Type, and Triggered By for every execution. Click Run Sync at any time to trigger a manual run immediately, regardless of the schedule.

Next Steps

View Sync Run History Monitor past runs and diagnose failures from the Overview tab.

Edit Data Mapping Add or update column mappings from the Setup tab.

Change the Schedule Switch between Manual and Interval from the Schedule tab.

Delete a Sync Remove a Sync by clicking Delete Sync from the Sync detail page.

Tips and Troubleshooting

Select BigQuery Dataset Dropdown Takes a Long Time to Load

Symptom: After reaching Set Up Sync, the Select BigQuery Dataset section shows a loading spinner for an extended period.

Cause: Sourcesible is fetching the list of datasets and tables from your connected BigQuery project. This can take longer if the project has many datasets or if the BigQuery API is slow to respond.

Fix: Wait up to 30 seconds for the dropdown to populate. If it does not load, go back to Destinations, verify the BigQuery connection is still Connected, and retry.

Destination Field Dropdown Is Empty in Data Mapping

Symptom: After selecting the BigQuery table, the Destination field dropdowns in the Data Mapping section show no options.

Cause: The selected BigQuery table has no schema defined, or Sourcesible could not read the table schema due to a permission issue on the service account.

Fix: Verify in BigQuery that the selected table exists and has columns defined. Confirm the service account used for the BigQuery Destination has roles/bigquery.metadataViewer assigned.

PostgreSQL Dataset Not Appearing in the Select Dataset List

Symptom: A Dataset backed by a PostgreSQL source does not appear in the Select Dataset screen.

Cause: The PostgreSQL source connection has expired or been disconnected, causing its Datasets to show as inactive.

Fix: Navigate to Data Sources and verify the PostgreSQL source connection is still active. If the connection has failed, reopen the source settings, test the connection, and save. Return to Sync Dataset the Dataset should reappear once the source is restored.

Insert Mode Creating Duplicate Rows

Symptom: After multiple sync runs, the BigQuery table contains duplicate rows for the same records.

Cause: Insert mode pushes new records on every run without checking for existing rows. Records synced in a previous run will be inserted again.

Fix: Switch the update mode from Insert to Upsert in the Sync's Setup tab. Upsert will update existing rows instead of creating duplicates. This requires the BigQuery table to have a primary key or unique identifier column mapped correctly.

Interval Schedule Not Triggering as Expected

Symptom: The Sync Run log shows gaps longer than the configured interval.

Cause: Sourcesible processes interval syncs in a queue. If the previous run is still processing or the system is under load, the next run may be delayed beyond the configured interval.

Fix: Check the Sync Runs log to confirm whether runs are completing before the next scheduled time. If runs are taking longer than the interval, consider increasing the interval duration or switching to Manual schedule and triggering runs outside of peak hours.