Skip to main content

Unified Data Connector Configuration

The unified data connector setup process is multi-step and involves close coordination with BigPanda support. Contact BigPanda support when you are ready to set up a data connector.

Streaming Modes

You can set up both historical and incremental streaming modes for each data connector. Most organizations will need to use both modes.

Historical mode

Historical modes sync data one time from a selected start date and after. This works well for one-time data loads or when you need to backfill data that was previously missed. Historical mode is the default, recommended method of initially syncing data. 

Historical mode

Historical streams can take minutes or days, depending on the amount of data sent.

Re-pulling historical data

If you need to re-ingest a specific historical window, such as after a ServiceNow data correction, a major migration, or a gap caused by an upstream outage, your BigPanda account team can trigger a targeted date-range backfill against an existing pipeline without rebuilding the connector.

Backfills run in chunks (typically per day) and execute in parallel with your incremental sync without disrupting it. Large ranges can be scheduled to run only during off-peak windows in your time zone.

To request a date-range backfill, reach out to your account team with:

  • The connector and pipeline name

  • The exact start and end date of the window to re-pull

  • Any preferred execution window (for example, midnight–6 AM EST)

While the backfill is running, you can view per-table row counts and chunk completion status in the BigPanda support response or in the BigPanda monitoring views shared by your account team

Incremental mode

Incremental modes run periodically and sync all new or updated records. This mode works well for scheduled, recurring data loads. Depending on what functionality you are using within BigPanda, your incremental sync time period may vary.

No historical data

Incremental mode runs periodically and syncs only new or updated records since the previous successful run. Once an incremental pipeline is started, its cursor moves forward from that point and does not look back to the configured Start Date. To ingest historical records, run a one-time historical sync (or request a date-range backfill) on the same source. Most production deployments use both: one historical sync to seed the dataset, then incremental mode for ongoing updates.

Salesforce Data Connector

The BigPanda Unified Data Connector (UDC) brings records from Salesforce objects to BigPanda. Use the Salesforce Data Connector to bring CRM and platform application data into BigPanda analytics workflows.

The connector uses the Salesforce REST API to pull records from Salesforce objects (for example, Case, Account, Contact, or managed-package custom objects). Authentication uses the OAuth 2.0 Client Credentials flow, which runs as a designated Salesforce user; no interactive login, password, or certificate required.

Run as a dedicated user

The Client Credentials flow runs as a single Salesforce user, and the connector pulls only the records that user can access. For predictable results, run the integration as a dedicated, Active user with API Enabled rather than an individual's personal account.

Prerequisites

Before you configure the Salesforce Data Connector, make sure you have the following:

  • Salesforce System Administrator access.

  • My Domain enabled for the org. To confirm, go to Setup > Quick Find > My Domain and note the current My Domain URL, for example https://<my_domain>.my.salesforce.com.

  • A Salesforce user to run the integration as. The user must be Active and have API Enabled. A System Administrator qualifies.

Configure Salesforce

Complete these steps in Salesforce before setting up the data connector with BigPanda.

1. Create an External Client App

  1. In the top-right corner of the screen, click the gear icon, then select Setup.

  2. In Quick Find, enter App Manager and open it.

  3. In the top-right corner, click New External Client App.

  4. Under Basic Information, complete the following fields:

    •  External Client App Name: A name for the app, for example BigPanda UDC Connector.

    •  API Name: Fills in automatically.

    •  Contact Email: Your email address.

    •  Distribution State: Select Local.

  5. Click Create.

New External Client App replaces Connected Apps

New Salesforce orgs hide the legacy New Connected App button by default. External Client App is the current method.

2. Enable OAuth and add the API scope

  1. Open the app, then go to the Settings tab.

  2. Under OAuth Settings, click Edit.

  3. Select Enable OAuth.

  4. In the Callback URL field, enter https://login.salesforce.com/services/oauth2/callback. This value is a placeholder, and the Client Credentials flow never uses it.

  5. Under OAuth Scopes, move the following scopes to Selected:

    •  Manage user data via APIs (api): Required.

    •  Perform requests at any time (refresh_token, offline_access): Recommended.

  6. Click Save.

API scope is required

Without the api scope, the connection test returns the error no valid scopes defined.

3. Enable the Client Credentials flow

  1. In the Settings tab, under OAuth Settings, find Flow Enablement.

  2. Select Enable Client Credentials Flow.

  3. Click Save.

4. Assign the Run-As user

  1. Open the app, then go to the Policies tab and click Edit.

  2. In the Client Credentials Flow section, find the Run As field.

  3. Enter the user's name, then select the user from the dropdown lookup. You must pick the user from the lookup rather than typing the name as free text.

  4. Click Save, then allow 2 to 5 minutes for the change to propagate.

Select the user from the lookup

If you type the name without selecting it from the dropdown, Salesforce rejects the save with Enter a valid execution user. If no Run-As user is set, the connection test returns no client credentials user enabled.

5. Retrieve the Consumer Key and Secret

  1. Open the app, then go to the Settings tab.

  2. ”Under OAuth Settings, open Consumer Key and Secret. This option may also appear as Manage Consumer Details.

  3. Complete identity verification. Salesforce emails a code to your address.

  4. Copy the Consumer Key and the Consumer Secret. These are the credentials you enter into BigPanda.

6. Locate your My Domain URL

  1. Go to Setup > Quick Find > My Domain.

  2. Copy the current My Domain URL, for example https://<my_domain>.my.salesforce.com. This is the instance URL that BigPanda uses. BigPanda derives the token endpoint from it automatically by appending /services/oauth2/token.

Use the My Domain URL, not login.salesforce.com

The Client Credentials flow must use the My Domain URL. Salesforce rejects client-credentials requests sent to the generic https://login.salesforce.com endpoint.

Provide authentication information to BigPanda

After completing the steps in Salesforce, contact your BigPanda account team to set up the data connector. Provide the following credentials:

Field

Description

Instance URL

Domain URL retrieved in step 6.

Client ID

Consumer key retrieved in step 5.

Client Secret

Secret retrieved in step 5.

Resources

Configure each Salesforce object to sync. For each object, you can choose specific fields to fetch, or omit the field list to load the Salesforce standard fields for that object.

Option

Description

Object names

The Salesforce objects to sync. Provide one entry per object. Use the exact API name. Custom objects end in __c. Invalid names fail validation before the sync begins.

Fields

Optional. The list of fields to fetch for each object. Field names must use supported Salesforce identifier formats. If you leave this blank, the connector fetches the Salesforce standard fields for the object. 

Sync preferences

Provide the following information about your sync preferences to BigPanda:

Required Configuration

Option

Description

Mode

Determine whether you'd like to set up an incremental or historical sync. Most organizations will need to use both modes. See the Streaming Modes section for more information.

Cursor Field

The Salesforce field the connector uses to track sync progress. For most objects, set this to LastModifiedDate.

Optional pipeline configuration

Option

Description

Query Limit

Maximum number of queries. Query limit cannot be set to 0. We recommend 50 as the default.

Start Date

Used for historical syncs only. Provide the date when you'd like the sync to begin. If no start date is provided, all data will be synced. We recommend syncing one year of data.

End Date

Used for historical syncs only. Provide an end date to backfill historical syncs.

Rate Limit

Requests per interval. (Default 5)

Rate Limit Timeout MS

Wait time in ms when rate-limited. (Default 1000)

Request Timeout

HTTP timeout, in seconds. (Default 60)

Output schema

The connector creates one record per row returned from each Salesforce object. When you specify fields for an object, the connector returns only those fields. When you leave fields blank, the connector returns the Salesforce standard fields for the object.

Troubleshooting

Error

Cause

Fix

no client credentials user enabled

No Run-As user is set.

Set the Run-As user in the Policies tab. See Assign the Run-As user.

Enter a valid execution user (on save)

You typed the Run-As name but did not select it from the lookup.

Re-select the user from the dropdown. See Assign the Run-As user.

no valid scopes defined

The app has no OAuth scopes.

Add the api scope. See Enable OAuth and add the API scope.

invalid_client

The Consumer Key or Consumer Secret is wrong.

Re-copy both values and provide them again. See Retrieve the Consumer Key and Secret.

In this section: