Skip to main content
  • BigPanda Documentation
  • Integrations
  • BigPanda Unified Data Connector

BigPanda Unified Data Connector

The BigPanda Unified Data Connector syncs data from your ITSM platforms to provide context and insight for the AI Incident Assistant, AI Incident Prevention, and AI Detection and Response. Ingested data is securely stored and made available in the IT Knowledge Graph, powering accurate answers, deep analytics, trend analysis, and advanced capabilities.

The Unified Data Connector has a standard connector for many tools, or a custom connection can be created for the tool of your choice.

Data Redaction

You can redact sensitive information ingested via the Unified Data Connector, including PII, PHI, and PCI, giving you confidence that your confidential data remains exclusively within your approved channels. See the Data Redaction documentation for details on managing sensitive data.

Streaming Modes

You can set up both historical and incremental streaming modes for each data connector. Most organizations will need to use both modes.

Historical mode

Historical modes sync data one time from a selected start date and after. This works well for one-time data loads or when you need to backfill data that was previously missed. Historical mode is the default, recommended method of initially syncing data. 

Historical mode

Historical streams can take minutes or days, depending on the amount of data sent.

Re-pulling historical data

If you need to re-ingest a specific historical window, such as after a ServiceNow data correction, a major migration, or a gap caused by an upstream outage, your BigPanda account team can trigger a targeted date-range backfill against an existing pipeline without rebuilding the connector.

Backfills run in chunks (typically per day) and execute in parallel with your incremental sync without disrupting it. Large ranges can be scheduled to run only during off-peak windows in your time zone.

To request a date-range backfill, reach out to your account team with:

  • The connector and pipeline name

  • The exact start and end date of the window to re-pull

  • Any preferred execution window (for example, midnight–6 AM EST)

While the backfill is running, you can view per-table row counts and chunk completion status in the BigPanda support response or in the BigPanda monitoring views shared by your account team

Incremental mode

Incremental modes run periodically and sync all new or updated records. This mode works well for scheduled, recurring data loads. Depending on what functionality you are using within BigPanda, your incremental sync time period may vary.

No historical data

Incremental mode runs periodically and syncs only new or updated records since the previous successful run. Once an incremental pipeline is started, its cursor moves forward from that point and does not look back to the configured Start Date. To ingest historical records, run a one-time historical sync (or request a date-range backfill) on the same source. Most production deployments use both: one historical sync to seed the dataset, then incremental mode for ongoing updates.

Frequently Asked Questions

General

How does the Unified Data Connector protect sensitive data and PII?

You can create filters for your tables to ensure that sensitive data is not included in the sync. Additionally, the Data Redaction feature can automatically scan and remove PII, including an advanced optional redaction stream for context-aware removal of PII and PHI.

How long does the initial sync take?

It depends on the rate limit in place, the amount of data synced, and the sync start date. The initial sync can take anywhere between a few hours to multiple weeks.

How often does the data sync?

The default sync frequency is set to 2 minutes, but it can be modified. You can set the frequency from 1 minute to 24 hours.

ServiceNow

How do you ensure that APIs don’t bypass manual review or approval processes when writing back into ServiceNow and that audit trails are preserved?

The Unified Data Connector does not write anything to ServiceNow. It only uses GET APIs to pull data. Just like other user activity, you can track the user activity of the Unified Data Connector user in the ServiceNow system logs.

What controls are in place to ensure that ServiceNow performance is not affected by large data pulls or frequent polling?

The Unified Data connector has the following features in place to control and limit the number of calls and load on the server:

  • Batch size control: Larger batch sizes result in fewer overall calls to the server.

  • Max calls per minute: This setting controls the maximum number of calls sent to ServiceNow. After it reaches the maximum, it pauses for the configured amount of time.

  • Incremental loads: Every incremental load starts from the state that the last load had completed.

Will customizations break if ServiceNow upgrades or changes its APIs?

Only a small number of unique calls are used for the sync. Calls made to ServiceNow are core REST APIs that don’t change often, such as now/table/{table_name}. If ServiceNow makes a change, the fixes to the Unified Data Connector will be minor and happen quickly.

Can UDC support dot walking reference tables (for example: assigned_to.email, assigned_to.name)?

Yes. The ServiceNow connector treats field names as a string, even if a dot is included in the string.

Troubleshoot connection issues

401 Unauthorized when testing or running the connector

  1. Confirm the service account is still active and has not been rotated, locked, or had its password reset since the connector was configured.

  2. Re-test using the credentials directly against your source's REST API (for example,GET /api/now/table/incident?sysparm_limit=1for ServiceNow). If that call also returns 401, the issue is on the source side, not the connector.

  3. Special characters in secrets — particularly backticks (`), ampersands (&), and other shell-meaningful characters — can be URL-encoded inconsistently by some clients. If your password or client secret contains these, regenerate it without them and re-submit.

  4. For OAuth, double-check bothclient_idandclient_secret, and confirm the OAuth token endpoint URL matches your instance.

  5. If you connect through a reverse proxy with a static-IP requirement, confirm the proxy IP has not changed since the connector was set up.

Connector authenticates but returns zero records

A 0-record result usually means authentication succeeded but the service account lacks read access to the target table. Verify with:

  • ServiceNow:GET /api/now/table/<table>?sysparm_limit=1executed as the BigPanda service account. If this returns rows in ServiceNow Studio but zero through the API, the account is missing table-level read ACLs.

  • Jira / JSM / Confluence:confirm the service account has read permission on every project, board, or space referenced by the pipeline configuration (JQL,space_keys, etc.).

Grant table-, project-, or space-level read access to the BigPanda service account and re-test the connection.

Drop or gap in record volume

If a Confluence sync returns far fewer records than expected, verify that every space you intend to ingest is listed in Space keys and that the Confluence service account has read access to each of those spaces. The connector only ingests spaces that are both listed in the configuration and accessible to the service account.

For other connector types, open a support ticket including the affected connector, the date and time the change was first observed, and any source-side changes deployed in the same window (firewall, IP allow-list, ACL, password rotation, ServiceNow upgrade). Most volume regressions trace back to a source-side change of this kind.

Intermittent 502 Bad Gateway responses on specific dates

If a historical sync over a specific date range consistently returns 502 responses, the most common cause is an individual record exceeding the API gateway's response-size limit (typically because a free-text field such as work_notes contains a very large attachment or pasted log). Two mitigations:

  • Reduce the per-call batch size by lowering sysparm_limit (the Batch size setting). Smaller batches are less likely to combine into an oversized response payload.

  • At the ServiceNow side, restrict the BigPanda service account's read ACL on the offending free-text column, or omit that column from the connector's Fields list.

If 502s persist after both mitigations, contact BigPanda support with a sample affected date so we can investigate at the connector level.

Unified Data Connector Configuration

The unified data connector setup process is multi-step and involves close coordination with BigPanda support. Contact BigPanda support when you are ready to set up a data connector.

Streaming Modes

You can set up both historical and incremental streaming modes for each data connector. Most organizations will need to use both modes.

Historical mode

Historical modes sync data one time from a selected start date and after. This works well for one-time data loads or when you need to backfill data that was previously missed. Historical mode is the default, recommended method of initially syncing data. 

Historical mode

Historical streams can take minutes or days, depending on the amount of data sent.

Re-pulling historical data

If you need to re-ingest a specific historical window, such as after a ServiceNow data correction, a major migration, or a gap caused by an upstream outage, your BigPanda account team can trigger a targeted date-range backfill against an existing pipeline without rebuilding the connector.

Backfills run in chunks (typically per day) and execute in parallel with your incremental sync without disrupting it. Large ranges can be scheduled to run only during off-peak windows in your time zone.

To request a date-range backfill, reach out to your account team with:

  • The connector and pipeline name

  • The exact start and end date of the window to re-pull

  • Any preferred execution window (for example, midnight–6 AM EST)

While the backfill is running, you can view per-table row counts and chunk completion status in the BigPanda support response or in the BigPanda monitoring views shared by your account team

Incremental mode

Incremental modes run periodically and sync all new or updated records. This mode works well for scheduled, recurring data loads. Depending on what functionality you are using within BigPanda, your incremental sync time period may vary.

No historical data

Incremental mode runs periodically and syncs only new or updated records since the previous successful run. Once an incremental pipeline is started, its cursor moves forward from that point and does not look back to the configured Start Date. To ingest historical records, run a one-time historical sync (or request a date-range backfill) on the same source. Most production deployments use both: one historical sync to seed the dataset, then incremental mode for ongoing updates.

Salesforce Data Connector

The BigPanda Unified Data Connector (UDC) brings records from Salesforce objects to BigPanda. Use the Salesforce Data Connector to bring CRM and platform application data into BigPanda analytics workflows.

The connector uses the Salesforce REST API to pull records from Salesforce objects (for example, Case, Account, Contact, or managed-package custom objects). Authentication uses the OAuth 2.0 Client Credentials flow, which runs as a designated Salesforce user; no interactive login, password, or certificate required.

Run as a dedicated user

The Client Credentials flow runs as a single Salesforce user, and the connector pulls only the records that user can access. For predictable results, run the integration as a dedicated, Active user with API Enabled rather than an individual's personal account.

Prerequisites

Before you configure the Salesforce Data Connector, make sure you have the following:

  • Salesforce System Administrator access.

  • My Domain enabled for the org. To confirm, go to Setup > Quick Find > My Domain and note the current My Domain URL, for example https://<my_domain>.my.salesforce.com.

  • A Salesforce user to run the integration as. The user must be Active and have API Enabled. A System Administrator qualifies.

Configure Salesforce

Complete these steps in Salesforce before setting up the data connector with BigPanda.

1. Create an External Client App

  1. In the top-right corner of the screen, click the gear icon, then select Setup.

  2. In Quick Find, enter App Manager and open it.

  3. In the top-right corner, click New External Client App.

  4. Under Basic Information, complete the following fields:

    •  External Client App Name: A name for the app, for example BigPanda UDC Connector.

    •  API Name: Fills in automatically.

    •  Contact Email: Your email address.

    •  Distribution State: Select Local.

  5. Click Create.

New External Client App replaces Connected Apps

New Salesforce orgs hide the legacy New Connected App button by default. External Client App is the current method.

2. Enable OAuth and add the API scope

  1. Open the app, then go to the Settings tab.

  2. Under OAuth Settings, click Edit.

  3. Select Enable OAuth.

  4. In the Callback URL field, enter https://login.salesforce.com/services/oauth2/callback. This value is a placeholder, and the Client Credentials flow never uses it.

  5. Under OAuth Scopes, move the following scopes to Selected:

    •  Manage user data via APIs (api): Required.

    •  Perform requests at any time (refresh_token, offline_access): Recommended.

  6. Click Save.

API scope is required

Without the api scope, the connection test returns the error no valid scopes defined.

3. Enable the Client Credentials flow

  1. In the Settings tab, under OAuth Settings, find Flow Enablement.

  2. Select Enable Client Credentials Flow.

  3. Click Save.

4. Assign the Run-As user

  1. Open the app, then go to the Policies tab and click Edit.

  2. In the Client Credentials Flow section, find the Run As field.

  3. Enter the user's name, then select the user from the dropdown lookup. You must pick the user from the lookup rather than typing the name as free text.

  4. Click Save, then allow 2 to 5 minutes for the change to propagate.

Select the user from the lookup

If you type the name without selecting it from the dropdown, Salesforce rejects the save with Enter a valid execution user. If no Run-As user is set, the connection test returns no client credentials user enabled.

5. Retrieve the Consumer Key and Secret

  1. Open the app, then go to the Settings tab.

  2. ”Under OAuth Settings, open Consumer Key and Secret. This option may also appear as Manage Consumer Details.

  3. Complete identity verification. Salesforce emails a code to your address.

  4. Copy the Consumer Key and the Consumer Secret. These are the credentials you enter into BigPanda.

6. Locate your My Domain URL

  1. Go to Setup > Quick Find > My Domain.

  2. Copy the current My Domain URL, for example https://<my_domain>.my.salesforce.com. This is the instance URL that BigPanda uses. BigPanda derives the token endpoint from it automatically by appending /services/oauth2/token.

Use the My Domain URL, not login.salesforce.com

The Client Credentials flow must use the My Domain URL. Salesforce rejects client-credentials requests sent to the generic https://login.salesforce.com endpoint.

Provide authentication information to BigPanda

After completing the steps in Salesforce, contact your BigPanda account team to set up the data connector. Provide the following credentials:

Field

Description

Instance URL

Domain URL retrieved in step 6.

Client ID

Consumer key retrieved in step 5.

Client Secret

Secret retrieved in step 5.

Resources

Configure each Salesforce object to sync. For each object, you can choose specific fields to fetch, or omit the field list to load the Salesforce standard fields for that object.

Option

Description

Object names

The Salesforce objects to sync. Provide one entry per object. Use the exact API name. Custom objects end in __c. Invalid names fail validation before the sync begins.

Fields

Optional. The list of fields to fetch for each object. Field names must use supported Salesforce identifier formats. If you leave this blank, the connector fetches the Salesforce standard fields for the object. 

Sync preferences

Provide the following information about your sync preferences to BigPanda:

Required Configuration

Option

Description

Mode

Determine whether you'd like to set up an incremental or historical sync. Most organizations will need to use both modes. See the Streaming Modes section for more information.

Cursor Field

The Salesforce field the connector uses to track sync progress. For most objects, set this to LastModifiedDate.

Optional pipeline configuration

Option

Description

Query Limit

Maximum number of queries. Query limit cannot be set to 0. We recommend 50 as the default.

Start Date

Used for historical syncs only. Provide the date when you'd like the sync to begin. If no start date is provided, all data will be synced. We recommend syncing one year of data.

End Date

Used for historical syncs only. Provide an end date to backfill historical syncs.

Rate Limit

Requests per interval. (Default 5)

Rate Limit Timeout MS

Wait time in ms when rate-limited. (Default 1000)

Request Timeout

HTTP timeout, in seconds. (Default 60)

Output schema

The connector creates one record per row returned from each Salesforce object. When you specify fields for an object, the connector returns only those fields. When you leave fields blank, the connector returns the Salesforce standard fields for the object.

Troubleshooting

Error

Cause

Fix

no client credentials user enabled

No Run-As user is set.

Set the Run-As user in the Policies tab. See Assign the Run-As user.

Enter a valid execution user (on save)

You typed the Run-As name but did not select it from the lookup.

Re-select the user from the dropdown. See Assign the Run-As user.

no valid scopes defined

The app has no OAuth scopes.

Add the api scope. See Enable OAuth and add the API scope.

invalid_client

The Consumer Key or Consumer Secret is wrong.

Re-copy both values and provide them again. See Retrieve the Consumer Key and Secret.

In this section: