Salesforce Bulk API & REST Auto-Switching

API Selection Logic

VersionForge automatically chooses the optimal Salesforce API for each extraction based on expected data volume:

| Scenario | API Used | Why | |----------|----------|-----| | Full extract (no since filter) | Bulk API 2.0 | Full loads can return millions of rows; Bulk API handles this efficiently via async CSV download. | | Incremental with < 10,000 estimated rows | REST API | Lower latency, synchronous, no job lifecycle overhead. | | Incremental with >= 10,000 estimated rows | Bulk API 2.0 | Throughput matters more than latency at this scale. |

The threshold (default: 10,000 rows) can be overridden per sync profile:

{
  "bulkThreshold": 5000
}

How the Estimate Works

Before an incremental extraction, VersionForge issues a lightweight SELECT COUNT() SOQL query filtered to the same since timestamp. This returns in milliseconds and doesn't count toward the Bulk API daily job limit. Based on the result, the connector routes to REST or Bulk.

If the count query fails for any reason, the connector defaults to REST -- the safer choice for small incremental pulls.

Bulk API 2.0 Lifecycle

When the Bulk API is selected, VersionForge manages the full job lifecycle:

1. Create -- POST a query job with the SOQL statement
2. Poll -- Check job status every 2 seconds until JobComplete, Failed, or Aborted
3. Download -- Fetch CSV results using paginated Sforce-Locator headers
4. Parse -- Convert CSV rows to structured records with type coercion

The maximum poll duration is 10 minutes. Jobs that exceed this are reported as extraction errors.

CSV Parsing

Bulk API 2.0 returns results as CSV. VersionForge's built-in parser handles:

• Quoted fields with embedded commas and newlines
• Escaped double-quotes ("")
• Type coercion: true/false to booleans, numeric strings to numbers (except IDs with leading zeros like 001...), empty strings and #N/A to null

REST API Pagination

When using the REST API, VersionForge follows the nextRecordsUrl cursor automatically until all results are collected or the safety cap (default: 50,000 records) is reached. Each page is a separate API call counted toward your org's daily limit.

API Quota Management

VersionForge checks your org's API limits during the prepare phase of each sync run. Warnings are logged when:

• Remaining daily API requests drop below 1,000
• The extraction itself is expected to consume a significant portion of remaining quota

Performance Tips

• Set a higher bulk threshold for orgs with strict REST API quotas -- pushing more traffic to Bulk API preserves your REST budget for other integrations
• Use incremental extraction whenever possible. Full extracts on large objects (Opportunity, Account) can generate hundreds of thousands of rows
• Schedule off-peak if your org has heavy user traffic during business hours, as Bulk API jobs share server resources