Schema Drift Detection

What Schema Drift Is

Schema drift occurs when the structure of data coming from a source system changes between sync runs. A field gets renamed. A new column appears. A field type changes from string to integer. A required field is removed entirely.

Without detection, these changes cause silent failures -- rows load with missing data, transforms apply to the wrong columns, and planning models quietly absorb garbage. Schema drift is one of the most common root causes of "the numbers don't match" investigations.

How VersionForge Detects Drift

Every time a sync profile runs successfully, VersionForge stores the extraction schema -- the full list of field names, types, nullability, and ordering from the source response. On the next run, the current extraction schema is compared field-by-field against the stored schema.

VersionForge detects four categories of drift:

Field Renamed

A field present in the stored schema is absent, and a new field with similar data appears. VersionForge flags this as a probable rename and includes a similarity score.

Field Added

A new field appears in the extraction that was not present in the stored schema. This is low-severity -- it does not break existing mappings -- but it may indicate a source system update that your mappings should account for.

Field Removed

A field present in the stored schema is missing from the current extraction. This is high-severity because any mapping that references the removed field will produce null values.

Type Changed

A field exists in both schemas but its data type has changed (e.g., string to number, date to string). This can cause transform failures or silent type coercion in the target.

What Happens When Drift Is Detected

1. Sync pauses before load

   The pipeline halts after extraction and before any transform or load step executes. No data reaches the target system while the drift is unresolved.

2. Drift report is generated

   VersionForge produces a drift report listing every detected change with its severity:

   DRIFT DETECTED on profile "NetSuite GL → Adaptive"
   Run: run_20260412_0830
   
     REMOVED  subsidiary_code    (was: string, nullable)
     RENAMED  dept_id → department_id  (similarity: 0.94)
     ADDED    cost_center_v2     (string, nullable)
     TYPE     amount: string → number
   

3. Implementer reviews and updates mappings

   The drift report links directly to the sync profile's field mapping editor. You update mappings to reflect the new schema -- point the subsidiary_code mapping to the new field name, acknowledge the type change, and decide whether to map the new cost_center_v2 field.

4. Sync resumes with updated schema

   After you save the updated mappings, the paused run resumes from the extraction step using the new schema as baseline. The updated schema is stored for future drift comparisons.

Preventing Silent Data Loss

Schema drift detection is enabled by default on all connectors and cannot be fully disabled. You can configure the sensitivity:

extraction:
  schemaDrift:
    blockOnRemoved: true     # always pause on removed fields
    blockOnTypeChange: true  # always pause on type changes
    blockOnRenamed: true     # pause on probable renames
    warnOnAdded: true        # log new fields but do not pause

For connectors with volatile schemas (e.g., Salesforce custom objects that change frequently), you can set blockOnAdded: false to reduce noise while still catching the high-severity changes that affect existing mappings.

Schema Versioning

VersionForge maintains a history of every schema version for each sync profile. You can view the schema at any point in time, compare two versions side by side, and trace when a specific field change was first detected. This history is critical for debugging data quality issues that span multiple sync cycles.