Pigment CSV Push API

Overview

VersionForge delivers data to Pigment through the CSV Push API (POST /api/v1/import/push/csv). Source records are formatted as an RFC 4180-compliant CSV file, uploaded as a multipart form-data attachment, and processed by Pigment's import engine against a pre-configured import configuration. This guide covers the upload mechanics, response handling, and partial-failure retry logic.

Upload Mechanics

CSV Formatting

VersionForge converts source records to CSV with these rules:

• Header row: Column names derived from the sorted field keys of the source records. Sorting ensures deterministic column order across runs.
• Data rows: Field values in the same column order as the header.
• Escaping: Fields containing commas, double quotes, or newlines are wrapped in double quotes. Embedded double quotes are doubled ("") per RFC 4180.
• Null values: Null and undefined fields are serialized as empty strings.

Example output:

account,amount,department,location,period
4100,120000,Engineering,San Francisco,2026-01
4200,95000,Sales,New York,2026-01
4300,,Marketing,Chicago,2026-01

Multipart Upload

The CSV is uploaded as a file attachment in a multipart/form-data request:

POST https://api.pigment.com/api/v1/import/push/csv?configurationId=cfg_abc123
Authorization: Bearer {apiKey}
Content-Type: multipart/form-data; boundary=...

--boundary
Content-Disposition: form-data; name="file"; filename="data.csv"
Content-Type: text/csv

account,amount,department,location,period
4100,120000,Engineering,San Francisco,2026-01
...
--boundary--

Query Parameters

| Parameter | Required | Description | |-----------|----------|-------------| | configurationId | Yes | ID of the Pigment import configuration to use. | | dryRun | No | Set to true to validate without committing. | | importMode | No | Override: create, update, or upsert. |

Import Config Validation

Before the CSV reaches Pigment's data engine, the import configuration validates:

1. Column matching -- every column in the CSV header must map to a column defined in the import configuration. Extra columns are ignored; missing required columns cause rejection.
2. Type coercion -- values are coerced to the column's defined type (string, number, date, boolean). Invalid coercions produce row-level errors.
3. Dimension matching -- values in dimension columns must match existing members in the corresponding Pigment entity, unless autoCreate is enabled on that column.

Response Handling

A successful push returns a JSON response with processing details:

{
  "status": "completed",
  "requestId": "req_abc123",
  "importId": "imp_xyz789",
  "recordsProcessed": 500,
  "recordsAccepted": 497,
  "recordsRejected": 3,
  "errors": [
    {
      "row": 42,
      "field": "department",
      "value": "Engineering - Platform",
      "code": "UNKNOWN_DIMENSION_MEMBER",
      "message": "Value 'Engineering - Platform' not found in entity 'department'"
    }
  ],
  "warnings": [],
  "metadata": {
    "dataSource": "csv_push",
    "targetTable": "headcount_plan",
    "startedAt": "2026-04-12T10:30:00Z",
    "completedAt": "2026-04-12T10:30:02Z",
    "durationMs": 2340
  }
}

VersionForge maps this response into its internal PushResult structure:

• insertedCount = recordsAccepted
• skippedCount = recordsRejected
• Per-row errors are logged and associated with their source records

Partial-Failure Retry

When some rows are rejected but others succeed, VersionForge executes a single retry for the rejected rows:

1. Extract rejected row indices from the errors array (1-based, where row 1 is the first data row)
2. Map row indices back to source records
3. Re-batch the rejected records into a new CSV
4. Push the retry batch to the same import configuration

Dimension Member Matching

Dimension matching is one of the most common sources of push failures. VersionForge provides two strategies to handle mismatches:

Auto-create (Pigment-side): If the import configuration column has autoCreate: true, Pigment automatically creates new dimension members on push. This is the simplest approach but offers no pre-push validation.

Pre-push validation (VersionForge-side): VersionForge calls the Pigment metadata API to fetch current dimension members before pushing. Mismatches are surfaced in the Safety Gate review queue, giving you a chance to fix mappings or approve new members before they are created.

Rate Limiting

Pigment enforces a limit of 500 requests per 5-minute window. VersionForge handles 429 responses with exponential backoff and full jitter:

• Attempt 1: wait 0-1 second
• Attempt 2: wait 0-2 seconds
• Attempt 3: wait 0-4 seconds
• Attempt 4: wait 0-8 seconds
• Attempt 5: wait 0-16 seconds

After 5 failed attempts, the push is aborted and the error is surfaced in the pipeline dashboard.