ESHOPMAN Workflow Compensation & Data Integrity

The Importance of Robust Workflow Compensation in ESHOPMAN

In the world of headless commerce, especially with ESHOPMAN's powerful Node.js/TypeScript core and its integration with HubSpot for storefront management, reliable data operations are paramount. Workflows are designed to ensure complex operations complete successfully or, if they fail, revert to a consistent state. This 'compensation' mechanism is crucial for maintaining data integrity across your ESHOPMAN store, ensuring that product, customer, and order information displayed on your HubSpot CMS storefronts remains accurate.

Recently, our ESHOPMAN community identified a critical area where several core workflow steps, specifically those related to updating entities via the Admin API, might not be fully restoring previous states upon failure. This can lead to unexpected data inconsistencies, silent data loss, or even workflow crashes that leave partial changes in place.

Understanding the Core Issue: Incomplete Data Restoration

The problem lies in how certain update*Step compensations within ESHOPMAN's core workflows handle data snapshots and restoration. When an update step is invoked, it typically snapshots only the fields intended for update. However, the compensation logic might attempt to restore a fixed list of fields. If a field in this fixed list wasn't part of the original snapshot, it can be treated as undefined, leading to various issues:

Scenario 1: Unrelated Fields Nullified

When a field is nullable, an undefined value from the compensation can be persisted as null. For example:

Customer Updates: Rolling back an email-only update could inadvertently set a customer's first_name, last_name, phone, and metadata to null. This directly impacts customer data visible in HubSpot's CRM and your ESHOPMAN admin.
Customer Group Updates: A name-only update rollback could nullify the metadata for a customer group.

Scenario 2: Compensation Crashes, Changes Persist

If a field is required (non-nullable), attempting to set it to undefined (which becomes null) will cause the underlying ORM to throw an error. This halts the compensation, leaving the forward change in place and the workflow in a failed state. Examples include:

updateCustomerGroupsStep: Fails if name is null.
updatePaymentCollectionStep: Fails if amount is null.
updateSalesChannelsStep: Fails if name is null.

In these cases, the data shown in your ESHOPMAN Admin or HubSpot will reflect the *new*, un-reverted values, despite the workflow failing.

Scenario 3: Metadata Changes Not Reverted

When only a specific key within an entity's metadata is updated (e.g., update: { metadata: { k: "x" } }), the snapshot mechanism might only select "metadata.k". This often results in the snapshot holding no metadata at all. Consequently, the compensation cannot restore the original metadata, leaving the new metadata value in place even if the workflow is marked reverted.

updateRegionsStep: Metadata changes are not reverted.
This also affects other steps (like customer updates) if only a specific metadata key is updated.

Impact on ESHOPMAN Merchants and Developers

The consequences of these issues are significant:

Data Inconsistencies: Your ESHOPMAN data, and by extension, the data powering your HubSpot CMS storefront, can become unreliable.
Silent Data Loss: Critical customer or product information might be silently wiped (set to null) without clear indication to the user.
Poor User Experience: Admin users interacting with the ESHOPMAN Admin API (or through HubSpot's storefront management features) might encounter generic 500 errors, with the underlying data issues hidden.

Proposed Solution: Enhancing Compensation Logic

The community has suggested a robust fix focusing on two key adjustments to the workflow compensation logic:

Snapshot Metadata as a Whole: Ensure that when metadata is involved in an update, the entire metadata object is snapshotted, not just specific keys. This can be achieved by adjusting the helper function that determines what fields to select for the snapshot:
```
const { selects, relations } = getSelectsAndRelationsFromObjectArray([data.update], { objectFields: ["metadata"] })
```
Restore Only Present Fields: In the compensation step, instead of restoring a fixed list of fields, dynamically restore only those fields that were actually present in the snapshot. This prevents undefined values from being passed to the persistence layer. Similar to how other robust ESHOPMAN steps already operate, a utility function like removeUndefined(...) can be used to filter out non-existent fields.

Implementing these changes ensures that when an ESHOPMAN workflow fails, the compensation truly restores the entity to its exact prior state, safeguarding data integrity and providing a more reliable experience for both developers and merchants managing their stores through HubSpot.

The ESHOPMAN team at Move My Store is committed to fostering a robust and reliable platform, and insights like these from our developer community are invaluable for continuous improvement. We encourage all developers to review their custom workflows and consider these best practices for maximum data integrity.

Ensuring Data Integrity: Understanding Workflow Rollbacks in ESHOPMAN Admin API