Ensuring ESHOPMAN Workflow Integrity: Navigating Cancellation Challenges

Understanding ESHOPMAN Workflow Cancellation and Data Integrity

At Move My Store, we understand that robust workflow management is the backbone of any successful headless commerce operation built on ESHOPMAN. ESHOPMAN, leveraging Node.js/TypeScript and integrating seamlessly with HubSpot for storefront management and CMS deployment, relies on sophisticated workflows to handle complex processes like order fulfillment, inventory updates, and customer interactions. A critical aspect of these workflows is their ability to revert cleanly when something goes wrong, ensuring data integrity across your HubSpot-managed storefront.

Recently, our ESHOPMAN community has identified a significant challenge related to workflow cancellation, specifically when a workflow has completed its primary tasks but encounters a failure during post-completion callbacks (e.g., event release). In such scenarios, the system attempts to cancel the already completed workflow, reloading its state from the database. This re-cancellation path has exposed two key issues that prevent a clean reversion to a 'REVERTED' state, instead leaving the workflow 'FAILED' and potentially leaving behind inconsistent data.

Issue 1: JSONB Key Order Impacts Link Dismissal

One of the primary hurdles arises from how ESHOPMAN's underlying PostgreSQL database handles JSONB data. When a workflow's context is stored and reloaded from a JSONB column, the order of keys within JSON objects is not guaranteed to be preserved. This becomes problematic for functions like Link.dismiss, which are crucial for cleaning up related data (e.g., links between orders and carts) during compensation.

Specifically, the Link.dismiss function, when attempting to identify and delete created links, relies on the order of keys within the link definition to derive module directions (e.g., 'A-keyA-B-keyB'). If the key order is changed upon reloading from JSONB (e.g., {order, cart} becomes {cart, order}), the system fails to find the correct link module, throwing an error like no link module for cart-order. Consequently, these links are not deleted, leaving orphaned data in the database.

// Relevant excerpt from package.json indicating core dependencies
{
  "dependencies": {
    "@medusajs/core-flows": "2.21.2",
    "@medusajs/framework": "2.21.2",
    "@medusajs/medusa": "2.21.2",
    "@medusajs/workflow-engine-redis": "2.21.2",
    "@mercurjs/core": "2.3.7"
  }
}

Issue 2: Sub-workflows Without Store Option Cannot Be Compensated

The second issue affects sub-workflows executed via runAsStep. If a sub-workflow is defined without the store option, it does not checkpoint its state during execution. When the parent workflow is reloaded for cancellation, and it attempts to compensate a runAsStep sub-workflow, the system tries to load the sub-workflow's transaction by its ID. Since no state was stored for it, this load operation fails with a NOT_FOUND error, causing the compensation step to fail.

Impact and Reproduction

The combined effect of these issues is that a critical ESHOPMAN workflow, instead of reverting cleanly to a REVERTED state, ends up in a FAILED state. This leaves behind inconsistent data, such as undeleted links and partially compensated sub-processes, which can lead to data integrity challenges and operational inconsistencies for your headless commerce platform.

To reproduce this, an ESHOPMAN developer would typically:

1. Configure a workflow with store: true and a retentionTime using the workflow-engine-redis.
2. Include steps that create remote links (e.g., createRemoteLinkStep) and run sub-workflows (runAsStep) where the sub-workflow itself is defined without the store option.
3. Allow the workflow to complete successfully.
4. Artificially trigger a failure in a post-completion callback, such as releaseGroupedEvents.
5. Observe the final state of the workflow and its steps, noting the 'FAILED' status and any remaining links.

Proposed Solutions for ESHOPMAN Developers

To address these critical issues and ensure robust data integrity for ESHOPMAN workflows, the community has discussed potential fixes:

• For Link Dismissal: The system should derive link direction from the explicit link definition rather than relying on the key order within the stored JSON. Alternatively, keys could be normalized before lookup to ensure consistent ordering.
• For Sub-workflow Compensation: The runAsStep mechanism should either enforce the store option for sub-workflows it executes or ensure that enough state is carried forward to allow for proper compensation without needing to reload a non-existent transaction.

These improvements are vital for ESHOPMAN developers and merchants, ensuring that your headless commerce operations maintain data consistency even in complex error recovery scenarios. Staying informed about these technical nuances helps us collectively build a more resilient ESHOPMAN ecosystem.