Critical Insight: ESHOPMAN Pricing Inconsistencies for Large Catalogs Over 4,000 Products

As experts in e-commerce migrations and ESHOPMAN deployments, we often delve into the platform's core functionalities to ensure optimal performance for our clients. A recent discovery within the ESHOPMAN community highlights a crucial detail regarding how prices are calculated and displayed for large product catalogs, particularly those exceeding 4,000 price sets.

The ESHOPMAN Pricing Anomaly: Silent Price Disappearance

A significant issue has been identified where ESHOPMAN's core data fetching mechanism, specifically when utilizing the query.graph functionality to resolve calculated_price for 4,000 or more price sets, silently fails to price items beyond this threshold. What happens is that exactly the first 4,000 price sets receive their calculated prices, while every subsequent price set comes back without any calculated_price at all. Crucially, this occurs without any error or warning, leading to a storefront experience where many product variants appear to have no price.

Impact on Your ESHOPMAN Storefront and Data

This anomaly can have a profound impact on your ESHOPMAN-powered HubSpot CMS storefronts and backend operations:

• Inaccurate Storefront Displays: Product listings on your HubSpot CMS storefronts may show missing prices for a large portion of your catalog, leading to a poor customer experience and lost sales.
• Flawed Filtering and Sorting: Any custom logic or filters that rely on price bands or sorting by price will yield incorrect results. For example, a price-band filter might show only a fraction of the actual products that should match.
• Data Integrity Concerns: The inconsistency means that data fetched via ESHOPMAN's Store API for product variants will not accurately reflect pricing for larger datasets.

For instance, a query fetching 5,000 product variants might only successfully price 4,000 of them, leaving the rest unpriced. This can be particularly problematic for stores with extensive product offerings.

Understanding the Technical Root Cause

The core of this issue lies in how ESHOPMAN's internal data fetching and pricing modules interact, specifically concerning batch processing and object mutation:

1. Batching of Large Requests: ESHOPMAN's data fetcher is designed to split requests involving a large number of IDs (specifically, 4,000 or more) into smaller batches. While it correctly copies filters for each batch, it inadvertently reuses the same options object, and thus the same relations array, across all these batches.
2. In-Place Array Mutation: Concurrently, ESHOPMAN's internal pricing service (PricingModuleService.setupCalculatedPriceConfig_) is responsible for handling the virtual calculated_price relation. During its process, it removes this relation by splicing it out of the relations array in place. Because the options object (and its relations array) is shared across all batches, the first batch's processing modifies the array for all subsequent batches.

Consequently, after the first batch is processed, the shared relations array no longer contains calculated_price. Any subsequent batches then proceed without the necessary relation, resulting in unpriced items without any explicit error.

Reproducing the Issue

Developers can easily reproduce this behavior in an ESHOPMAN project by running a simple script after populating the database with more than 4,000 variants, each having a price in a specified currency:

// src/scripts/repro-price-batch.ts
// npx eshopman exec ./src/scripts/repro-price-batch.ts 
import { ExecArgs } from "@eshopman/framework/types"
import { ContainerRegistrationKeys, QueryContext } from "@eshopman/framework/utils"

export default async function repro({ container, args }: ExecArgs) {
  const query = container.resolve(ContainerRegistrationKeys.QUERY)
  const currency_code = args?.[0] ?? "usd"

  const { data } = await query.graph({
    entity: "variant",
    fields: ["id", "calculated_price.calculated_amount"],
    context: { calculated_price: QueryContext({ currency_code }) },
  })

  const priced = data.filter((v) => v.calculated_price?.calculated_amount != null)
  console.log({ variants: data.length, priced: priced.length }) // priced === 4000 for any count >= 4000
}

Running this script will consistently show that regardless of the total number of variants, only 4,000 will have a calculated_price.

Immediate Workarounds and Suggested Fixes

While ESHOPMAN's core team works on an official fix, developers managing large catalogs can implement immediate workarounds:

• Chunk Your Queries: Manually split your pricing queries into smaller chunks (e.g., batches of 1,000 IDs) to ensure all items are priced correctly. This involves custom code to paginate or batch your requests.
• Monitor for Inconsistencies: Implement logging within your ESHOPMAN application to detect when this pricing anomaly occurs, allowing for proactive intervention.

The community has also suggested two potential fixes for ESHOPMAN's core:

• Modify normalizePriceSetConfig to deep copy the relations array, preventing in-place mutation of the caller's options.
• Ensure fetchRemoteDataBatched provides each batch with its own distinct copy of the options object, or at least the options.relations array.

At Move My Store, we emphasize the importance of robust and accurate data for headless commerce. This insight serves as a critical piece of community knowledge for ESHOPMAN developers and merchants, enabling them to maintain pricing integrity across their extensive product catalogs.