Ucommerce Synchronization Overview
- Morten Lindequist Køhler
Ucommerce Connector uses a synchronization manager, which is responsible for reading configuration parameters and based on these parameters determines when actual synchronization should start.
Synchronization Manager
Synchronization manager continuously runs and once in a while checks the ECommerce configuration settings from Perfion. The frequency of how often this check is performed is based on parameters set in Perfion ECommerce Settings. If parameters related to configuration update intervals or synchronization frequency are not set in Perfion, e.g. default check periods are used, then it defaults to its internal check interval, which is 10 minutes. There are three parameters used to determine how often manager has to check for new configurations and also how often data synchronization should run: ConfigurationUpdateFrequencyInMin, SyncFrequencyInMin, DataMaxAgeInMin. Refer to ECommerce Data Mapper – Settings for more information about how these parameters work.
Synchronization manager writes what it is doing and when to the log file. Please note, that if there is some configuration issue in Perfion, e.g. ECommerce mappings are set incorrectly for ECommerce API, or mappings are missing some parameters needed for Ucommerce Connector (e.g. Sync type parameters are not defined correctly or mappings do not have all mandatory parameters), then synchronization manager will report about these failures to log file and wait some time before it will try to read configuration data again. Therefore, if the ECommerce mappings were set up correctly and work, it is not recommended to modify any settings parameters in Perfion without having access to Ucommerce Connector log file. Any failures in synchronization manager will not be reported back to Perfion.
Logging
There are two types of logging implemented in Ucommerce Connector: log file and log in Perfion.
Ucommerce Connector Log File
This log file will have information about everything what is happening in Ucommerce Connector during synchronization manager check of configuration and also when performing actual product and category synchronizations. The location of the log file and how detailed logging will be, one can define in Ucommerce Connector configuration file (refer to Ucommerce Connector Configuration File). Each Ucommerce Connector instance will have its own log file.
There is one log file created each day and there are no log file cleanup procedures, so if disk storage is limited, then users have to cleanup old log files manually.
Log in Perfion
Ucommerce Connector can report some of log related data back to Perfion. The information reported to Perfion is, however, limited and only information related to synchronization of individual product or category items is logged. If during synchronization something wrong will happen with one product, then Ucommerce Connector will be able to send this information back to Perfion and save it to the Perfion feature configured to be used with SyncMsg mapping parameter. The information will be saved as part of the original product data, which failed during synchronization. Therefore, if there is something wrong with one product data, then after synchronization, all information of what went wrong with that particular product, will be available for users as part of that product’s data.
The log information per synchronization entity item will be updated each time synchronization runs and selects that item for synchronization. For example, if a product is chosen to be synchronized based on its sync parameters. If the product failed to synchronize for the first time, but was synchronized later without any errors, then log related information will be handled accordingly. On error it will be update with new error and warning messages, and on success, any old log information will be removed.
SyncMsg parameter can be defined for product and for category. Each log information written to SyncMsg related feature will be related to the actual product or category. Any information sent to Perfion will be also written to Ucommerce Connector log file, so for full information what is happening one may need to refer to this file. Even though the information provided for each synchronization item is limited, it allows to detect most of data related issues without leaving Perfion.
Ucommerce Connector Basic Concepts and Error Handling
The Ucommerce Connector was designed to run forever. Any errors happening inside Ucommerce Connector should not stop it completely. In most cases it will report the errors to the log file and will continue its designed cycle of life. For example, restarting PC where Ucommerce Connectors run should not be a problem.
Error Handling
Any errors while retrieving configuration data from Perfion or Ucommerce and also any ECommerce configuration parsing validation errors are considered critical errors in Ucommerce Connector and synchronization manager will not start any synchronizations until all required data is in order.
The synchronization will stop if there are any errors while retrieving initial information about data in Perfion and in Ucommerce before the actual synchronization of product and category items. For example, when product synchronization starts, Ucommerce Connector tries to get all necessary data from Perfion to know which products it has. The similar procedure is repeated for Ucommerce products, so it can compare both and find differences. Any failures in this process will be treated as critical failure and will stop synchronization.
If synchronization initialization is successful and Ucommerce Connector starts, let’s say, product synchronization, then product items will be processed independently. This means that if there is any failure with one product, it will not directly affect synchronization of other products. Any errors happening within a single product item synchronization process will be logged independently to Ucommerce Connector log file and also back to Perfion where necessary.
Ucommerce Connector Recovery from Failures
The Ucommerce Connector was designed to recover from any crash and it is also safe to restart at any time. It will download the newest configuration data and will continue synchronization. In case the last synchronization failed, let’s say at 50%, then it will continue further from where it left, because any item synchronization in Ucommerce Connector is final. This means that after each entity item is synchronized with Ucommerce, its status will be updated back to Perfion. The next time Ucommerce Connector will read data, it will no longer pick the same items, which were synchronized previously.
NOTE: regardless of what are predefined synchronization intervals, the first synchronization will run immediately after Connector first start, unless synchronization is disabled using parameters in Perfion.
Ucommerce Connector was designed so that synchronization can run at any time regardless of what are the synchronization intervals. The Perfion is considered as master of data and therefore (in theory) the more often the synchronization runs, the closer data in Ucommerce is to the data in Perfion. If data synchronization is not desired at some point in time, then active measures must be taken to set up configurations in Perfion not to perform any updates at that time interval. For example, if some existing data is currently being updated (modified) in Perfion and should not be sent to Ucommerce yet until modification/update is complete, then users have to set those particular items explicitly not to be synchronized. If this is not done, then users may experience situations, when data is synchronized with Ucommerce in the middle of modification process. There are many ways to make sure that some data is not sent to Ucommerce.
Independent Item Error Handling
All requests sent to ECommerce API may fail for various reasons, e.g. because of timeout or error can be received because of invalid data sent to API, etc. Ucommerce Connector is designed to handle these errors by using configurable timeout intervals and number of retries for ECommerce API. These settings can be set in Ucommerce Connector configuration file (refer to Ucommerce Connector Configuration File).
By default Ucommerce Connector is configured to retry three times after any failure while sending or receiving data to/from ECommerce API. There are various types of errors which may happen when sending and receiving data from this API, but timeout and retry parameters allow to mitigate most of them.
Synchronization
Individual item synchronization starts when synchronization manager initializes it based on configurations it has.
Individual item synchronization consist of two parts: category synchronization and product synchronization.
In order to handle product relations to categories correctly, it is required that all categories are up to date and also that all categories known to Perfion exist in Ucommerce. Therefore, before starting product synchronization Ucommerce Connector will always perform category synchronization.
Regardless if it is category or product synchronization, the typical procedure of synchronization is more or less standard:
Task # | Task | Description |
1 | Get reduced entity data from Perfion | Retrieves information from Perfion about all available entity items. Only limited data of each entity item data set is downloaded. For example it reads entity item key, sync parameters and some other important information, which will be used to understand the current status of data in Perfion. |
2 | Get reduced entity data from Ucommerce | Retrieves information from Ucommerce about all available entity items. Only limited data of each entity item data set is downloaded, for example, item ID. |
3 | Compare data | Compares the data received in task #1 and task #2 and finds out which data has to be synchronized. The result after processing is three lists of entity items: items for creating new, items for updating existing and items for deletion. During this process the data will be selected based on various rules. Refer to Synchronization Algorithm for more information how items are selected. |
4 | Create new items | The task will create all selected items as new data in Ucommerce. |
5 | Update existing items | The task will update all selected items in Ucommerce. |
6 | Delete not existing items | The task will delete all selected items from Ucommerce. |
Synchronization Algorithm
All data in Perfion related to synchronization must have a unique key (defined by KeyField in ECommerce mappings for product and category entities). Any items retrieved via ECommerce API from Perfion which do not have a key, will not be used in synchronization. The items with duplicate key will also be rejected.
The similar check is also performed for SyncID field, which is used to store entity item’s ID value from Ucommerce. There can be no duplicate entity items in Perfion, which use the same SyncID value. If any of these items are found, they will be removed.
Before synchronization Ucommerce Connector retrieves information about all available entity items from Perfion and also from Ucommerce. It retrieves only basic information needed to determine if item has to be synchronized. For entity item data from Perfion it includes all sync type parameters. Items from Ucommerce will be retrieved with item IDs.
In the first step Ucommerce Connector will use a common algorithm for all entity types to select preliminary items for further processing. This selection does not take into account any other item data but instead validates item keys and sync IDs and creates three list of items: items for creation, for update and for deletion. The table below shows how entity items will be selected to these three lists.
Condition | Action |
SyncID value in Perfion is not set or SyncID value is set, but the Ucommerce ID value was not found in any of items retrieved from Ucommerce | The item is added to the list of items for creation in Ucommerce |
SyncID value in Perfion is set and it matches one of Ucommerce item IDs | The item is added to the list of items for update in Ucommerce |
Ucommerce item ID is not found in any of Perfion item SyncID values | The item is added to the list of items for deletion from Ucommerce |
The second step in determination which item has to be synchronized uses entity specific rules and apply them to each item in the lists selected in the first step.
Order | Condition | Action | Reason |
Context: New category creation rules, Data source: The list of categories for creation | |||
1 | If Ucommerce Connector is not set to allow creation of new items, (parameter SyncType does not have “Add” option, refer to ECommerce Data Mapper – Settings). | No Sync | Ucommerce Connector is not allowed to add new items to Ucommerce. |
2 | If the item’s SyncType parameter is set to “NoUpdate” value | No Sync | The category item is used in Ucommerce, but marked not to be synchronized. |
3 | If the item’s SyncCategory value is set to “false” | No Sync | The category item is selected not to be used in Ucommerce. |
4 | Always | Sync | In any other case a new category will be created in Ucommerce. |
Context: Existing category update rules, Data source: The list of categories for update | |||
1 | If Ucommerce Connector is not set to allow update of existing items, (parameter SyncType does not have “Update” option, refer to ECommerce Data Mapper – Settings) | No Sync | Ucommerce Connector is not allowed to update any existing data in Ucommerce. |
2 | If the item’s SyncType parameter is set to “NoUpdate” value | No Sync | The category item is used in Ucommerce, but marked not to be synchronized. |
3 | If the item’s SyncCategory value is set to “false” | No Sync And Add item to deletion list | The category is selected not to be used in Ucommerce. Since the category already exist in Ucommerce, it must be removed and therefore, it will be added to the list of categories for deletion. |
4 | If the item is marked as deleted (IsDeleted parameter, refer to Category Mapping Parameters) in Perfion and also as deleted in Ucommerce | No Sync | There is no need to update deleted items in Ucommerce, which cannot be used anyway. |
5 | If the item’s SyncDate value is not set | Sync | Missing SyncDate value is a designed way to force item synchronization. |
6 | If the item’s SyncDate value is set, but DataMaxAgeInMin parameter is not used (refer to ECommerce Data Mapper – Settings). | No Sync | DataMaxAgeInMin parameter controls how often data has to be refreshed in Ucommerce. If it is not set, then the category will not be periodically synchronized with Ucommerce and users will need to manually select categories for synchronization, e.g. by removing the item’s SyncDate value. |
7 | If the item’s SyncDate value is set, and DataMaxAgeInMin parameter is set (refer to ECommerce Data Mapper – Settings), then Ucommerce Connector will check SyncDate value. If date is not older than maximum allowed data age. | No Sync | Data refresh is enabled, but the time did not come yet for the current item to be re-synchronized. |
8 | Always | Sync | In any other case the category item will be updated in Ucommerce. In this case it will be a category item, which has expired date. |
Context: Existing category deletion rules, Data source: The list of categories for deletion | |||
1 | If Ucommerce Connector is not set to allow deletion of Ucommerce items, (parameter SyncType does not have “Delete” option, refer to ECommerce Data Mapper – Settings) | No Sync | Ucommerce Connector is not allowed to delete existing items in Ucommerce. |
2 | If Ucommerce Connector is set to delete Ucommerce items (parameter SyncType has “Delete” option, refer to ECommerce Data Mapper – Settings) and the item in Ucommerce is already deleted | No Sync | The Ucommerce Connector is allowed to delete Ucommerce items, but the item in Ucommerce is already deleted, so there is no reason to delete again. |
3 | Always | Sync | In any other case the category item will be deleted in Ucommerce. |
Context: New product creation rules, Data source: The list of products for creation | |||
1 | If Ucommerce Connector is not set to allow creation of new items, (parameter SyncType does not have “Add” option, refer to ECommerce Data Mapper – Settings) | No Sync | Ucommerce Connector is not allowed to add new items to Ucommerce. |
2 | If the item’s SyncType parameter is set to “NoUpdate” value (refer to ECommerce Data Mapper – Settings) | No Sync | The product item is used in Ucommerce, but marked not to be synchronized. |
3 | If the item’s SyncProduct value is set to “false” | No Sync | The product item is selected not to be used in Ucommerce. |
4 | Always | Sync | In any other case a new product will be created in Ucommerce. |
Context: Existing product update rules, Data source: The list of products for update | |||
1 | If Ucommerce Connector is not set to allow update of existing items, (parameter SyncType does not have “Update” option, refer to ECommerce Data Mapper – Settings) | No Sync | Ucommerce Connector is not allowed to update any existing data in Ucommerce. |
2 | If the item’s SyncType parameter is set to “NoUpdate” value (refer to ECommerce Data Mapper – Settings) | No Sync | The product item is used in Ucommerce, but marked not to be synchronized. |
3 | If the item’s SyncProduct value is set to “false” | No Sync And Add item to deletion list | The product is selected not to be used in Ucommerce. Since the product already exist in Ucommerce, it must be removed and therefore, it will be added to the list of products for deletion. |
4 | If the item’s SyncDate value is not set
| Sync | Missing SyncDate value is a designed way to force item synchronization. |
5 | If the item’s SyncDate value is set, but DataMaxAgeInMin parameter is not used (refer to ECommerce Data Mapper – Settings) | No Sync | DataMaxAgeInMin parameter controls how often data has to be refreshed in Ucommerce. If it is not set, then the product will not be periodically synchronized with Ucommerce and users will need to manually select products for synchronization, e.g. by removing the item’s SyncDate value. |
6 | If the item’s SyncDate value is set, and DataMaxAgeInMin parameter is set (refer to ECommerce Data Mapper – Settings), then Ucommerce Connector will check SyncDate value. If data is not older than maximum allowed data age. | No Sync | Data refresh is enabled, but the time did not come yet for current item to be re-synchronized. |
7 | Always | Sync | In any other case the product item will be updated in Ucommerce. In this case it will be a product item, which has expired date. |
Context: Existing product deletion rules, Data source: The list of products for deletion | |||
1 | If Ucommerce Connector is not set to allow deletion of Ucommerce items, (parameter SyncType does not have “Delete” option, refer to ECommerce Data Mapper – Settings) | No Sync | Ucommerce Connector is not allowed to delete existing items in Ucommerce. |
2 | Always | Sync | In any other case the product item will be deleted in Ucommerce. |
NOTE: for each context in the table the rules are checked in defined order until the condition is matched exactly. If rule is not matched, then the rules will be checked until one does. In each context, the last rule is the one which will be always matched, e.g. it does not have any condition.
Synchronization Order
After the data for synchronization is selected into three different types of lists, these lists are processed in the following order:
Deletions
Creations
Updates
The order for synchronization is chosen based on Ucommerce data structure. Ucommerce data is hierarchical and data have to be handled in a certain order to avoid situations, when one item cannot be synchronized, because it is related to another item, which was not synchronized yet and is not in Ucommerce. One item dependency to another item issue is relevant to three types of data in Ucommerce:
Categories are hierarchical, therefore, to create a child category one must have a parent category.
Products have relations to categories, therefore, categories must exist in order to relate products to them.
Products may have relations to other products, therefore, the related product must exist in order to create a product.
In order to make sure that all data which has dependencies is created correctly, Ucommerce Connector will create first only a temporary data for each category and product without any next level dependencies. This data will be hidden from users in Ucommerce, e.g. products will belong to a dummy category used by Ucommerce Connector, which is marked as deleted. This assures, that temporary products are not visible from Ucommerce until they are finally updated to their final state. The same is done also with new categories. They will be marked as deleted and will not be accessible directly from GUI.
After all new items with dummy data are created as it was described above, these items are added into the list of items for update. During update process, those new temporary items will be updated to their final state just like any other existing item in Ucommerce, which was marked for update in Perfion. This extra step of data manipulation makes it possible to process data items independently from each other. That means, that during update process of a child category, the parent category will be available and there will be no need to create it first and only then create a child category. The hierarchy may become really complicated and there may be situations with circular dependencies (e.g. chicken and egg problem) and this approach solves these issues.
Data Synchronization Overview
During synchronization, the data is processed in chunks. The default size of the chunk is 25 items, but it is user configurable (parameter ECDataChunkSize in Ucommerce Connector configuration file). The same procedure is used for all creation and update synchronization tasks for products and categories. It is not used only with delete tasks, which don’t require data retrieval from Perfion.
A chunk of data is retrieved from Perfion with a full data set required for a task. For example, if data synchronization is not limited from ECommerce Settings, then all data is retrieved from Perfion for each item under synchronization. However, if user chose not to synchronize some data, e.g. media or product variants, then Ucommerce Connector will automatically control the quantity of data to request via ECommerce API. For example, if SyncMedia (from ECommerce Settings in Perfion) parameter is set to ‘false’, then Ucommerce Connector will automatically disable all binary data retrieval from ECommerce API, which can lead to significantly faster data retrieval. The similar data quantity control is performed when retrieving product data if variants synchronization is disabled. If parameter VariantSupportEnabled is set to ‘false’, then Ucommerce Connector will exclude all variant data from requests sent to ECommerce API.
After a chunk of entity items is retrieved via ECommerce API, the Ucommerce Connector will start processing all items in parallel. At the same time, Ucommerce Connector will send request to ECommerce API to retrieve the next chunk of data for synchronization in parallel to individual item processing, so that when item processing is completed, the next chunk of data is available without any extra delays.
During individual item processing, each item synchronization is independent to other items and all procedures for a single items are executed sequentially, while several items are processed at the same time in parallel.
Item deletions from Ucommerce follow the same logic as described above, e.g. are processed in parallel and the only difference is that they do not need data retrieval from Perfion.
In order to synchronize products, Ucommerce Connector will additionally download data about categories. Ucommerce Connector will send a single request to ECommerce API to get all category Keys and SyncIDs, then it will retrieve all category IDs from Ucommerce. Finally, it will compare both lists and will determine the complete list of category IDs and Keys, which are known in Perfion and Ucommerce and which will be used for product synchronization. This procedure is performed only once before product synchronization.
Data Handling in Ucommerce
Ucommerce uses NHibernate for their data handling, which is an ORM (Object Relational Mapping) solution for Microsoft .NET platform. NHibernate is a backbone of Ucommerce API and handles any data querying and updates in underlying Ucommerce database. It converts the relational data into the e-commerce objects, which allows simplified and quick way to work with data without going directly to database. However, NHibernate is not as fast as direct data access in database. Moreover, NHibernate translates any data request to universal SQL queries, which are then executed in Ucommerce database, but this process is not always optimal and it is not always possible for external entities (such as Ucommerce Connector) to interfere with this process.
Ucommerce Connector was optimized for best performance while using NHibernate, but the main focus is on handling individual synchronization items independently. Ucommerce Connector will handle data per synchronization entity item (e.g. product, category), so in these situations it will eager load the full data set into memory, update the entity item state and then will allow NHibernate to apply changes to database.
New Product Creation
The creation of a product consist of two parts:
New temporary product creation
Update of temporary product with real data from Perfion
The creation of temporary product takes three SQL requests to Ucommerce database. The temporary product will be created with correct product name and SKU, but it will not have any other data. The temporary product will belong to the deleted temporary category used by Ucommerce Connector, so the temporary product will not be accessible from Ucommerce GUI.
After creation of temporary product, it will be known only to Ucommerce, e.g. the new temporary product ID will not be updated to Perfion. The first time the product will be known to Perfion as existing in Ucommerce will be only after product is successfully updated in product update procedure, which is executed after all new temporary products are created. Refer to Existing Product Update for more information about how product will be processed further.
In case of Ucommerce Connector failure, e.g. PC lost power, OS restart, etc. while Ucommerce Connector is running, all temporary products in Ucommerce will remain as temporary data until Ucommerce Connector starts a new synchronization. During new synchronization all those temporary products will be automatically deleted and Ucommerce Connector will update Ucommerce to correct state based on data from Perfion.
Existing Product Update
The table below shows how product update is performed and how many actions each task requires. In case of ECommerce API, an action is a request to ECommerce API web service, while in case of Ucommerce API, it is a SQL query to Ucommerce database.
Task # | Task | Actions (source) | Conditions | Description |
1 | Load existing product part 1 – main data | 1 (Ucommerce API) | Always performed | The existing product data is loaded. This task will load most of the product data from Ucommerce database, but not all. The product data in Ucommerce is complex, spans across several database tables and cannot efficiently be loaded all at once, so the process of data loading was optimized by distributing data load in several queries. This is the first one, and in total it can take up to four queries to fully load Ucommerce product. |
2 | Load existing product part 2 – product and variant prices | 1 (Ucommerce API) | If price synchronization is enabled | Retrieves information about all product and its variants prices. The data is loaded only if ECommerce Settings parameter SyncPrices is set to ‘true’. |
3 | Load existing product part 3 – product and variant description properties | 1 (Ucommerce API) | Always performed | Retrieves information about all product and its variants description properties. Description properties include product and variant related descriptions and multilingual information. |
4 | Load existing product part 4 – variant data | 1 (Ucommerce API) | If variant synchronization is enabled | Retrieves all product variants with their main data. Note, the data for variant prices and description properties is retrieved in tasks #2 and #3. The variant data is loaded only if ECommerce Settings parameter VariantSupportEnabled is set to ‘true’. |
5 | Update of product and variant state in memory | - | Always performed | The Ucommerce Connector will compare existing loaded product data with incoming data from Perfion and will update product and its variants states accordingly. Since all product related data is loaded into memory, the update process does not require to execute any queries. |
6 | Update product state in database | 0-N (Ucommerce API) | Always performed | All product related changes are saved to database. The number of executed queries to Ucommerce database will depend on how many changes there are. This procedure is not directly controlled by Ucommerce Connector and all changes to product state are determined automatically by NHibernate used by Ucommerce, which tracks product changes and determines all queries needed to execute in order to update the product state. Since the product and variant structure in database is rather complex, there may be a huge amount of queries to database executed at this task. All queries are executed in a transaction, but for a new (temporary) product update, the number of queries can variate from tens to hundreds. For example, an update of a new (temporary) product with four variants, where product and variant data includes several custom fields and where some of those fields are multilingual, requires execution of around 100 queries. |
7 | Update Perfion | 1 (ECommerce API) | Always performed | Updates synchronization related parameters in Perfion. In case of error, the information about error will be updated (SyncMsg) and in case of success, other parameters will be updated (SyncID, SyncDate, SyncMsg). |
Existing Product Deletion
The deletion of existing product does not require retrieving of any data from Perfion and after deletion, there will be no data sent to Perfion.
The deletion of existing product executes many SQL queries to Ucommerce database. The number of executed queries depends on how many data product has. The Ucommerce Connector will first load the product from database in order to check if it exist before deletion, which requires execution of a single query. Then, deletion will be initiated via NHibernate, which will perform all required actions to remove product related data from Ucommerce database. The number of executed queries in order to remove all product related data usually is counted in tens. For example, deletion of a product with four variants, where product and variant data includes several custom fields and where some of those fields are multilingual, requires execution of around 50 queries.
New Category Creation
The creation of a category consist of two parts:
New temporary category creation
Update of temporary category with real data from Perfion
The creation of temporary category typically takes three SQL requests to Ucommerce database. The temporary category will be created with correct category name, but it will not have any other data. The temporary category will be created as deleted category, so it will not be accessible from Ucommerce GUI.
After creation of temporary category, it will be known only to Ucommerce, e.g. the new temporary category ID will not be updated to Perfion. The first time the category will be known to Perfion as existing in Ucommerce will be only after category is successfully updated in category update procedure, which is executed after all new temporary categories are created. Refer to Existing Category Update for more information about how category will be processed further.
In case of Ucommerce Connector failure, e.g. PC lost power, OS restart, etc. while Ucommerce Connector is running, all temporary categories in Ucommerce will remain as temporary data. The Ucommerce does not support category physical deletion and therefore, all temporary created categories will remain in Ucommerce forever, unless users will manually delete them via direct access to Ucommerce database.
Existing Category Update
The table below shows how category update is performed and how many actions each task requires. In case of ECommerce API, an action is a request to ECommerce API web service, while in case of Ucommerce API, it is a SQL query to Ucommerce database.
Task # | Task | Actions (source) | Conditions | Description |
1 | Load existing category | 1 (Ucommerce API) | Always performed | The existing category data is loaded. |
5 | Update of category state in memory | - | Always performed | The Ucommerce Connector will compare existing loaded category data with incoming data from Perfion and will update category state accordingly. Since all category related data is loaded into memory, the update process does not require to execute any queries. |
6 | Update category state in database | 0-N (Ucommerce API) | Always performed | All category related changes are saved to database. The number of executed queries to Ucommerce database will depend on how many changes there are. This procedure is not directly controlled by Ucommerce Connector and all changes to category state are determined automatically by NHibernate used by Ucommerce, which tracks category changes and determines all queries needed to execute in order to update the category state. The category structure in database is significantly less complex as compared to a product, so the number of queries needed to be executed at this step is significantly lower. All queries are executed in a transaction, but for a new (temporary) category update, the number of queries can variate from a few, up to 10 or more. For example, an update of a new (temporary) category, where category data includes several custom fields and where some of those fields are multilingual, requires execution of around 8 queries. |
7 | Update Perfion | 1 (ECommerce API) | Always performed | Updates synchronization related parameters in Perfion. In case of error, the information about error will be updated (SyncMsg) and in case of success, other parameters will be updated (SyncID, SyncDate, SyncMsg). |
Existing Category Deletion
The deletion of existing category does not require retrieving of any data from Perfion and after deletion, there will be no data sent to Perfion.
The deletion of existing category executes two SQL queries in Ucommerce database. The Ucommerce Connector will first load the category from database in order to check if it exists before deletion, then it will “soft” delete it by updating category “Deleted” property to ‘true’.
Synchronization Performance
Synchronization performance is directly related to how fast the data can be handled via ECommerce API and Ucommerce API.
ECommerce API has no predefined limits, but it is not designed to return quick responses for record by record requests. Its main task is to combine many types of data in some special format acceptable for ECommerce applications and also to return data in larger quantities. ECommerce API uses Perfion API for raw data request to get data from Perfion, then it combines that data and finally sends response back to user.
Ucommerce API is designed to handle many requests and responds promptly, but data structure in Ucommerce database is relatively complex and spans over many database tables, so manipulating this data requires a large number of queries to execute.
Ucommerce Connector handles all requests to Ucommerce API and also to ECommerce API in parallel to increase performance. Instead of processing one item at a time, the Ucommerce Connector will process up to 20 items simultaneously.
Ucommerce Connector handles data from Perfion in chunks, e.g. it takes 25 products from ECommerce API and then processes all those products one by one in parallel mode. Then it takes another 25 products, etc. This allows to optimize performance, because while data is processed, the data from ECommerce API can be retrieved in parallel without wasting any additional time.
The overall data synchronization performance in Ucommerce Connector, in most cases, depends on Ucommerce API performance. There is no universal figure to show how much time data synchronization will take, because it all depends on how much data is synchronized with Ucommerce. The main factors influencing synchronization time:
How many variants products have. A variant in Ucommerce has the same structure as a product it belongs to. There may be slight differences in custom properties used by parent product and variants, but variants have the same complexity of the product. One can count product synchronization time not by how many products there are to synchronize, but instead add number of products to the number of all variants from all products, and this total count will be more useful while determining synchronization performance. Then performance figures will be more linear if counted per each product and variant item as compared to counting using only per each product item.
How many custom properties product, variant and category has. Each custom property requires at least one extra query to be executed. If properties are multilingual, then one can multiply number of queries required by number of languages specified for Ucommerce Connector. Product properties have more complex structure than categories, so product properties synchronization will take more time.
The most time consuming synchronization tasks ordered by the most time consuming task first:
Product creation. Product creation is the same as product update, but in addition it has to go through temporary product creation procedure. Moreover, new product creation requires adding all product related data to database, so Ucommerce Connector will have to execute a lot of queries to update product state.
Product deletion. Product deletion is almost always time consuming, unless temporary product is deleted. If existing product is deleted, then in order to delete the product and all data related to it, the product data should be first read from database, and then deleted. Since product structure is rather complex, there are usually many queries which has to be executed in order to delete everything related to a product.
Product update. In product update, we exclude those cases when a new product has to be created. In most cases, when existing product is updated, usually it will not have many changes, then such product update will not require many queries to be executed. This operation can be compared to category creation and update procedures in terms of performance.
Category creation. Category creation takes most of the time from category related synchronization tasks, but it cannot be directly compared to most of product related synchronization tasks. Category creation will take significantly less time and a typically will require less than 10 queries to be executed.
Category update. In category update, we exclude those cases when a new category has to be created. Typical category update will require very few queries to be executed and it is very fast compared to most of other synchronization tasks.
Category deletion. Takes almost no time, because categories are “soft” deleted and it requires only two queries.
Image Handling
Products, variants and categories may have images in Ucommerce and ECommerce API supports transferring of binary data from Perfion to Ucommerce Connector, but at the moment Ucommerce API does not support image upload to their parent CMS image library. Therefore, the image data synchronization feature is currently not available.
The following sections will explain how one can setup images to be used for product, category and variant synchronization, but until Ucommerce API does not support image upload, this configuration will not be usable and it is recommended to turn off image synchronization from ECommerce Settings by setting SyncMedia parameter value to ‘false’. Moreover, image configurations in ECommerce Mappings are a subject to change, because supported features will be directly related to what features Ucommerce API will provide for image handling.
Image Data Configuration in ECommerce Mapping
Images for Ucommerce Connector can be defined by using different ECommerce Output Kinds and multiple parameters can be added to ECommerce Mappings to define more images, than Ucommerce fixed properties allow. One can use Ucommerce custom properties in product and category definition to add more images. Each image related parameter in mappings can represent a single image of image or file type feature (multiple images are not supported). The example of how one can configure multiple types of images for category is shown in the Figure 12.
Figure 12 shows all ECommerce Output Kinds which can be used to define images, but there can be multiple features used for each Output Kind. In this example, the “To” mapping “Image” refers to Ucommerce fixed category definition property, while the other three properties are custom category definition properties with names “CustomImage1”, “CustomImage2” and “CustomImage3”.
Another important fact is, that all images, which are defined using URLs, must be accessible from where Ucommerce Connector is running. Refer to Supported Image Types for more information.
Image configuration for categories, products and variants follow the same rules. One can use fixed Ucommerce properties if available and also add additional images by using custom properties.
Supported Image Types
There are two types of images one can specify in ECommerce mappings:
By using URL of an image. Only absolute URLs can be used and images must be accessible from where Ucommerce Connector is running.
By using Base64 encoded string of an image. Images defined using embedded option are transferred via ECommerce API with URL as using the first option, but in addition there will be binary image data attached to response.
If images are defined using Base64 encoded inline data (binary data in an ASCII string format), then Ucommerce Connector will not need to take additional request to get data from external server, but then the request with such data may take more time to receive via ECommerce API to Ucommerce Connector. Base64 encoded inline data takes more space than raw binary data which can be downloaded directly from external image server. According to Wikipedia, Base64 encoded binary data size is about 137% of the original data size or to be more precise: Base64 encoded image size = original binary image size * 1.37 + 814 bytes.
Caching in Ucommerce
Ucommerce uses several types of caching when showing data in their parent CMS system administration (Ucommerce admin), when it performs various searches or there could be some custom distributed cache setups. This can be an issue sometimes to validate if all data which was just synchronized from Perfion to Ucommerce was correctly synchronized and also to be able to access the new data immediately from Ucommerce back-end and front-end.
The Ucommerce Connector uses Ucommerce API to perform synchronization directly with Ucommerce database and does not control any caching state in Ucommerce admin. So even if Connector have just finished synchronization and all new data was saved in Ucommerce database, it does not mean that it will be immediately accessible from Ucommerce admin or from any other place where this data can be used. Usually it can take some time until the cache state is updated in Ucommerce admin to show the new data. In order to control the cache state for synchronized items use the Perfion Ucommerce Cache Manager (refer to Perfion Ucommerce Cache Manager Setup) which was purposed built to clear the Ucommerce cache for newly synchronized entities. Note that the Perfion Ucommerce Cache Manager may not clear all caches used in Ucommerce and that there will be some delay after the data was synchronized until the cache for those entities are actually cleared. The Perfion Ucommerce Cache Manager can be configured with a custom cache update frequency, so this delay can be controlled to some extent.
Other ways to control caching in Ucommerce:
From Ucommerce configuration. The cache refresh interval can be configured as a global setting from Ucommerce configuration files. One can set how long the data can stay in cache until it will be refreshed from database. Such setup does not allow controlling the independent data entities and it could also become a performance issue in case the data age in cache is reduced too much for frequent refreshes.
From Ucommerce admin. To reset cache manually go to Ucommerce admin -> Settings -> Search. Then click the button “Index everything from scratch”. Note, this is not intended way to reset cache state and this operation may take a significant amount of time to complete. Moreover, the index update is just one of the ways to refresh the new data access. It may refresh some indexes so the new data can be found, but it may not show the new data correctly.
Because of caching issues in Ucommerce admin, there is a risk that data in Ucommerce can be updated by overwriting the data coming from Perfion, but which is not shown yet. The Ucommerce admin does not check if the data in database was updated since the last time it was loaded to cache and allows overwriting the new data in database with data coming from admin. Therefore, it is recommended to clear the cache of newly synchronized data in Ucommerce to avoid such issues.
The concept of Connector is such, that Perfion is considered as data master. This means that any data modifications should happen only from Perfion in order to keep the data synchronized. Therefore, we advise not to modify any data from Ucommerce admin which falls under Connector synchronization scope. If any of modifications are done to the data from Ucommerce admin, that data will be removed next time the Connector will perform synchronization of the entities, which were modified manually in Ucommerce admin.