Shopify Synchronization Overview

Shopify Connector uses a synchronization manager, which is responsible for reading configuration parameters and based on these parameters determines when actual synchronization should start.

Synchronization Manager

Synchronization manager continuously runs and once in a while checks the ECommerce configuration settings from Perfion. The frequency of how often this check is performed is based on parameters set in Perfion ECommerce Settings. If parameters related to configuration update intervals or synchronization frequency are not set in Perfion, e.g. default check periods are used, then it defaults to its internal check interval, which is 10 minutes. There are three parameters used to determine how often manager has to check for new configurations and also how often data synchronization should run: ConfigurationUpdateFrequencyInMin, SyncFrequencyInMin, DataMaxAgeInMin. Refer to “ECommerce Data Mapper – Settings” for more information about how these parameters work.

Synchronization manager writes what it is doing and when to the log file. Please note, that if there is some configuration issue in Perfion, e.g. ECommerce mappings are set incorrectly for ECommerce API, or mappings are missing some parameters needed for Shopify Connector (e.g. Sync type parameters are not defined correctly or mappings do not have all mandatory parameters), then synchronization manager will report about these failures to log file and wait some time before it will try to read configuration data again. Therefore, if the ECommerce mappings were set up correctly and works, it is not recommended to modify any settings parameters in Perfion without having access to Shopify Connector log file. Any failures in synchronization manager will not be reported back to Perfion.

Logging

There are two types of logging implemented in Shopify Connector: log file and log in Perfion.

Shopify Connector Log File

This log file will have information about everything what is happening in Shopify Connector during synchronization manager check of configuration and also when performing actual product and category synchronizations. The location of the log file and how detailed logging will be, one can define in Shopify Connector configuration file. Each Shopify Connector instance will have its own log file.

There is one log file created each day and there are no log file cleanup procedures, so if disk storage is limited, then users have to cleanup old log files manually.

Log in Perfion

Shopify Connector can report some of log related data back to Perfion. The information reported to Perfion is, however, limited and only information related to synchronization of individual product or category items is logged. If during synchronization something wrong will happen with one product, then Shopify Connector will be able to send this information back to Perfion and save it to the Perfion feature configured to be used with SyncMsg mapping parameter. The information will be saved as part of the original product data, which failed in synchronization. Therefore, if there is something wrong with one product data, then after synchronization, all information of what went wrong with that particular product, will be available for users as part of that product’s data.

The log information per synchronization entity item will be updated each time synchronization runs and selects that item for synchronization. For example, if a product is chosen to be synchronized based on its sync parameters. If the product failed to synchronize for the first time, but was synchronized later without any errors, then log related information will be handled accordingly. On error it will be update with new error and warning messages, and on success, any old log information will be removed.

SyncMsg parameter can be defined for product and for category. Each log information written to SyncMsg related feature will be related to the actual product or category. Any information sent to Perfion will be also written to Shopify Connector log file, so for full information what is happening one may need to refer to this file. Even though the information provided for each synchronization item is limited, it allows to detect most of data related issues without leaving Perfion.

If ECommerce data is used with more than one channel, then logging will work in the same way, but we recommend to define independent SyncMsg parameter features for each channel to avoid interference. If two or more channels are synchronizing the same entity item and there is only one SyncMsg feature defined for all channels, then each Shopify Connector instance will overwrite data set by other Shopify Connector instances, which synchronize other channels.

Shopify Connector Basic Concepts and Error Handling

The Shopify Connector was designed to run forever. Any errors happening inside Shopify Connector should not stop it completely. In most cases it will report the errors to the log file and will continue its designed cycle of life. For example, restarting PC where Shopify Connectors run should be no problem.

Error Handling

Any errors while retrieving configuration data from Perfion and also any ECommerce configuration parsing validation errors are considered critical errors in Shopify Connector and synchronization manager will not start any synchronizations until all required data is in order.

The synchronization will stop if there are any errors while retrieving initial information about data in Perfion and in Shopify before the actual synchronization of product and category items. For example, when product synchronization starts, Shopify Connector tries to get all necessary data from Perfion to know which products it has. The similar procedure is repeated for Shopify products, so it can compare both and find differences. Any failures in this process will be treated as critical errors and will stop synchronization.

If synchronization initialization is successful and Shopify Connector starts, let’s say, product synchronization, then product items will be processed independently. This means that if there is any failure with one product, it will not directly affect synchronization of other products. Any errors happening within a single product item synchronization process will be logged independently to Shopify Connector log file and also back to Perfion. Note, the Connector’s log file will have more information about any errors or warning detected while performing synchronization.

Shopify Connector Recovery from Failure

The Shopify Connector was designed to recover from any crash and it is also safe to restart at any time. It will download the newest configuration data and will continue synchronization. In order for it to do it automatically, the Shopify Connector must be set up as Windows service.

In case the last synchronization failed, let’s say at 50%, then after it was restarted, it will continue further from where it left, because any item synchronization in Shopify Connector is final. This means that after each entity item is synchronized with Shopify, its status will be updated back to Perfion. The next time Shopify Connector will read data, it will no longer pick the same items, which were synchronized previously.

Shopify Connector was designed so, that synchronization can run at any time regardless of what are the synchronization intervals. The Perfion is considered as master of data and therefore (in theory) the more often the synchronization runs, the closer data in Shopify is to the data in Perfion. If data synchronization is not desired at some point in time, then active measures must be taken to set up configurations in Perfion not to perform any updates at that time interval. For example, if some existing data is currently being updated (modified) in Perfion and should not be sent to Shopify yet until modification/update is complete, then users have to set those particular items not to be synchronized. If this is not done, then users may experience situations, when data is synchronized with Shopify in the middle of modification process. There are many ways to make sure that some data is not sent to Shopify.

Independent Item Error Handling

All requests sent to ECommerce API and Shopify API may fail for various reasons, e.g. because of timeout or error can be received because of invalid data sent to API, etc. Shopify Connector is designed to handle these errors by using configurable timeout intervals for both APIs and also by using number of retries configurations for both APIs. These settings can be set in Shopify Connector configuration file (refer to “Shopify Connector Configuration File”).

By default Shopify Connector is configured to retry three times after any failure while sending or receiving data to/from ECommerce API or Shopify API. There are various types of errors which may happen when sending and receiving data from both APIs and timeout and retry parameters allow to mitigate most of them.

Shopify Connector will report about all problems which happen during sending or receiving of data to the log file, but it will report errors about individual failed items only in a case, when after predefined number of failed request, it could not finish the job. That means that only the last known error will be reported back to Perfion, while log file will have all error messages, which were received each time it tried. Moreover, each sent request has its unique auto-generated ID, which allows to find information about all related requests and responses. Note: to access this information, logging level must be set to “Info” or “Debug” level.

Individual Item Synchronization

Individual item synchronization starts when synchronization manager initializes it based on configurations it has.

Individual item synchronization consist of two parts: category synchronization and product synchronization.

In order to handle product relations to categories correctly, it is required that all categories are up to date and also that all categories known to Perfion exist in Shopify. Therefore, before starting product synchronization Shopify Connector will always perform category synchronization. Category synchronization and product synchronization are independently controlled using ECommerce configurations, so in cases when categories are not used, this step will be skipped. 

Regardless if it is category or product synchronization, the typical procedure of synchronization is more or less standard:

Synchronization Algorithm

All data in Perfion related to synchronization must have a unique key (defined by KeyField in ECommerce mappings for product and category entities). Any items retrieved via ECommerce API from Perfion which do not have a key, will not be used in synchronization. The items with duplicate key will also be rejected.

The similar check is also performed for SyncID field, which is used to store entity item’s ID value from Shopify. There can be no duplicate entity items in Perfion, which use the same SyncID value. If any of these items are found, they will be removed.

Before synchronization Shopify Connector retrieves information about all available entity items from Perfion and also from Shopify. It retrieves only basic information needed to determine if item has to be synchronized. For entity item data from Perfion it includes all sync type parameters and also the parameter, which shows if item is published or not. Items from Shopify will be retrieved with item IDs and parameter, which shows if item is published or not.

In the first step Shopify Connector will use a common algorithm for all entity types to select preliminary items for further processing. This selection does not take into account any other item data but instead validates item keys and sync IDs and creates three list of items: items for creation, for update and for deletion. The table below shows how entity items will be selected to these three lists.

The second step in determination which item has to be synchronized uses entity specific rules and apply them to each item in the lists selected in the first step.

Product Update

The table below shows how product update is performed and how many requests each task requires. The tasks described in this table are also used for product creation, custom collection update and custom collection creation. The difference in between different entity updates or creations will be only in which tasks are used for each. The product update is the most complex procedure of them all, so it uses all tasks, while other entities will use only some of these task. Refer to following sections to see which of tasks shown in the table will not be used with other entities.

New Product Creation

Creation of a new product is a simplified operation as compared to product update procedure (refer to “Product Update”). Creation of new product will perform only the following tasks from product update table: #-1, #0, #4, #5, #7, #8, #9 and #11.

Product creation does not require to cleanup old data, or update variant data, so tasks #1, #2, #3, #6 and #10 are not required.

The task #9 will not delete any collects when new product is created, because collects does not exist in Shopify until the new product is created and collects are manually added. At this task, only new collects will be added.

Custom Collection Update

Update of custom collection is very similar to product update (refer to “Product Update”), but since custom collections are less complex objects, it does not have to perform some of tasks shown in product update table. It will perform tasks: #0, #1, #4, #10 and #11.

Task #-1 is a preparation task, which is used only with product synchronization.

Custom collections does not have variants and may have only one image, so tasks #2, #3, #5, #7 and #8 are not required.

Custom collection references to products (collects) are handled only from product data synchronization, so tasks #6 and #9 are not performed. Deletion of collects are automatically cleaned up by Shopify after custom collection is removed. If the new custom collection is added, then it does not have any product relations yet and product update will handle any relation updates if necessary. If custom collection data is updated, then it does not affect anyhow its relation to products.

New Custom Collection Creation

Creation of a new custom collection is a simplified operation as compared to product update procedure (refer to “Product Update”). Creation of a new custom collection will perform only the following tasks from product update table: #0, #4 and #11.

Task #-1 is a preparation task, which is used only with product synchronization.

Creation of a new custom collection does not require to cleanup old data, so tasks #1, #6 and #10 are not required.

Custom collections does not have variants and may have only one image, so task #2, #3, #5, #7 and #8 are not required.

Task #9 is not required, because product to category relations are handled only via product synchronization.

Product Deletion or Deactivation

The deletion or deactivation of product uses only one request. After deleting the main entity item Shopify will cleanup all related data and will remove any custom collection to product relations (collects) and will also automatically remove any related metafields.

In case deactivation is used instead of deletion, then instead of delete request Shopify Connector will update product “publish” parameter value to “false”. In this case, all the extra data belonging or related to this product (collects and metafields) will remain as they were before deactivation.

Custom Collection Deletion or Deactivation

The deletion or deactivation of custom collection uses only one request. After deleting the main entity item Shopify will cleanup all related data and will remove any custom collection to product relations (collects) and will also automatically remove any related metafields.

In case deactivation is used instead of deletion, then instead of delete request Shopify Connector will update custom collection “published” parameter value to “false”. In this case, all the extra data belonging or related to this custom collection (collects and metafields) will remain as they were before deactivation.

Synchronization Performance

Synchronization performance is directly related to how fast the data can be handled via ECommerce API and Shopify API.

ECommerce API has no predefined limits, but it is not designed to return quick responses for record by record requests. Its main task is to combine many types of data in some special format acceptable for ECommerce applications and also to return data in larger quantities. ECommerce API uses Perfion API for raw data request to get data from Perfion, then it combines that data and finally sends response back to user.

Shopify Admin API is designed to handle many requests and responds promptly, but it has usage limitations, which limits how much data can be requested or sent to Shopify. Shopify uses a so called “leaky bucket” algorithm to control the amount of calls users can use in a period of time. Simplifying how their algorithm works, one can say in conclusion, that Shopify allows two API calls per second (or four, if one uses Shopify Plus account). Refer to https://help.shopify.com/api/getting-started/api-call-limit for more information of how it works.

The Shopify limits mentioned above are more theoretic limits and does not take into account how much time each request-response takes. For example, if product has multiple images, then a single request to Shopify to update this product can take more than three seconds. If data is sent to Shopify sequentially, then it is difficult to exceed the limits specified by Shopify. This, however is not good enough in order to transfer large amounts of data to Shopify, because it does not allow data handling in bulk mode, e.g. to create several products in one request. To handle data in Shopify one needs to work with independent entities and sometimes some entities need more than one request to be properly created/updated. This leads to situation, where update of single entity item, e.g. product, takes three or more request to Shopify.

Shopify Connector handles all request to Shopify and also to ECommerce API in parallel to increase performance. Instead of sending one product update request to Shopify, which may take more than three seconds each, the Shopify Connector will send up to 20 request to Shopify to update multiple products at the same time. Moreover, Shopify Connector will control amount of requests it sends to Shopify in order not to violate its data transfer rules. This allows to send more data to Shopify and to maximize the utilization of allowed number of requests.

Shopify Connector handles data from Perfion in chunks, e.g. it takes 25 products from ECommerce API and then processes all those products one by one in parallel mode. Then it takes another 25 products, etc. This allows to optimize performance, because while data is sent to Shopify, the data from ECommerce API can be retrieved in parallel without wasting any additional time.

The overall data synchronization performance in Shopify Connector, in most cases, depends on Shopify performance. There is no universal figure to show how much time data synchronization will take, because it all depends on how much data is sent to Shopify. The main factors influencing synchronization time:

• How large images are used and how many of them are used in each entity. A simple product update takes under one second in Shopify, but any additional image will slow down update significantly. If large images are used and there are many of them, one should not expect fast synchronization performance.

• How images are transferred to Shopify.

  • If images are defined using URL, then Shopify will need to make additional requests to external server(s) to download those images. That means that if the product has 10 images, then Shopify will have to make another 10 requests before it can complete product creation/update.

  • If images are embedded in request using Base64 encoded inline data (binary data in an ASCII string format), then Shopify will not need to take additional several requests to get data from external server(s), but then the request with such data may take more time to send to Shopify. Moreover, Base64 encoded inline data takes more space than raw binary data, which can be downloaded directly from external image server. According to Wikipedia, Base64 encoded binary data uses about 137% of the original data size.

• If user have chosen to keep the existing variant data in Shopify when updating product (by setting ECommerce Settings KeepExistingVariantData parameter to ‘true’), then each product update will take many extra requests to Shopify to get existing variant data. In this case Connector will need to retrieve all product variants, all variant inventory items and all variant inventory levels from Shopify in order to have all necessary data for update.

• If metafields are used. If product or custom collection metafields are used, then Shopify Connector will need minimum one additional request to Shopify for each entity item to read existing metafields before they could be updated. Moreover, if there are any metafields, which has to be deleted for product or custom collection and Shopify Connector is not set up to keep deleted metafields, then deletion of each metafield will take one request to Shopify. If 10 metafields has to be deleted, then metafield deletion time may take longer than total time of typical product data synchronization.

• Inventory level (variant quantity data for single location) data update. If variant quantity data is used, then each product update will require several requests to be sent to Shopify in order to update all quantities for all variants and for all locations. The number of required requests will be: number_of_variants * (number_of_locations + 1). If there are 5 variants and 5 locations, then Connector will need to send 30 requests to Shopify.

• If product to custom collection relations are used and how many relations for each product there are. If product to custom collection relations are enabled, then product synchronization will use minimum one additional request to Shopify to get information about existing relations. Moreover, any addition of new relation or removal of old relation will take an extra request to Shopify. So if product was related to one category and then it was assigned to two new categories, then updating this information will take three extra requests to Shopify. One request to get information about existing relations, and then two more to add two new relations.

• Number of channels used at the same time. If two or more instances of Shopify Connector are used at the same time to synchronize several Shopify stores, and all are actively synchronizing data, then there could be some delays from ECommerce API. ECommerce API retrieves the same data from Perfion for all channels and handles all requests synchronously. This does not add any significant performance issues, but it depends on data quantity in Perfion and in some cases it could affect overall performance. Multiple channels does not affect so much Shopify performance, because each channel belongs to independent Shopify store and all data transfer limits are based per Shopify store.

The most time is used when the entity item has to be updated in Shopify, because then Shopify Connector must check what data exist in Shopify and handle all changes. Refer to “Product Update” to see how product update works and how many requests it takes to update its data. In order to update products which have a lot of changed data, the procedure could be speedup by replacing products (removing old one and then creating it again as a new one) instead of updating. When product is deleted from Shopify, then Shopify will handle all related data deletion, which is much faster than trying to do this from Shopify Connector. This, however, have implications, that product gets a new Shopify entity ID and one cannot keep the old product data (e.g. deleted metafields or existing variant data cannot be used for update).

Image Handling

Products, variants and custom collections may have images in Shopify. Custom collection and variant may have only one image, while products may have many.

The images can be in .png, .gif or .jpg format.

Product and variant images in Shopify are product images. The only difference is that variant may have only one image, which is defined as one of product’s images. Practically product and variant can share the same image in Shopify. Shopify allows the total amount of 250 product and variant images.

The product and variant images are handled differently in Perfion. Products and variants in Perfion are separate entities and each have their own images. This have some limitations of how images can be organized, but it also offers some benefits.

For example, the same image cannot be shared by product and variant if data is synchronized from Perfion, but one can achieve the same effect by not defining product image at all and instead defining only variant image. Then, the image of the first variant will become a product image, which will be also a variant image.

Since product and variant images are controlled separately in Perfion, then it is possible to have more than one image for each product variant. Shopify Connector allows to utilize this concept. Read more about it in “Using Multiple Images for Variants”.

Product and variant images support metafields in Shopify, but Shopify Connector cannot use metafields, because of how image data is handled by ECommerce API.

Image Data Configuration in ECommerce Mapping

Images for Shopify Connector can be defined by using different ECommerce Output Kinds and multiple parameters can be added to ECommerce Mapping feature to combine images of different types (URL vs embedded) at the same time. Each image related parameter can represent single or multi-value image or file type features, and it is also allowed to use a simple string feature with single or multi-value URL values as a source for images. The example of how one can configure multiple types of images for product is shown in the Figure 3.

Open

Figure 3: Configuring product to have multiple images fro different sources

Figure 3  shows all ECommerce Output Kinds which can be used to define images, but there can be multiple features used for each Output Kind. All feature names, defined in ECommerce Mapping column must be unique. One can also use a literal value with “Field” Output Kind, but then, only one image mapping can be defined in such way.

Another important fact is, that all images, which are defined using URLs must be publically accessible from Shopify. Refer to “Supported Image Types” for more info.

If images are defined using several Output Kinds, then images will be handled in predefined order based on Output Kind:

Supported Image Types

There are two types of images one can import to Shopify:

• By using URL of an image. Only absolute URLs can be used and images must be publically accessible.

• By using Base64 encoded string of an image.

When images are imported to Shopify, it will save all images locally. If image data in Shopify was added by using URL, then Shopify will retrieve data from remote image server where that image is hosted and will save internally in its own image server. Therefore, it is important to make sure that all images sent to Shopify by using URL are publicly accessible. If they are not, then Shopify will not be able to retrieve those images and will issue an error. If image URLs are not publicly accessible, then one can use embedded images instead. 

If images are defined using Base64 encoded inline data (binary data in an ASCII string format), then Shopify will not need to take additional request to get data from external server, but then the request with such data may take more time to send to Shopify. Base64 encoded inline data takes more space than raw binary data which can be downloaded directly from external image server. According to Wikipedia, Base64 encoded binary data size is about 137% of the original data size or to be more precise: Base64 encoded image size = original binary image size * 1.37 + 814 bytes.

Category Image Handling

There can be only one image assigned to custom collection in Shopify, but Perfion allows using multiple images for categories. The category images are selected in the same way as for products and variants using the concept described in “Image Data Configuration in ECommerce Mapping”.

Even though one can use multiple images in Perfion and also set up ECommerce mappings to use multiple images, the Shopify Connector will use only the first found image and will also issue a warning if there are more than one image specified for the category. Therefore, it is recommended to use only one image for categories in Perfion.

Using Multiple Images for Variants

Shopify supports only one image for variant, but Perfion controls variants independently and allows to have multiple images for each variant. Since the ECommerce API does not limit how many images can be transferred for each variant, the Shopify Connector has an extra feature, which allows to send more than one variant image to Shopify.

Images for product and variant are configured the same by using different types of mappings as it is described in “Image Data Configuration in ECommerce Mapping”. In order to allow multiple images for each variant, one has to set variant’s SyncAllImages parameter value to “true”. This will allow Shopify Connector to add all found variant images as product images and they will all be added together, e.g. before images for other variant are added. The Shopify will not “officially” use all variant’s extra images as variant images and only the first image will be assigned to the variant. However, Shopify uses templates for showing products and one can modify the standard template to manage product images differently.  Figure 4 shows how Shopify product images with multiple images for single variant will look like in Shopify store.

Open

Figure 4: Shopify product page with multiple variant images

Product Image Order Determination

Shopify allows specifying product image positions how they will appear in Shopify store, e.g. which product image will be the first, the second, etc. Perfion does not support this feature, because ECommerce API does not support adding extra data to images.

Moreover, the product images in Shopify consist of product and variant images. Perfion controls product and variant images independently, and therefore, there is no simple way to control all product image positions explicitly. The table below shows in which order images will be handled in Shopify Connector and then sent to Shopify.

Image Caching Setup

Ecommerce API uses Perfion API to deliver image content. Before using images with Shopify Connector, one has to setup ECommerce parameters in Perfion and also make sure that image/file caching works correctly.

• In ECommerce Settings add correct image and file services related parameters.

  • ImageServiceUrl and FileServiceUrl parameters. These parameters define URLs to Perfion image and file services. These URLs are used when Perfion API handles images or files (one can use images as attachments with Shopify Connector and attachments are handled via file services, so file service setup is as necessary as image service setup). If URLs are not accessible, then ECommerce API may fail. It is also important to know that if Perfion image or file services are slow, then it will directly affect ECommerce API response times. The ECommerce API responses may also fail if image or file services are not accessible.

  • An example of ImageServiceUrl parameter value: http://localhost:8080/image

  • For more details about these parameters refer to “ECommerce API” manual.

  • ImageServiceUrlParameters, ImageHeight, ImageWidth and OutputFolder parameters. These parameter are not so important for Shopify Connector, but they are related to image and file services and may also be useful to know about. Refer to “ECommerce API” manual for more information.

• Setup image and file services to use caching. This is part of Perfion API setup and has to be done in the server where Perfion API is installed. The caching is enabled by default, but one must make sure that Perfion API service user has necessary write access rights to the Perfion API cache folder. The procedure below shows how to add Perfion API service user with write access rights to cache folder’s properties:

  • Go to the folder, where Perfion API is installed and open Perfion\Cache folder properties (right click with a mouse on the “Cache” folder and choose “Properties”).

  • Example of Perfion API cache folder location: C:\inetpub\wwwroot\PerfionAPI\Perfion\Cache.

    • Open “Security” tab and add a new user.

    • In “Select Users, Computers, Service Accounts, or Groups” window click “Locations…” button.

    • Select the top location and click OK.

    • In “Enter the object names to select” text box enter the user name of Perfion API site application pool, which is set up in IIS (Microsoft Internet Information Services).

    • The format: "IIS Apppool\<Perfion API app pool name>" (without quotes)

  • Example: "IIS Apppool\PerfionAPI_Shopify"

    • Click "Check Names" button. If the name was specified correctly, it should be resolved and if not, then a new "Name Not Found" window will open.

    • Once the user name is specified correctly and found, click OK button.

    • Select added new user from the list and allow full control permissions. Click OK.

  • More information about application pool identities: Application Pool Identities .

• Test if image service cache works.

  • Find any image type feature in Perfion which has an image or create one for test purposes (it can be deleted after the test).

  • Select the image and open its preview. This can be done by right clicking with mouse on image icon (in grid or in item editor) and choosing “View Image…” from context menu.

  • A new “Preview” windows will open, which will show the image and also some extra image parameters. In the bottom left corner there is image GUID. This is image ID and it will be used when testing how cache works.

  • Open a web browser and use the following URL to see if the same image from Perfion is shown in web browser.

  • The format of URL: http://<PerfionApiSiteAddress>/Perfion/image.aspx?id=<image_GUID>

  • Or: <ImageServiceUrl>?id=<image_GUID>

  • Example: http://localhost:8080/Perfion/image.aspx?id=da00e137-9932-4ebf-9a7a-ba6e81a51555

    • Check if a copy of image was saved in the image cache folder. Go to Perfion API image cache folder, e.g. “\Perfion\Cache\Images”. In that folder there should be a copy of the same image, which was shown in the browser. The name of the cached image has the same image GUID in its file name.

    • Example of cached image file name: da00e137-9932-4ebf-9a7a-ba6e81a51555_#_#_#_True_False_#_image2.jpg

Metafield Handling

Metafields are special parameters in Shopify, which can be added to the main Shopify entities to allow users to save extra data which cannot be saved otherwise using standard Shopify parameters for each entity.

Metafields in Shopify are assigned to the main entities such as product, custom collection (category), variant and image, but they are a separate entity and are stored separately from main entities. This affects how they can be added, modified and removed in Shopify and they do not work like normal parameters of each entity. The differences of how metafields are handled in Shopify Connector based on to which entity they belong is shown in the table below.

Refer to “Product Update” to see how metafields are handled for product update. Product creation, custom collection update and custom collection creation uses the same procedures as product update.

Variant metafields are handled differently in synchronization as compared to, for example, product update, because variant data is always recreated in Shopify during new product creation or existing product update.

Metafield Format

Metafields are complex parameters as compared to typical product or custom collection parameters. The typical parameter has a name and a single value or multiple values. Metafields, however, are defined by using 5 parameters, where 4 are mandatory:

• Namespace – Parameter must have min 2 characters, max 20 characters and together with Key parameter defines the unique combined metafield “Key”. Allowed characters are A-Za-z0-9_.~-@#*$. Namespace is case sensitive.

• Key – Parameter must have min 3 characters, max 30 characters and together with Namespace parameter defines the unique combined metafield “Key”. Allowed characters are A-Za-z0-9_.~-@#*$. Key is case sensitive.

• Type – defines metafield value type. The values are shown in the table below.

• Shopify API version 2021-07 introduced breaking changes and the previously supported 2 types were deprecated by replacing them with new types. The table below shows the currently supported types and how they relate to the previously used types. The type values are case insensitive.

• Value – The actual value of metafield. The requirements for a value will depend based on metafield type.

• Description (optional) – This parameter is optional and defines metafield value description.

The unique combined metafield “Key” is required to be unique within given Shopify entity. For example, the product may have several metafields with Key parameter value “Instructions”, but then Namespace parameter value for each must differ. For example, metafields “Installation”-“Instructions” (Namespace-Key) and “Washing”-“Instructions” will be unique.

Metafield Limitations

Perfion ECommerce setup allows defining only simple (name vs. value) parameters and one cannot define complex metafield parameter without specifying it in some special way. In order to support metafields, the special metafield definition was chosen. The metafields are defined in ECommerce setup as normal parameters, but metafield parameter name and value must be formatted following some rules.

Where:

• Metafield – is a fixed keyword which tells to Shopify Connector that it is a metafield. Case insensitive.

• Namespace – is the value of metafield Namespace parameter. Case sensitive.

• Key – is the value of metafield Key parameter. Case sensitive.

• Type – defines the type of the metafield. This should be one of the values form the table with supported metafield types. Case insensitive.

Some examples of metafield names are shown in the table below:

Metafield value may consist of 2 parameters and therefore, it must be specified using 2 EC Mapping values where one mapping is created for the metafield value and the other mapping is created for metafield value description.

To define the metafield value one has to create a normal Field type EC mapping where:

• ECommerce Mapping column should have the literal metafield value which will be used with all entity (category/product/variant) items or the feature name which will specify the metafield value for each entity item independently. Refer to the Figure 5 to see an example.

• ECommerce Output Kind column value should be set to “Field”.

• ECommerce To column value should define the metafield name.

• ECommerce Entity column should define the entity name to which the metafield belongs.

To define the metafield value description one has to add another mapping. The metafield value description is an optional field, so this has to be done only if metafield value description has to be used. The definition is very similar to how the metafield value was defined. There are 2 differences:

• ECommerce Output Kind column value should be set to “Attribute”.

• ECommerce To column value should be set to the same name as for the metafield value mapping, but with added suffix “@description”. Refer to the Figure 5 to see an example.

Metafield Synchronization Process

Metafield synchronization with Shopify depends on the entity they belong to.

Variant is not tracked in Perfion independently and therefore variant data (including metafields) is always replaced with a new variant data even if the data have not changed. Even though the variant data is always recreated, the Perfion Connector has a feature which allows keeping the existing variant data in Shopify even if it does not exist in Perfion anymore. This allows any 3^rd party tools to update variant data which is not available in Perfion. To enable this feature one must enable EC Settings parameter KeepExistingVariantData. In addition, one must set the product parameter SyncKeepDeletedMetafields to “true” in order to keep the metafields in Shopify which are not defined in Perfion. If the feature which allows to keep the current variant data is activated, then the variant data will still be recreated in Shopify, but Shopify Connector will first read the data from Shopify, merge it with the data available in Perfion and create that data again in Shopify.

Product and custom collection metafields are synchronized in the same way using this process:

• Before product or custom collection synchronization all metafields for each entity are retrieved from Shopify.

• Metafield data is updated in Shopify together with the data of their parent entity, therefore, the data received from Shopify is used to determine which metafields has to be updated, which has to be added and which has to be deleted.

• New and existing metafields data are added to their parent entity data and are synchronized all at once together with their parent entity.

• Deleted metafields must be handled independently in Shopify. That means that all metafields, which has to be deleted, will be deleted one by one. This process, however, is optional and one can chose to skip it by setting SyncKeepDeletedMetafields parameter to “true” value for product and/or category.

Product special metafields can be handled in two different ways in Shopify. It allows to define these metafields directly as fixed product’s data, but these metafields technically are no different than any other product metafield and can also be updated independently. Shopify Connector will synchronize these special metafields together with all other metafields and does not support the other method where data can be defined directly.

Shopify Inventory System

Shopify have recently changed how product variants are handled in their stores and the newest Shopify Connector version supports it. The new system is significantly more complicated than the one before and users should be aware how to properly use it.

The old product variants had a single quantity parameter and that value was valid for any locations where the variant was sold. The old system was like a physical store would have, e.g. in that physical store the same variant will have only one quantity definition. For example, there are 10 “cups” for sale in store “A”.

The new inventory system, however, introduced a more complex configuration, where product variant got some dimensions and it is not any longer treated as a variant in one physical store, but rather as a global product variant which can be sold in multiple stores. So now the single product variant is defined as: 5 “cups” for sale in store “A”, 3 “cups” for sale in store “B” and 2 “cups” for sale in store “C”. The definition of the variant is still the same, but now it may have multiple quantities which belong to different Shopify locations. Even though the new system is more complicated, the Shopify users can still continue to use only a single location like before. For example, the product variant in Perfion is still defined almost in the same way as before, but in the new system one may define multiple quantities instead of 1.

Even though the difference from user perspective is not so big, the structural changes in Shopify how the data is handled become significantly more complicated. The new inventory system in Shopify looks like shown in Figure 6.

As shown in Figure 6, the “Product Variant” block shows how the old product variant looked like in the old system and product quantity was a single parameter defined directly in “Product Variant” block. In the new system each product variant have got a new entity called “InventoryItem” and each inventory item could be related to multiple locations via so called “InventoryLevel” entity. The quantities in the new system are now saved in “InventoryLevel” block.

From Perfion perspective, when one has to properly configure a product variant for synchronization, the following has to be done:

• Any locations used to control product variant quantities must be defined in ECommerce Settings mappings as reference locations (“RefLoc_*” parameters).

• Any parameters related to product variant, which in new inventory system were expanded into multiple entities, will still be configured as before in ECommerce Variant mappings.

• ECommerce Variant mappings configuration have slightly changed for product variant quantity definition. Before there was a single quantity field and now there can be more than 1. In order to be able to map all data correctly, Perfion uses a special quantity assignment format. The parameter name for quantity definition has the following format - “Quantity_<reference_location>”. Here “reference_location” is the name of reference location as it is specified in ECommerce Settings mappings, e.g. using “RefLoc_*” parameters.

• Even though the configuration from Perfion side is rather simple, one still has to understand how Shopify inventory system works in order to define data correctly. For example, Shopify uses fulfillment services and if one enables such service, then product variant will be able to handle only a single quantity value. If in such case there will be more than a single quantity defined in variant mappings, then Shopify will not allow to update variants. It is strongly recommended to learn how the new inventory system works in order to understand the following:

  • What is fulfillment service and how inventory system works when such service is enabled

  • How normal locations differs from fulfillment locations

  • How switching between normal locations and fulfillment locations work

• Most of ECommerce Variant mapping parameters have names which refer to parameters used by Shopify in Product Variant, Inventory Item and Inventory Level entities. In case there are some questions how to set those parameters or what they mean, please refer to Shopify documentation, e.g. “https://help.shopify.com/en/api/reference/products/product-variant ” and “https://help.shopify.com/en/api/reference/inventory ”.

To read more about Shopify inventory system please refer to “InventoryLevel - REST ”.