Overview

Applicable versions

The following Magento version has currently been tested and are approved for this version of the Perfion Connector:

• Magento Open Source 2.4.4, 2.4.5, 2.4.7

• Magento Commerce / Magento Commerce Cloud 2.4.4, 2.4.5, 2.4.7

The basics of what this connector does

The connector (depicted in a very simplified way in Figure 1) works as follows:

Open

Figure 1: Magento Connected to Perfion

Magento asks Perfion for data based on the configured channels. Data can be attributes, categories, and products. Perfion will provide that data in the way that it has been mapped in Perfion for the given Ecommerce Channel. The Perfion Connector will process the received data and create the attributes, categories, and products.

Installation

Prerequisites

Before installing the connector make sure, you have the following prerequisites set up:

1. You have a running Magento 2 installation with the latest version of PHP 8.1 or PHP 8.2 running. Please consult official Magento 2 system requirements for your version:

   1. Magento 2.3: https://devdocs.magento.com/guides/v2.3/install-gde/system-requirements-tech.html

   2. Magento 2.4: https://devdocs.magento.com/guides/v2.4/install-gde/system-requirements.html

2. Composer (Composer ) installed and executable on the server.

3. You have a Perfion Database v. 4.6 (or newer).

4. You have a Perfion API set up on IIS Perfion running at least 4.6 (or newer). The Perfion API endpoint must be accessible from the Magento server.

   1. Please note: If a firewall protects your Perfion instance in your infrastructure setup, please put all Magento server IPs in the allow list of your firewall.

5. Your Perfion license must include licenses for the Perfion Ecommerce API with at least one Ecommerce Channel and the Magento 2 connector.

Installing the Ecommerce API in Perfion

Auto create features and configuration for the ECommerce API

The first step to configure the ECommerce API is to set features and feature values, allowing a starting point. It is not a problem if you have already created these features using an older version of Perfion. In this case, you can simply skip this section.

This section takes you through the few steps needed to auto create features and feature values needed for the ECommerce API to work.

• Start Perfion and make sure it is version 4.6 or later.

• Until version 5.1: In Perfion main window, go to “Administration” and “Settings”. In the new window you can now see the tab “Integration”. Here simple click “Install” for “Install ECommerce API Features and Data” as shown in here:

Open

Figure 2: Installing the ECommerce API in Perfion

Answering “Yes” in the “Are you sure”-dialog will start the process and it will take about 5 to 10 seconds, depending on your hardware, to complete. A message-dialog will show up when done.

To see the most important of the features that have been added, go to “Feature Data”, then expand the “String”-features node and finally expand the “ECommerce Features”-node. Here you will find seven selectable features where some needs to be filled out by you and the rest can and should be left as is.

Below is an overview on what each of the ECommerce-features on the left-hand side contains:

• ECommerce Channel: Here you create all the channels you needed. Roughly speaking, you will need a channel per output channel. For the Magento Connector you are likely to need one channel per store .

• ECommerce Entity: Here are a list of all entities supported currently by the ECommerce API: Site, Category, Product, Variant and Language Deviation. This is used internally by the connector and should not be modified.

• ECommerce Mapping: Here you find all mappings from Perfion to some Channel (that is, to Magento). A mapping is a way of expressing that “X in Perfion is called Y in Magento”. All mappings can be edited by clicking this node.  This is where the main part of the configuration of the connector will take place, and we will do that in section 3.1.5.

• ECommerce Output Kind: Here is a list of all the Output Kinds, that is, ways of expressing how the “Y” from previous item should be output knowing “X”.

• ECommerce Settings: Here you find the settings needed to set up the Connector.

• ECommerce System: Here you will find a list of ECommerce Systems to pick from. Version 4.5.52 auto creates four ECommerce Systems: Shopify, Ucommerce, Sana and Magento, since these are the systems recognized by Perfion and the systems for which Perfion provides a connector. If you are making your own output from the ECommerce API, it makes good sense to add a new ECommerce System to this list.
  Note: From version 4.5.52, when accessing the ECommerce API for data belonging to one of the known channels, it is checked that the Perfion license does include a license to connector used.

• ECommerce Type: This is just a selectable feature that you will need if you want to use support for variants. The two values here “Product” and “Variant” should not be modified in any way. Support for variants is described in section 4.2.

Installing the connector in Magento

An installation via Composer is the only valid way to properly install and manage the connector in a Magento 2 instance.

For installation, please login to your server via SSH on your command line interface (CLI) and navigate to the directory with the Magento installation.

Installing the connector on Magento Open Source

Require necessary packages in the project and install the packages

composer require perfion/perfion-magento-connector
php bin/magento setup:upgrade

Enable the connector modules and run the database migrations

php bin/magento module:enable Perfion_ObjectDataImporter
php bin/magento module:enable Perfion_Connector
php bin/magento setup:upgrade

Installing the connector on Magento Commerce / Magento Cloud

Require necessary packages in the project and install the packages

composer require perfion/perfion-magento-connector
composer require perfion/perfion-objectdata-importer-magento2-commerce
php bin/magento setup:upgrade

Enable the connector modules and run the database migrations

php bin/magento module:enable Perfion_Connector
php bin/magento module:enable Perfion_ObjectDataImporterCommerce
php bin/magento setup:upgrade

Updating the connector in Magento

Updating the connector in Magento 2 only requires a straightforward command after you’ve logged onto the server and navigated to the path of your Magento installation.

Updating the connector on Magento Open Source

composer update perfion/perfion-magento-connector
php bin/magento setup:upgrade

Updating the connector on Magento Commerce / Magento Commerce Cloud

composer update perfion/perfion-magento-connector
composer update perfion/perfion-objectdata-importer-magento2-commerce
php bin/magento setup:upgrade

Configuring the connector

This chapter takes you through all configuration needed for setting up the connector for the first time. First section will take you through the necessary steps in the Magento backend interface. Next, in section 3.1, will take you through the steps needed in Perfion.

When you have completed this chapter, you have a very basic setup of Magento 2 with the Perfion Connector.

Configuration in Perfion

The next sections will take you through all the necessary steps to set up an entire configuration for a connector for a website in Perfion.

Set up a website feature for your Magento instance

To get started, you need to create a dedicated feature for your Magento store.

You simply can name the new feature “Magento”. But if you plan to connect multiple Magento stores to your Magento store, it is recommended to name the feature based on the name of your store.

In Perfion go to “Features” - “String” and create a new feature as shown in Figure 3:

After that, click on the three dots at the main menu at the bottom and open the section designer menu. In the section “Website” add a new option by clicking on the grey cell above “Sana Commerce” and create a new option either via right-click and “Add New” or by hitting CTRL+N on your keyboard.

You can then add a name for the section, select the control type “Categorizer”, select the new feature you created in the step before in Figure 3 and click “OK” to save the changes. Your section designer should now look like shown in Figure 4:

Set up your category structure

While you could just use the default category structure for the Magento Connector, it is highly recommended to create your own category structure. It is required to set it up like this if you plan to connect your Perfion instance to more than one Magento store.

The following example in Figure 5 shows the default category structure on the left-hand side at the top and a custom category structure on the left-hand side at the bottom.

After you’ve set up the category structure, you can simply add the products via drag and drop from the right-hand side to the category on the left-hand side.

Create Ecommerce Channel

A Channel is the ECommerce API equivalence of a store (aka storeview) in Magento. Please go to “Feature Data” - “String” - “ECommerce Features” - “ECommerce Channel” in Perfion and create a new channel for your Magento store as shown in Figure 6:

Create Ecommerce Search

For an ecommerce channel to only retrieve the products that are assigned to a specific store, you need to configure an ecommerce search beforehand.

The following example only retrieves the products from the new feature (Magento) that we just created:

Create Ecommerce Mappings

The following mappings must be configured:

• Language Deviation

• Category

• Product

Important information for creating the Ecommerce Mapping:

• Magento has fixed rules about the attribute code names, which will be created automatically based on the Perfion features. When you specify the field names in the Ecommerce Mapping, please keep the following rules in mind:

  • Max. 60 characters

  • Start with a letter

  • Only contain letters and numbers

  • Lowercase (Magento connector takes care of that in case you forget)

  • No whitespace (Magento connector takes care of that in case you forget)

• These rules are essential especially if you’re using the Wildcard selector in the Perfion Ecommerce Mapping, where Perfion will use the Feature name as the field name.

• If a Feature/Field name doesn’t match the criteria mentioned above, the connector will not create the attribute.

• Magento has a fixed set of default attributes that can be used in the Ecommerce Mapping. If you plan to do that, please make sure that the data types match.

  • Example: In Magento, the “color” attribute is a single select attribute. If you have the same attribute in Perfion but as a “multi-value” feature, the data types won’t match, and there will be a conflict during import. In this case, do not map your “color” feature to the “color” attribute of Magento but specify a custom field name (e.g. customername_color).

Language Deviation Mapping

The Language Deviation mapping is used to store the product data for a specific store view in Magento. Please specify all Perfion-internal language codes that you want to use in Magento to the specific Magento store-view codes.

Please note:

• You must specify the store view code in the Ecommerce Mapping exactly the same as you created the store view codes in Magento.

• If you, for example, have multiple store views in Magento that should use English language, you can map one language in Perfion to multiple store views. Please make sure to always map you’re default language to “admin” so that Magento has values on global scope as well.

• If you only use one language in Perfion, please specify the mapping for “admin”, which represents the global scope of Magento.

Category Mapping

In the Ecommerce Mapping, please create a new “Context” for Ecommerce Entity “Category” and add features/fields to the mapping.

Please note that as “feature” name you must enter the name of the new feature you just created for the store. In our example it would be “Magento”.

Please make sure that you always map the following mandatory fields:

Product Mapping

In the Ecommerce Mapping, please create a new “Context” for Ecommerce Entity “Product” and add features/fields to the mapping.

As feature name you please specify “Product/WebDetail/NAME_OF_YOUR_ECOMMERCE_SERACH” to receive only products in the mapping that are assigned to the categories for this store.

Mandatory fields:

For fields that are not defined in the mandatory fields tables, it is important that the mapping names in the TO field, match the FROM names. You are welcome to lowercase your TO names, but nothing new should be added to the feature names.

To send more product information to your Magento store, simply add more fields to the ecommerce mapping. The Magento Connector will automatically create the attributes for the fields that are not known by default.

Optional fields:

Set up Stock Management in Perfion

This section is optional. You might want to provide the stock information for products through Perfion, in case stock management for your products does not happen in a 3^rd-Party system (e.g. ERP system).

The connector currently supports the following fields:

If you’re using Magento Multi Source Inventory(https://docs.magento.com/user-guide/catalog/inventory-management.html) to manage multiple warehouses, stores, pickup locations, drop shippers, etc. you must provide one more field:

Configuration in Magento

After the connector has been installed, it needs to be configured to reach the Perfion API. This “connection to Perfion” is the only thing you need to configure in Magento. The rest of the configuration will be done in Perfion.

Set up channels in Magento

In Magento Admin UI, go to Perfion à Channels. You will see an empty list of channels as shown in Figure 10:

Simply click on the button “Add channel”. This will bring you to the channel creation form shown in Figure 11:

After making/entering the necessary configuration settings, you can click “Save”, and the channel is ready for receiving data. You can now repeat this process to create a new channel for the other data type as well.

Please create the channels in the following order:

1. Attribute

2. Category

3. Product

Setting up cronjobs

By default, the connector does not automatically fetch data from your Perfion instance. You can always fetch and process the data manually. If you want to automate the import of the Perfion data, you need to set up cronjobs to fetch and process the data automatically.

Setting up the cronjobs depends on how often you change product data in Perfion and how quickly you need the product data changes imported in your Magento installation.

You need to set up cronjobs for the following commands:

Retrieve and import data immediately in one command

The easiest way to setup your automated imports is to use the command:

php path/to/bin/magento perfion:run –channel CODE

This enables you to fetch and process in one go.

Fetch data from Perfion

php /path/to/bin/magento perfion:fetch CODE > /dev/null 2>&1

The placeholder CODE reflects the auto-generated code for a given channel and data type combination, which the Perfion connector automatically generates. You can get the value from the channel list in the admin UI, as shown in Figure 12.

Process received data

The fetch command only fetches the XML from Perfion and does not directly import them. To process the XML and import the data, you need to run the following command:

php /path/to/bin/magento perfion:process > /dev/null 2>&1

Additional remarks:

• Please note that attributes import must run before the category/product import.

• Please note that categories always need to be imported before the product import. Otherwise, the automatic category assignment won’t work.

• It is recommended that you run the index process of Magento after the import as well.

Purge old batches

The connector stores the received data from Perfion in the database. To prevent the database from filling up and potentially having no more disk space available, you need to add a cronjob that purges the old batches from the database.

0 0 * * * php bin/magento perfion:purge > /dev/null 2>&1

By default, the connector deletes all batches that are older than 7 days. If you want to change the number of days you want to keep in the database, you can add the optional parameter more-than-days.

Example: Delete old batches older than 14 days

0 0 * * * php bin/magento perfion:purge --more-than-days 14 > /dev/null 2>&1

Example 1: Full import every hour

Running a full import every hour requires a very simple cronjob configuration:

# Purge
0 0 * * * php bin/magento perfion:purge > /dev/null 2>&1

# Full import every hour
0 * * * * php /path/to/bin/magento perfion:fetch channel_attribute > /dev/null 2>&1
5 * * * * php /path/to/bin/magento perfion:fetch channel_category > /dev/null 2>&1
10 * * * * php /path/to/bin/magento perfion:fetch channel_product > /dev/null 2>&1
15 * * * * php /path/to/bin/magento perfion:process > /dev/null 2>&1

Example 2: Full import at night, incremental/delta import during the day

If you have a large product catalogue you might not want to do a full load every time but rather incremental delta updates during daytime and a full load at night.

In this example the full load is always done at midnight and the incremental updates every full hour between 08:00 and 20:00.

Cronjob configuration:

# Purge
0 0 * * * php bin/magento perfion:purge > /dev/null 2>&1

# Full import
0 0 * * * php /path/to/bin/magento perfion:fetch channel_attribute > /dev/null 2>&1
5 0 * * * php /path/to/bin/magento perfion:fetch channel_category > /dev/null 2>&1
10 0 * * * php /path/to/bin/magento perfion:fetch channel_product > /dev/null 2>&1
15 0 * * * php /path/to/bin/magento perfion:process > /dev/null 2>&1

# Delta import
0 8-20 * * * php /path/to/bin/magento perfion:fetch channel_category_incremental > /dev/null 2>&1
5 8-20 * * * php /path/to/bin/magento perfion:fetch channel_product_incremental > /dev/null 2>&1
10 8-20 * * * php /path/to/bin/magento perfion:process > /dev/null 2>&1

Optional: Setting up message queue consumer

If you want to fetch new data for a channel manually via the admin UI instead of (or in addition to) a cronjob, you need to start a message queue consumer. Without a running consumer, the requests will not be handled, and no data will be fetched.

The following command needs to be run to start a message queue consumer:

php /path/to/bin/magento queue:consumers:start perfion.channel.fetch

Please note that it is not recommended – since the process needs to be long running and a CLI usually has a timeout – to run this process manually but rather use a tool like Supervisor(Supervisor: A Process Control System — Supervisor 4.2.5 documentation) to create a permanent and long-running process.

Batch storage location

XML Batches from Perfion are downloaded to your Magento directory, in the directory var/batches/perfion.

Please note, that the user running the connector (either manually or via CRON), should have read and write access to this directory.

Additional mappings

This section describes some additional mappings.

Map media gallery for products

If you want to add additional product images to the media gallery in Magento, you need to create a new feature in Perfion where you can select multiple images at once. In Figure 13 you can see an example:

After you’ve created the new feature, you can go to the Ecommerce Mapping and map the new feature to the field name “gallery”, as shown in Figure 14.

Setting up variants

The Magento 2 connector has support for Variants already built in. Please consult the ECommerce API manual for properly setting up variants in Perfion before.

In the Ecommerce Mapping you need to map the following additional features for context “Product”:

• Feature: ECommerceType
  ECommerce Output Kind: ECommerceType

• Feature: ECommerceVariantDimensions
  ECommerceOutputKind: VariantDimension
  Field: VariantDimension

After that you just need to create an ECommerce Mapping for context “Variant” as well and add all features you want to add to your Magento store. The following image in Figure 15 shows a complete example:

Creating multiple shops

The Magento 2 connector has support for multiple shops in one Magento installation already built in. This feature is part of the in the ECommerce API.

To connect multiple shops, you simply need to repeat all steps of previous chapters.

Please make sure that you define the Magento website code in the Ecommerce Mapping as well. You can find the correct Magento website code in the Magento admin UI if you go to “Stores” - “All Stores”, as shown in Figure 16.

In addition to that, it is highly recommended to create a new Root Category in Magento and assign it to a new store. In doing this you ensure that every Magento instance has their own category structure. Otherwise, they will be mixed.

Other settings

Attribute cleanup

In addition to cleaning up unused attributes, you have the option to also delete NULL valued attributes in the database. This might sometimes be useful if you have data integrity issues.

You can enable this setting in: Stores->Configuration->Services->Perfion, where you can select “Delete options without a value”.

Troubleshooting appendix

In this section you will find answers to common troubleshooting issues.

Database settings, and batch retention

If you are working with a database configured with the feature 'innodb_file_per_table' disabled, it is very important that you either enable this feature, or make sure to setup a cron job, that cleans up the perfion batch log table from the get go (see the bin/magento perfion:purge cli command mentioned earlier in this document) . When the setting is disabled, disk space allocated for the batch logs cannot be reclaimed which, if let be, can cause the disk space usage to expand significantly.

If you have already encountered this problem, you can change the setting on an already existing database with the following sql commands. This will also enable the feature for all newly created databases) :

SET GLOBAL innodb_file_per_table = 1;

Genstart serveren

ALTER TABLE perfion_channel_batch FORCE;

OPTIMIZE perfion_channel_batch; 

Because of the batch log tables potentially large size, sometimes it is easiest to clear the table manually, if you did not configure the above settings before installation.

Using the Perfion API Query analyzer to test the variant setup

A useful tool when configuring your Perfion integration is the Perfion API Query Analyzer.

In the query analyzer, you can right click the text window, and select the Validate query. In the XML text that is inserted, you can replace value=’Channel A’ to Value=’MagentoStore’, and click “Execute”. This will give you hints to mistakes in your Ecommerce configuration, or your Ecommerce Settings.

Example output:

Mapping variants

• Please note, when configuring the EcommerceVariantDimension in your product grid, the feature name you fill in, is case sensitive, which means it must match the “Perfion Unique Name” of the feature exactly.

Mapping to multiple channels, and multiple languages

When mapping multiple channels, make sure that you do the following:

On your primary channel, make sure you map the Language Deviation primary language to admin.

On your secondary channels, you need to map the website default language to admin as well as the store view language to the storeview name.

Multi select attributes

When you use multiple languages, make sure that your multi select attributes are translated as well, since attributes cannot be created with an empty value.

Localizable selectable features

Importing of localizable selectable features requires to add “Admin” as a language deviation (Use the language that Magento was installed with).

Admin language deviation is used to determine the identifier for the individual languages.

Magento installation language is also the “admin” language, therefore it should be added as a language deviation in ecommerce mapping in Perfion.

Integration timeouts

If you encounter integration timeouts, often identified with the message “error fetching http headers”, you can do the following to fine tune how long a timeout is allowed.

When running the perfion integration commands either manually or using CRON jobs, you can specify a PHP timeout, by prepending the command:

php -d default_socket_timeout=<SECONDS> <COMMAND>, where <SECONDS> is a number representing the amount of seconds before a request is dropped (the number -1 represents no timeout), and <COMMAND> is the perfion command you wish to execute.

Example for fetching and processing items for all channels: php -d default_socket_timeout=-1 bin/magento perfion:run

PHP Insufficient memory issues

Depending on the server setup regarding the maximum PHP memory consumption, you might need to specify a higher memory limit, when executing commands either manually or in CRON jobs.

To execute a command with a specific memory limit, you can prepend your commands with:

php -d memory_limit=<MAX_LIMIT_IN_BYTES> <COMMAND>, where <MAX_LIMIT_IN_BYTES> is the maximum allowed memory consumption in bytes (-1 is unlimited), and <COMMAND> is the perfion command you wish to execute.

Example for fetching and processing items for all channels: php -d memory_limit=-1 bin/magento perfion:run

You can specify multiple settings when executing commands. Eg. if you need to set both timeout limit, and memory limit, it looks as follows:

php -d default_socket_timeout=-1 -d memory_limit=-1 bin/magento perfion:run

Attribute sets

If you want to use attribute sets in Magento, you will need to build them, and group your attributes manually in Magento, however, you can chose which attribute set products belong to, from Perfion. We suggest that you do an initial transfer of products to Magento, which will ensure, that all attributes mapped in your Ecommerce mapping are created in Magento, and are ready to be assigned to attribute set groups.

When this is done, you can create your attribute sets, and group the attributes as you wish in the Magento editor.

When this is done, you can create the attribute­_set mapping in Perfions Ecommerce Mapping section, to select a field that controls the attribute set inside Perfion.

An idea could be to:

• Create a selectable string feature, and add it to the product configuration in Perfion

• Map the selectable feature in the Ecommerce mappings FROM column, to the attribute_set in the TO column. (Type: Field)

• Create a selectable value, and select it, on your products new attribute set feature.

Please note that products with no attribute set, will be assigned the Default attribute set in Magento..

Attribute set groupings will not be modified on imports, however new attributes from Perfion, will automatically be added to the Perfion grouping in Magento.

Selectable string feature

Mapping

Selectable on product

Text field length in Magento limited to 255 characters

If you are using Text features in Perfion, and you use a text length of more than 255 characters, it is not possible to transfer these values to Magento, since Magento has a standard limit of attribute text lengths of 255 characters. If you wish to use this functionality, you have the possibility to modify your database column length for catalog_product_entity_varchar, to support this.

You will get an attribute error if you try to use this functionality without first configuring your database manually to support it.