Feature Data Import

In this section we will go into details how to import data into features, e.g. how to specify import data.

The import allows adding new feature values, updating existing values and deleting existing values.

The import data is table like data and can be specified as import data headers and import data values.

Data Mapping

One can define data for many features during import, which will be imported simultaneously, but only one feature (key feature) can be used to map the existing data in Perfion with the new incoming import data. For example, if the data has to be updated, then existing data should be found before the update can be performed. 

The key feature must always be the first feature in the import data and almost any feature can become a key feature. Moreover, there are various conditions and special keywords reserved by Perfion, which can influence how the key feature values are searched. The following sections will explain how key feature can be defined and how its use can change based on special keyword usage in the import data.

Key Feature

The key feature is a feature, which will be used to find the existing data in Perfion in order to relate it to the incoming import data. The key feature is always the first feature (import data column) in the import file. The key feature values in the import data will define if the import data will update values or if it will create new values. If the key feature value will not be found in Perfion, then new data will be created, and if the data will be found, then the update will be performed. This search is performed for each import data row (import feature item).

The following column types can be used as key feature:

• The import feature’s base value.

• The import feature’s item ID.

• The base value of the feature, which belongs to the import feature.

Key feature values cannot be empty and if ID is used for reference, then all referenced IDs must exist in Perfion.

Only features with String or Link base feature type can be used as key feature.

Key feature values in the import data must be unique. Perfion can only import items with unique key feature values. It will by default ignore any import data rows with duplicate key feature values. The only exception is when user sets the feature data “Import Method” parameter to “Keep the first occurrence of records with duplicate keys” value. Then Perfion will still import only items with unique key values, but it will take only the first item with duplicate key and will ignore all other items with the same key value (e.g. duplicates). It is recommended to never use items in import with duplicate key values, because the import results may be different than they are expected to be.

It is important that feature data values in Perfion are unique if that feature is used as a key feature. If feature in Perfion have duplicate values, then only the first found feature item value can be used for the update. All other feature values will be ignored.

Stage Feature

Stage feature is a special feature, which can be setup for any Perfion feature in order to control feature’s lifecycle.

The stage feature adds an extra dimension to that feature in a sense that for each existing feature item there can be multiple lifecycle versions. For example, if one has “development” environment and “production” environment, then each feature with stage setup may have two versions for the same feature item data. If there is one product, e.g. “Product A” and there are two different environments, then there will exist two “Product A” products. One of them will be part of stage “development” and another will be part of stage “production”.

From the import point of view, if the stage feature is set up for the import feature, then the import data should include an extra column, which will define to which stage environment the feature item will be related to. If stage feature is set up, then key feature and stage feature together will be used as a combined key feature to search for existing data.

Stage feature cannot be used as key feature, e.g. it cannot be the first column in the import data and the stage feature values cannot be empty. Moreover, the stage feature cannot be localizable, so language codes cannot be used to define stage feature column name.

Note that if the import feature has data before setting up the stage feature for the import feature, then that data will not have any stage values defined. In order to import to a feature, which has stage feature set up, all import feature values must have stage value set, e.g. it cannot be blank (empty). Perfion will allow importing using empty stage feature values, but it may influence data integrity and data can be misinterpreted. Therefore we recommend to always use stage feature values when importing data to Perfion.

Parent feature

In order to allow modification of item hierarchy structure, a special “parent” feature is used. Refer to Item Hierarchy for more information about hierarchical data.

Parent feature’s name is referred to using “_parent” keyword (column name). Parent feature is not a real feature, but it is used to refer to the key feature and it has the same properties as the key feature, but in reality it is just a reference used to handle hierarchical data. The “_parent” column values refers directly to the key feature column values in the import data and can be described having the same column type as for the key feature column. For example, if the key feature will be defined as the import feature’s base value, e.g. using “_value_en”, then “_parent” column will also refer to the same type value, however it will allow specifying value of another import feature item.

The “_parent” column functions like the key feature, but independently. If the key feature will be used to search for import feature item for update, then the “_parent” feature will search using the same data, but using different value. The purpose for Parent feature is to link the import feature item defined by the key feature value to another import feature item, which will become a parent for the current feature item.

Note that the “_parent” column cannot be used if the key feature has Inheritable data property or if import feature does not have Hierarchical (Allow Hierarchy) data property.

In case the Parent feature is used when import feature has Stage feature, then the same principles apply to this column as for the key feature. The actual search value will be a combined key consisting of the value defined in the “_parent” column and the value defined in the Stage feature column.

The “_parent” column will define the higher hierarchy level item (the parent item or virtual item) the current item has to belong to. For example, in Figure 4 one can see a hierarchical structure. Such structure can be built by using the following import data:

The empty “_parent” column value will remove any hierarchy, e.g. the item will become the top hierarchy level item. The same will happen if the value defined in “_parent” column will not be found.

In order to use “_parent” column, the import feature must have Selectable and Hierarchical (Allow Hierarchy) data properties. In the example from the table above, the import feature is “Product”, which is a base feature. The key feature is also “Product” feature (import feature itself) and its base values are used as keys.

If the import feature items have hierarchical structure, but the import data does not have “_parent” column specified, then hierarchical structure of existing items will not be changed and after the import will remain the same as it was before the import. 

 The “_parent” column name cannot be used with language suffix. If key value is localizable, the “_parent” column name will be used without language suffix.

If the value defined in “_parent” column during item update cannot be found, then that import feature item will lose any relations it had, e.g. it will become top hierarchical level item, just like if it has not been defined at all. The actual values will also depend on which import parameter “Import Method” option was chosen.

Data Column Types

To specify import data columns one has to use either a normal feature name, a special feature name (e.g. Stage feature and/or Parent feature) and also normal feature names with custom format, which allows defining localizable data or data referenced in some special way, e.g. by using ID instead of data value.

The column names are case insensitive.

By data column types the import data columns can be classified like this:

• Import feature’s base value column

• Import feature’s item ID column

• Stage feature column

• Parent feature column

• Normal feature column

• Binary feature column

• Selectable feature item ID column

• Localizable column

• Comment column

Import Feature’s Base Value

Feature’s base value is the feature’s own value. To refer to this value, a special keyword is used as a column name: “_value”.

“_value” column name can be localizable. For example, if the import feature has Localizable data property, then “_value” column name must be formatted differently, e.g. by using a language suffix.

Import Feature’s Item ID

Import feature’s values are called items and each item has ID. One can use this ID to import data directly to chosen items. However, only import feature’s item IDs can be used and only if used as key feature, e.g. if one uses this column in the import file, then it has to be the first column.  Moreover, all item IDs must exist in Perfion if they are defined in the import data.

One cannot use item IDs to create new IDs when creating new items. Perfion will control ID creation for import feature items and one can only refer to existing IDs. To create new items one must use the import feature’s base value instead. Only the update of existing items can be performed if the key feature is used with base feature IDs.

To use the import feature item IDs as the key feature one has to use a special reserved keyword “_ID” as a column name. ID column name is not localizable, therefore one cannot use language suffix with ID column name.

The item ID column will not be used together with Stage column as the key feature. Item ID refers to exact item ID, which is unique in Perfion and Stage value has no meaning in this case, and will not be as part of combined key as it was a case with the import feature’s base value.

Stage Feature

Stage feature column can be used in two different ways depending on how the import feature is set up:

• When Stage feature is used for lifecycle management for the import feature. In this case Stage feature has a special meaning and it will be used as a secondary key feature value in the import to find the existing data or when adding a new data. The stage feature column can be defined using the original stage feature name, keywords “stage” or keyword “_stage”. The stage column cannot be the first column in the import data.

• When Stage feature is not used for lifecycle management for the import feature. In this case the Stage feature will be a simple feature just like any other feature.

Parent Feature

Parent feature column is always referred by using “_parent” column name and it is used only for hierarchical data creation or modification and only if the import feature has Hierarchical (Allow Hierarchy) and Selectable data properties.

Normal Feature

Normal feature in this context is the feature that belongs to the import feature. It can be referred in the import data by using its base value and by using feature’s name as the import data column name.

The column name is used in the same was as with the import feature’s base value column. The only difference is that one has to use the feature’s unique name as a column name instead of “_value” keyword.

Normal feature can be localizable and if it is, then one has to use localization suffix with the feature name.

The normal feature can also be used as the key feature.

Binary Feature

Binary features are defined in the import data file in the same way as normal features. One can refer to binary feature using its name, which can also be localizable.

There is an exception that binary feature base value cannot be used as a key feature. The binary feature can be used as a key feature only if referenced by ID, but only if the binary feature has Selectable data property and where ID is selectable feature’s default value ID (e.g. selectable feature item ID). Moreover, import using ID will work only if importing to binary feature itself, e.g. if binary feature is selected as the import feature.

Selectable Feature Item ID

Normal features in the import cannot be referred to by using IDs, only the import feature can be referred by ID. However, if the normal feature has Selectable data property, then it can also be referred by using ID of its default values (items), e.g. by using selectable feature’s item IDs. Selectable feature may have several items as its own data, where user can chose one of those values (or several values if feature have Multi-value data property) as the actual selectable feature value. In this case the actual value can be selected by using the value of an item, or by using ID of an item.

Selectable feature default value (item) ID is shown in Figure 5. To update selectable feature’s value one will need to reference it by using selectable feature name and then add “_ID” suffix at the end of it. If selectable feature’s name is “SelectableFeature”, then it has to be defined as “SelectableFeature_ID” column name in the import data.

Selectable feature item ID column cannot be used as the key feature column.

Open

Figure 5: Selectable feature ID and localization

Localizable

If import feature has Localizable data property, then in the import data it has to be referred using a special format.

To define a localizable column name one has to add a language code suffix to the column name.

Some examples and comparison of localizable and non-localizable column names are shown in the table below.

If feature has localizable feature and there are multiple languages defined in Perfion, then one can import to the same feature more than once in the same import data file. For example, one can import English values in one data column and Danish values for the same feature in another data column. This is only valid for explicit (not selectable) features. In the table below it is shown how one can import multiple values in different languages for the same feature.

If feature is localizable and have more than one language, but not all of those languages are used in the import, then those languages which were not defined in the import file will be blank (empty) or not defined.

Comment

If the column name starts with a hash character (#), then the import data column is considered a comment column and will be ignored during the import. This is a special case, which allows quickly excluding columns in the import data by adding “#” character in front of a column name.

If comment columns are used in the import data then the column order will change. For example, if the first column is commented, then the second column becomes the first column. The comment columns will be treated in the same way as if they do not exist in the import data.

Data Column Values

Data column values depend on the data column type. Each feature have its own base data type and one should use only the data, which match this type as data column values. However, data column values can also be defined based on which data properties each feature has. For example, if a feature has Multi-value data property, then one can define several data values for this type of feature at once. If a feature has Selectable and Hierarchical (Allow Hierarchy) data properties, then one can define the feature value, which is hierarchical.

Binary Data

Binary data is a file or an image in Perfion. Both data types are handled almost in the same way and there is practically no difference how the data column values have to be defined for images as compared to the files.

The binary data value is a simple string type either identifying a path or an http(s) locator, pointing to the binary data.

When using paths, it is recommended to always use only the absolute paths. Note that these paths are local to the server that runs the import. When running an import from the Windows Client, that would be the machine running the Windows Client. Imports may also be run by the Perfion Application Server, any paths will then be relative to the server running the Perfion Application Server.

Using http(s) locators, e.g. https://my.images.server/product-image.png, will enable Perfion to download the images from the specified URIs. Perfion supports authentication by prefixing the username and password to the URI, e.g. https://user:pass@my.images.server/product-image.png.

Binary data in Perfion consist of two values:

• Binary data name. It is a name of the binary file. Perfion will use the file name as it is defined in the path to the file, but without a path prefix, e.g. only the file name with extension will be used.

• Binary data. It is an actual binary content of the file. During the import, the file content will be read and uploaded to Perfion.

Multi-value Data

Features which have Multi-value data property may have multiple values. Instead of defining only a single value, one can define several feature values at once. To separate values one must use a semicolon character (;) as value separator. If one wants to import values 10, 25 and 80 at once, then they have to be defined like a single value with all three values separated with semicolon, e.g. “10;25;80”.

Multi-value features may also save value order if the feature has Sortable data property. In this case the value order after the import will be the same as it is defined in the import data.

Hierarchical Data

In Perfion it is possible to have features with hierarchy. A feature, which has Selectable and Hierarchical (Allow Hierarchy) data properties can be defined with multiple hierarchy levels. To reference these level, each level is separated with a pipe character (|).

Open

Figure 6: Example of hierarchical data

In Figure 6 it is shown how selectable hierarchical feature data looks like. To create such structure one will need to define selectable feature value as “Level1|Level2|Level3”. “Level1” and “Level2” will be created as virtual items and “Level3” will be a normal item. If hierarchy values already exist, then “Level1|Level2|Level3” string used as data value will refer to the normal item “Level3”.

It is allowed to import hierarchical data to those features, which have Hierarchical (Allow Hierarchy) data feature and have String as a base data type.

If during the import the hierarchy value is not found in existing values, then all missing hierarchy values will be created automatically.

Selectable Feature Values

Features which have Selectable data property are imported using slightly different rules than explicit features. The data values and their type is the same, but Perfion will import those values using different rules.

Selectable features have two values. The first one is the list of default values and the second one is the actual selectable feature value. The import to selectable feature is the same as setting the actual selectable feature’s value, but in order to do that, the value, which is assigned to selectable feature, must exist as one of selectable feature’s default values. Therefore, Perfion will first perform the operation similar to merging of existing and incoming values and then will assign incoming data value(s) as actual selectable feature value(s).

If selectable feature has File or Image base data type, then the import will be slightly different from the procedure described above. File and Image features have binary values and Perfion does not support comparison of binary values. Therefore, instead of merging incoming values with existing default selectable feature values Perfion will always add incoming values as new values.

If selectable feature have also Multi-value data property, then the process will be very similar, but several incoming values will be handled at the same time.

Empty Data Handling in Perfion

Perfion treats empty data in the same way as if there were no data. By default all features in Perfion have “empty” values, which also means that there are no data saved in database. If one saves the data as empty string, then Perfion will not save it in the database and will ignore saving procedure, because it is default value and does not have to be saved. If data values exist in Perfion and then it is overwritten with “empty” data, then Perfion will delete the existing data to restore data values to default values. Handling empty data as no data allows Perfion to have low memory footprint and effectively save complex data structures.

Data Validation

The feature data import procedure will perform various kinds of validations to check if the data is specified correctly and will not proceed with the import if any data is not valid. Perfion performs the following validations:

• Import data column name validation. Perfion checks if data columns do not use special characters, if features defined in the data columns exist in Perfion, if different types of columns are specified following other rules (e.g. when specifying ID or localizable column names), etc.

• Data IDs validation. If any import data columns are specified as ID columns, then Perfion will check if all data used in those columns (data IDs) are valid, e.g. exist in Perfion.

• Data type validation. Perfion checks if incoming data have valid values based on the data column types. For example, if data has Number type, then Perfion will check if data values are numeric.

• Other.

The validations mentioned above are not user controllable and are always performed during the import. There may be more validations not mentioned in the list above and this type of validations are always a subject for update.

In addition to standard validations, the import may also perform additional data validations. Additional validations can be enabled by using import parameters.

The additional data validations are not performed on all import data. The validations will be performed only on those columns, which are selected to be used in validation. Moreover, each type of validation will limit the scope of data to which it can be applied. Currently, there is only one type of additional validation available, which allows to validate incoming selectable feature values. In this case, only the data which belongs to selectable feature data columns can be used.

In order to select which columns in the import data has to be used with additional validations, the import data column name should be prefixed with a ‘$’ (dollar) character. For example, if the import data column is “ProductColors”, then in order to select this data column for validation it has to be renamed to: “$ProductColors”.

Finally, the import also performs a simple feature validation for each feature under the import which have simple validation rules. This type of validation will depend on rules set up for each feature in “Feature Definition Window” in “Validation” tab. If simple validation rule for a feature use “Reject illegal values” enforcement, then feature data importer will stop the import when illegal values are detected.

Selectable Feature Data Validation

Selectable features have default values and users can select which value(s) from the default value list will be selected as actual value(s). The standard behavior of Perfion importer when importing selectable feature values is such, that any values specified in the import file (but which does not exist as selectable feature default values) will be imported and used as actual values, but Perfion in addition will add those values to selectable feature default value list. In most cases such behavior is preferred, because it allows importing data to Perfion without configuring selectable feature’s default values before the import. This is also preferred way importing data to selectable features when default values are not known in advance. In this case all selectable feature’s default values will be created continuously each time when the new data is imported.

In case when selectable feature’s default values must not be updated, then users can use selectable feature data validation during the import. This validation will make sure that the incoming data will not update selectable feature’s default values. The selectable feature’s actual values can be only selected from default value list, so in this case data will not be imported to selectable feature.

The validation is performed before the actual import, where selectable feature data is checked with data in Perfion. If any new data is found, then any unknown values are reported to the import log and the import procedure is stopped.

Importer

Feature data can be imported using different types of Perfion importers. Each importer have different requirements so in order to see how data can be imported using the particular importer, please refer to documentation for that importer. The supported importers:

• GUI Importer. GUI Importer can be opened from Perfion application. Refer to “Import / Export GUI” manual.

• Actions. The import can be executed by using the IMPORT action command. Refer to “Action User Guide” manual.

• API. One can implement a custom importer by using Perfion Importer interface. Refer to “Import / Export API” manual.

Verification of Imported Data

It is important to check the data in Perfion after the import in order to make sure that everything was imported correctly.

There are several areas where data can be imported differently than it was expected, because of data types, complex data handling rules or based on chosen import options.

• Dates and numbers, especially if decimal numbers are used. Such data depends on culture settings in Windows and if the culture does not match the culture in the import data file, then incorrect values can be imported.

• If hierarchical data is used. If the import’s purpose is to change some feature items from “normal” to “virtual” type or vice versa, then there can be significant changes made. Technically “normal” and “virtual” items are the same in Perfion, but the usage and meaning of these item types are different.

• If inherited data is used. Perfion will show data based on which feature items have value and in which hierarchy level those items are. The data may exist in the import, but it may be ignored based on inheritance rules. For example, if the parent item for feature with inheritable data property has value, then the child item will ignore any new values coming from the import and will always inherit the parent’s value.

• If binary data is imported. If binary path was not correctly defined or binary was not available at the time of the import (e.g. file locked by operating system, web based resource temporarily inaccessible, etc.), then there can be some binaries, which will not be imported and one may need to repeat the import procedure for those items separately.

• If “Delete existing values when missing in import” option was selected during the import, then some data can be deleted, even if it is not defined in the import file.

• If “Keep the first occurrence of records with duplicate keys” option was selected during the import, then Perfion may skip some of import data items during import because of duplicate item keys. Perfion will import only the first found item with the same key and will skip all other items found later with the same item key value. If the import data had items with duplicates, we recommend checking how those items were imported into Perfion.

Binary Data Import

Image and file (binary) data import is similar to normal data import, but the actual import procedure is different. Binary features are like String type features where feature value is a binary file name, but in addition they also have the binary data, e.g. file content. When binary features are imported, first all binary data must be uploaded to Perfion and only then binary features can be imported like any other features.

Binary features can be mixed with normal features during the import. From user perspective binary features will not differ from normal features how they are defined in the import file. However, when Perfion starts the import procedure, internally it separates binary and normal features and imports them separately.

Example of the import data, where two normal features and two binary features are present:

The place of binary features in the import data is not important. Binary feature cannot be used as the key feature and therefore it cannot be defined in the first import data column. Exception is if one imports data to the binary feature itself, e.g. binary feature is the import feature.

The import procedure of the import data shown in the table above:

• All normal features will be selected and imported all at once.

• All binary data will be uploaded to Perfion from the first found binary feature column, e.g. all binary files defined in “BinaryFeature1” column. From the example “file1.txt”, “file2.txt” and “file3.txt” files will be uploaded.

• The first found binary feature data will be imported. In this case Perfion will use the same key feature data (“_ID”, “Stage” and ”_Parent” columns) to import binary feature as normal feature, but Perfion will also have all the references to the binary files uploaded before and they will be automatically be assigned to the binary feature.

• All binary data will be uploaded to Perfion from the second found binary feature column, e.g. all binary files defined in “BinaryFeature2” column. From the example “image1.jpg”, “image2.jpg” and “image3.jpg” files will be uploaded.

• The second found binary feature data will be imported. The procedure is the same as for the first binary feature as it was defined in step 3. This time, however, the following data will be imported:

• The procedure will be completed at this point, but if there were more binary features defined in the import file, then procedure would continue until all binary features are imported. It will repeat the same process for each binary feature as it was described in steps 2, 3 and 4, 5.

The binary upload operation is usually time consuming and takes much longer than regular data import. To minimize issues with binary data upload, Perfion will handle binary feature items in chunks. The default value is 1000 items. This means that when binary feature import starts, it will upload all binary data to Perfion for 1000 items, then it will run the normal data import for those 1000 items. Finally, it will take another 1000 items and repeat the process. Each time a chunk of binary feature items are processed, the data becomes available in Perfion.