What is the Catalog Framework

Catalogs play a crucial role in migrating changes between Apliqo FPM models. Here’s how it works:

1. Definition: A catalog is essentially a container that captures the specific changes you want to implement in the target instance. It acts as a blueprint for the modifications you need to make.

2. Export Process: Once you’ve defined the necessary changes in the catalog, they are exported from the source instance to a designated folder on the source instance’s file system.

3. Import Process: The changes in the catalog can be imported into the target instance from a designated folder.

Catalogs are organized into Collections. These collections serve as a sequence of catalogs that must be exported or imported in a specific order to achieve larger changes in the target system. Additionally, collections can be used as a means of grouping similar catalogs together.

The Catalog User Interface (UI) can be accessed through the following path: Administration > Catalog. On this page you can define, export, import catalogs and collections.

Catalog Settings

Before using the Catalog Framework, please make sure the settings are correct for your environment. The settings can be view on the Settings tab. The Apliqo FPM Settings are the global Apliqo FPM settings used by the Catalog Framework.

There are also specific settings for the Catalog Framework.

• Catalog File Path: The Catalog File Path refers to the root directory where all catalogs are exported to. When exporting a catalog, the folder for the Collection and then Catalog is added to the Catalog File Path. For example, if the Catalog File Path was: c:\TM1 Model\Apliqo FPM\Catalog\. I was exporting the FP2024.01.01 catalog which is part of the Patch Collection. The full file path for the catalog when exporting or importing the catalog, would be c:\TM1 Model\Apliqo FPM\Catalog\patch\FP2024.01.01.

The Catalog Framework uses Bedrock processes to export, clear and import data. To export or clear a slice of data from a cube you need to define a Filter. A Filter is divided into parts. The first part is the dimension and the elements that are to be part of the filter. For example, to define a filter that exports the data associated with Actual and Budget elements in the Version dimension would be stated as: Version :: Actual ++ Budget. Where the ‘::’ is the Element List settings and ‘++’ is the Element Separator setting. Add to the filter to export only the data associated with the year 2024, the updated Filter statement would now be: Version :: Actual ++ Budget && Year :: 2024. Where the ‘&&’ separate the dimension and elements set using the Dimension Element Set settings. These settings can be changed depending on the dimensions name and metadata in your specific model.

To reduce file sizes and processing time, the data can be filtered to exclude certain types of data.

• When the Suppress Zeros has the value 1. Null data will be excluded from the file. Generally, null data is a zero or blank string values in the data point. When the data is loaded it is assumed data slice is cleared first and the non-null data incrementally loaded. Therefore, the file does not need to contain null data.

• When the Suppress Consolidations has the value 1. Numeric data that is the result of a consolidated element is excluded from the file. The Suppress Consolidations parameter affects the how filter statement is interpreted. If the Suppress Consolidations has value 1 and a consolidated element is specified in the filter, the leaf ancestors of the consolidation will be added to the slice data being exported. If the Suppress Consolidations has value 0 and a consolidated element is specified in the filter, only the consolidated element will be added to the slice data being exported.

• When the Suppress Strings has the value 1. String data that is enter against a consolidated element is excluded from the file.

• When the Suppress Rules has the value 1. Calculated data points are excluded from the file. If data is loaded into a calculated datapoint, it is disregarded.

The field separator and field quote characters can be set in the File Delimiter and File Quote settings respectively. So that these characters are not confused with the data being exported, non-printable characters are used by default. These are expressed using their ASCII code. These characters can be changed if there is any confusion in the local environment.

The Catalog Framework can execute processes when exporting or importing a catalog. When executing a process, a process parameter value can be passed to the process being executed. For example, a process has parameter, pDim that determines the dimension that the process should use. To specify the process should use the Version dimension, the Filter passed to the process would be: pDim :: Version. Where the ‘::’ is the Parameter Separator that separates the parameter from the parameter value. If more than one parameter is to be specified, then the parameter set is separated by the Parameter Set setting.

The default settings should work in most cases. These settings can be changed for any item in a catalog.

Defining a Catalog

Add a new collection

To add a Collection, click on the Collection tab. Click on the Add Collection button. In the popup, enter a description for the new collection. This will be used at the collection’s folder name on the file system.

Adding a new catalog

To add a Catalog, click on the Add Catalog button. Select the collection the catalog is to be part of. Give the catalog a description. This will be used at the catalog’s folder name on the file system.

Order

The Order specifies the sequence in which catalog items are to be executed during when exporting or importing the catalog. If the Order is the same for two or more items, the Item Number determines the order.

The Order can be change for a number of items. Select the items to be change. Click on the Change Index button. In the popup, specify the Action and the Index to be applied. The Order can be increased or decreased by a number of indexes or changed to specific index.

When exporting or importing a catalog, the process catalog twice. On the first pass, it processes metadata changes. On the second pass, it handles data changes.

Type

Type is for the type of TM1 object that is being added to the catalog. The type determines the Action, Object and Filter that can be performed when importing or exporting the catalog. The Types are:

Action, Object and Filter

The Action, Object and Filter depend on the Type selected. For some Types, additional popups become available to provide more information,

CHO - Chores

Catalog items that have the CHO type can have the following actions.

CUB - Cube and cube data.

Catalog items that have the CUB type can perform the following actions that affect the cube.

The following actions check that the cube exists. If the cube does not exist, the cube is built from the catalog. If the cube’s dimensions do not exist, then those dimensions that do not exist are built from the catalog. Once the cube has been built, the data is changed as specified by the Action.

The following actions do not check that the cube exists. If the cube does not exist, the step is skipped.

The Filter is defined in the cube popup which comes available when the Action and Object (Cube) is selected. In popup:

Open

In the Settings widget, at the top, are the Settings for the Item. Any setting that is left blank, the default setting is used. See the Catalog Settings above for more information.

In the bottom widget, you’ll find the dimensions of the cube. To filter the data of the cube that is to be exported, select an element from the drop-down list. Up to 3 element can be selected for any dimension. If more that 3 elements need to be selected, the elements can be typed into the Element n column. Remember to separate each element with the Element Separator. Close the popup,

DIM - Dimensions

Catalog items that have the DIM type can perform the following Actions.

ELE - Element

Catalog items that have the ELE type can perform the following Actions.

EXE - Execute Process

Catalog items that have the EXE type can perform the following Actions.

The Filter is defined in the process popup which comes available when the Action and Object (Process) is selected. In popup:

Open

In the Settings widget, at the top, are the Settings for the Item. Any setting that is left blank, the default setting is used. See the Catalog Settings above for more information.

In the bottom widget, you’ll find the parameters of the process. Enter the Parameter Value to be used when running the process. The parameter Default Value is used when the parameter is left blank.

FIL- File

Catalog items that have the FIL type can perform the following Actions.

RUX- Rule

Catalog items that have the RUX type can perform the following Actions.

VUE - Views

Catalog items that have the VUE type can perform the following Actions.

SUB - Subsets

Catalog items that have the SUB can perform the following Actions.

Clearing an Item

In the catalog select the Items to be cleared. Click on the Clear button.

Copying

Copying within a catalog

Items can be copies with at catalog. Select the Items that are to be copied. Click on Copy Catalog button. Set the Copy Catalog and the Add to Catalog to be the same. This is the default values for these fields. Click on Copy. The selected items will be added to the bottom of the catalog and given the same Order as the copied Items.

Copying another catalog

Another catalog can be copied into the current catalog. Click on Copy Catalog button. For Copy Catalog select the catalog that is to be copied. Click on Copy. This will add the Items from the selected catalog to the bottom of the current catalog. The Items are given the next Order index for the current catalog.

In the Catalog Framework there is a Cookbook collection. This is a collection of common catalog items in Apliqo FPM that can be copied into any new catalogs.

Renaming Catalogs and Collections

Renaming a Catalog

On the Create Catalog tab, click on the Settings button. Change the Name of the catalog.

On the Collection tab, change the catalog name in the Name column.

Rename a Collection

On the Collection tab, change the collection name in the Name column.

Deleting Catalogs and Collections

Deleting a Catalog

To delete a Catalog go the Collections tab. For the Catalog that is to be deleted click on Delete icon. In the popup confirm that you want to delete the catalog by entering the name of the catalog that is to be deleted. Click on Delete.

Deleting a Collection

To delete a Collection, go the Collections tab. Select the collection that is to be deleted then click on Delete icon. In the popup confirm that you want to delete the collection by entering the name of the collection that is to be deleted. Click on Delete.

Exporting the Catalog

To Export the catalog click on the Create Catalog button.

Exporting the catalog has three stages.

Stage 1: Review

To review the catalog in the user interface, click on Create Catalog button. Do not Create Catalog File or Export Catalog. Click on Create Catalog. This will update the Filter for items that are using the CUB or EXE action.

Stage 2: Catalog file only

To create the catalog definition file but not export the data associated with the catalog. Select Create Catalog File but do not Export Catalog. This will update the Filter for items that are using the CUB or EXE action. Additionally, create a catalog.csv file in the //[Catalog Root File Path]/[Collection Name]/[Catalog Name]/ directory.

Stage 3: Export catalog

In addition to Stage 1 and Stage 2 selecting to Create Catalog File and Export Catalog will export the catalog data as specified by the catalog.csv file to the //[Catalog Root File Path]/[Collection Name]/[Catalog Name]/ directory.

Run splicer

Splicer is a utility built into the Apliqo FPM that converts the cube rules to and from a neutral state. Running the Splicer when Exporting the catalog, first de-splice the rules, then export the rules in their de-spliced state. Before finishing, re-splice the rules.

De-splicing and re-splicing the rules take time to compile and is locking event on the server. It is only necessary if you are migrating rules the catalog and you can expect that the target model has different spliced rules.

Importing a Catalog

Go to the Import Catalog tab of the Catalog page.

• Directory: Enter the full path to the catalog.csv file.

• Backup: Prompts the import process to back-up the Apliqo FPM model before applying in the catalog.

• Process Rule Changes: Apply any changes to rules as defined in the catalog.

• Run Splicer: Splicer is a utility built into the Apliqo FPM that converts the cube rule to and from a neutral state. Selecting this option de-splices the rules before importing the catalog. Before finishing the rules are re-spliced. De-splicing and re-splicing the rules take time to compile and is locking event on the server. It is only necessary if you are migrating rules the catalog and you know the source model has different spliced rules.

• Copy Processes: Prompts the import process to copy the .pro file from the catalog to the data folder of the Apliqo FPM model. This will require a restart of the model if any processes are copied.

• Copy Chores: Prompts the import process to copy the .cho file from the catalog to the data folder of the Apliqo FPM model. This will require a restart of the model if any chores are copied.

• Copy Subsets: Prompts the import process to copy the .sub file from the catalog to the data folder of the Apliqo FPM model. This will require a restart of the model if any subsets are copied.

• Copy Views: Prompts the import process to copy the .vue file from the catalog to the data folder of the Apliqo FPM model. This will require a restart of the model if any views are copied.

Trouble Shooting

Whenever a catalog is exported or imported, an }APQ.Server.Catalog.Export.json or }APQ.Server.Catalog.Import.json file is produced. In these files are step the process took to export or import the catalog. Any record that has a danger type should be investigate using the TM1Server logs and TM1 process logs.

This is the end.