FPM contains several parallel enabled processes to provide faster way of data copy between FIN General Ledger TB, FIN General Ledger and planning cubes. The processes are mainly part of roundtrip data processing.
Parallelization Mechanism Introduction
As of TM1 version 11 there are no dedicated thread synchronization functions for parallel processes and thus current solution relies on polling thread status in }APQ C3 Thread Control cube. The polling is always provided by the main thread, child threads are only responsible to signal their status into }APQ C3 Thread Control cube. The polling mechanism relies on issuing periodic CubeSaveData command on the }APQ C3 Thread Control cube and is run by the main process in short intervals to enable the main process to see any data changes in the }APQ C3 Thread Control cube from the child threads. This makes sure the main thread can detect data changes in the control cube during the child threads execution as TM1 commit model will normally not allow to see the data changes in the cube from any other process until the process modifying data exits. This approach has its downsides, but it doesn’t rely on any semaphore files or external software and is implemented purely as TI code. It is important to understand that all parallel enabled processes in FPM have common code for the logic of both main process and child threads, so only one TI process implements both.
Note: The situation will change with TM1 version 12, which will no longer support CubeSaveData command, but will support thread synchronization out of the box. As a possible solution to this problem, there could be two versions of the parallel enabled processes in the FPM model - one for version 11 and second for version 12 and above. Switching between these can be simply provided by modified roundtrip configurations or by assigning proper process version to respective roundtrip step.
Parallelization Mechanism Details
Main Thread Role
only monitor threads
involve in data processing - timeout miss risk
Spawning Threads
Data Allocation to Threads
no mapping between source and target cube dimensions
mapping between source and target cube dimensions
Thread Initiation Polling
The polling interval is a configuration variable that can be set in }APQ Settings cube and should be set to a reasonable value. Setting the value too low can lead to various locking situations, setting the value too high will cause delay the main process execution as the signal to continue with main process execution comes at periodic intervals.
Since there are no tools to control child threads from the main thread, the parallel processing algorithm implements two different timeouts for child thread monitoring to prevent main process stalling in the polling loops and thus having zombie processes running on the TM1 server.
First timeout is used to monitor if all child threads have initiated correctly within a specified time period. Two types of exception can happen during this period:
The child thread will not initiate and will exit before it started processing data (for example due to a data source error).
The child thread has time related issues during its prolog initialization (for example due to infinite loop in the prolog code or a source view that calculates slowly).
Thread Completion Polling
Messages Collection and Final Message
The polling interval is a configuration variable that can be set in }APQ Settings cube and should be set to a reasonable value. Setting the value too low can lead to various locking situations, setting the value too high will cause delay the main process execution as the signal to continue with main process execution comes at periodic intervals.
Since there are no tools to control child threads from the main thread, the parallel processing algorithm implements two different timeouts for child thread monitoring to prevent main process stalling in the polling loops and thus having zombie processes running on the TM1 server.
First timeout is used to monitor if all child threads have initiated correctly within a specified time period. Two types of exception can happen during this period:
The child thread will not initiate and will exit before it started processing data (for example due to a data source error).
The child thread has time related issues during its prolog initialization (for example due to infinite loop in the prolog code or a source view that calculates slowly).
The main process will monitor each thread initiation status in the }APQ C3 Thread Control cube and if any of the child threadsParallel processing gains efficiency by dividing source data into smaller segments and processing each segment concurrently but independently through multiple processes. This division is achieved by selecting appropriate subsets of elements from the parallelization dimension and allocating each subset to a distinct process, which collectively work towards the final outcome.
While parallel processing typically enhances data processing speeds, its advantages are largely influenced by how the source data is distributed to the parallel processes.
Optimal performance can be attained by selecting a dimension for parallelization that guarantees a balanced distribution of data across all parallel processes. If the data distribution is uneven, it can negatively impact the performance of parallel processing.
In severe situations, parallel processing might even lag behind the speed of a single-process operation.
It is important to utilize knowledge of data to properly choose a dimension for parallelization as there are no system statistics available in TM1 to check for data distribution across different dimensions of a cube.
Parallelization in FPM is generally enabled on following dimensions:
FIN CurrencyFIN BSEG 1,FIN BSEG 2,FIN BSEG 3,FIN BSEG 4FIN BSEG 1 Plan,FIN BSEG 2 Plan,FIN BSEG 3 Plan,FIN BSEG 4 PlanFIN Company
Please note that other dimensions from FIN General Ledger or FIN General Ledger TB are not suitable for parallelization due to dependencies implied by rule calculations in FIN General Ledger that could affect calculation results depending on order in which the parallel processes finished.