Page Comparison

...

• TM1RuleParser - implements custom actions that get called from the generated parser to create object tree representing parsed TM1 rule.

• ArgumentParser - implements CLI parsing and is used by Splicer.py to run required actions by instantiating TM1SpliceExecutor and running its method corresponding to the action required from CLI.

TM1RuleParser (class in TM1RuleParser.py)

This class is responsible to translate parsed rule into objects in memory that represent the parsed syntax. Methods with name starting with make_ or add_match custom actions as defined in TM1RuleGrammar.peg (described later). These methods are called during parsing of the TM1 rule and are fed with data from the parser defined in TM1RuleGrammar.py.

Representation of Parse Tree

Let’s explain how the parser stores the parsed rule in memory on following example.

...

This rule will be parsed by the TM1RuleParser method get_parse_tree and the parsing result will be returned in an instance of TM1Splice.Parser.Grammar.TM1RuleGrammar.TreeNode class. The class contains elements property, which is a list of parsed tokens. You may see a simplified representation of the parsed rule on below picture.

...

It is important to understand how the parser accesses the parsed tokens during the custom actions. Let’s demonstrate it on a following example:

Following simplified code represents a PEG rule that we will further refer to in the custom action methods implemented in TM1RuleParser.

DirDimElemPairList      <- DirDimElemPair DirDimElemPairs* %add_dim_to_map
DirDimElemPairs         <- _ "," _ DirDimElemPair %add_dim_to_map2
DirDimElemPair          <- DimIdentifier _ ":" _ DirElementList %make_dim_elem_pair
DimIdentifier           <- "'" [^\'\"\@\<\!\[\]\^\*\>\=\\\|\?\/\;\,\~\%\&\:\n\r]+ "'" %make_ident

The parser generated by Canopy interfaces all tokens trough an elements list. This list is always passed as an argument to a custom action method and contains tokens indexed according to a PEG parsing rule that activated the custom action.

For a better understanding let’s explain how the first parsing rule will affect content of the element list when add_dim_to_map method is called. Note: This method will called once the parser has parsed a dimension/element pair or pairs.

    def add_dim_to_map(self, input, start, end, elements):
        dim_map = dict()
        if elements[0] and elements[1]:
            for dim, elem in [*[elements[0]], *(elements[1].elements)]:
                dim_map[dim] = elem
        return dim_map

As you can see from the method header and implementation, the elements list is used in the code logic to access DirDimElemPair content by accessing elements[0]. Index 0 corresponds with position of DirDimElemPair non-terminal symbol in the rule. It is first symbol that has been parsed and since the elements list is zero based, the index of the symbol is 0. It is important to understand that content will be available after full expansion of the DirDimElemPair non-terminal symbol. That means the parser has to expand all non-terminal symbols until it reads terminal symbols from the input rule.

Similarly for DirDimElemPairs, the symbol is second in the rule and thus its index will be 1. Note that DirDimElemPairs is followed by * in the PEG rule, which indicates zero or n repetitions of the non-terminal symbol. That's the reason why we access elements[1].elements as we intend to read content of all occurrences of the DirDimElemPairs non-terminal symbol as have been parsed.

Non-terminal symbol as per definition needs to be further expanded, so we can follow the parsing logic of DirDimElemPair until DimIndetifier rule. You may see from the rule itself, once it will be parsed, the parser will call a custom method make_dim_elem_pair.

    def make_dim_elem_pair(self, input, start, end, elements):
        if type(elements[4]) is list:
            pairs = []
            for (hierarchy, element) in elements[4]:
                pairs.append((hierarchy if hierarchy else elements[0], element))
            return (elements[0], pairs)
        else:
            if not elements[4][0]:
                elements[4] = (elements[0], elements[4][1])
            return (elements[0], [elements[4]])

Again you may notice the custom action method header is following the same pattern as in previous case. We have elements list from which we can access all tokens as parsed during expansion of DirDimElemPair. In this case we can find DimIdentifer contents in elements[0], elements[1] would contain all whitespace characters that could follow a dimension identifier (or if there are none, elements[1] will be empty). Next, elements[2] would contain a single character : that is used to divide an element name from the dimension name. And similarly, elements[3] would contain all whitespace characters that could follow a : separator. Now, elements[4] refers to tokens that were collected when parsing DirElementList non-terminal symbol.

We followed expansions of each non-terminal symbol in above examples, but the parser needs to use terminal symbols to be able to properly select which parsing rules apply. Let’s demonstrate this on the last rule the parser would use when reading a dimension name. The rule that applies in this case is expansion of DimIdentifier. This rule can be interpreted as follows: DimIdentifier is a string enclosed in single quotes consisting of at least one allowed character. If the parser reads such a token, it will call custom action method make_ident. Let’s have a look how the method is implemented:

def make_ident(self, input, start, end, elements):
    ident = ""
    for char in elements:
        ident = ident + char.text
    # ignore leading/trailing apostrophes
    ident = ident.strip("'")
    return ident

The method has the same header as in above cases, elements list will now contain only terminal symbols as the rule for DimIdentifier suggests. Note: Terminal symbols are basically characters that are read from the source text being parsed.

That means the parser will “consume” the terminal symbols until the last matching character, which is a single quote that concludes a dimension name used in an area statement or a directive definition. The consumed characters will be available in elements list. Again we must refer to the matching PEG grammar rule to identify how the characters read from source will map to individual elements in the list. In our case, let’s say we have parsed the following string:

'T Year'

The elements list will consist of terminal symbols as in below table.

Since terminal symbols are represented as TreeNode objects by the parser, the actual content is stored in text property of each TreeNode object contained in the elements list. Therefore to get the identifier, we must concatenate the text property of each member. As a very last step we will strip leading and trailing single quotes as they have no value for splicing.

TM1RuleParser Properties and Methods

The class defines following attributes:

...

ArgumentParser (class in ArgumentParser.py)

The class implements command line parsing and is used by main application defined in Splicer.py.

...