name: LanguageModelXML
synopsis:

>     g++ [flags ...] file ... -l /isip/tools/lib/$ISIP_BINARY/lib_mmedia.a

>     #include <LanguageModelXML.h>

>     boolean read(Sof& sof, long tag, const String& cname = CLASS_NAME);
>     RuleModel getRuleModel();

quick start:

>     Sof sof;
>     sof.open("lm_xml_obj.sof", File::READ_ONLY);

>     LanguageModelXML lmxml;
>     lmxml.read(sof, 0);

>     LanguageModelIHD lmihd;
>     lmihd.setRuleModel(lmxml.getRuleModel());

>     HierarchicalDigraph hg;
>     hg.assign(lmihd.getHDigraph());

description:
The broad goal of the LanguageModelXML class is to parse XML format grammars as specified by the W3C Speech Recognition Grammar Specification Version 1.0 and convert them into IHD with the help of other LanguageModel classes and to perform the reverse conversion. LanguageModelXML helps perform the XML->IHD conversion by converting the XML to ABNF then calling LanguageModelABNF functions to convert from ABNF->BNF. These BNF productions may then be used as the argument of a setRuleModel() for a LanguageModelIHD, and the equivalent IHD may be obtained with a call to LanguageModelIHD's getHDigraph(). The reverse conversion (IHD->XML) starts when the BNF production rules for an IHD are obtained via a LanguageModelIHD object's getRuleModel(). These BNF productions are then fed to a LanguageModelXML object's setRuleModel(), initiating the conversion from BNF->XML. The resulting XML may then be retrieved by outputting the LanguageModelXML object to a file or through the getXMLModel() method. The XMLParser class is used to parse all XML format content.
When an XML grammar is converted to IHD format, occurrences of the special rule NULL (<ruleref special='NULL'/>) will translate into "dummy" nodes (nodes which require no input). The default name of these dummy symbols will be "isip_rule_null", but any other name may be specified by defining the isip_rule_null attribute in the grammar start tag. For example:
<grammar isip_rule_null='dummy_symbol'>
This will set the name of the dummy nodes to "dummy_symbol".
dependencies:

- Console
- DiGraph
- HierarchicalDigraph
- Filename
- LanguageModelBase
- Pair
- Sdb
- Short
- Sof
- Triple
- Ulong
- XMLParser
- XMLToken
- LanguageModelABNF

public constants:

- define the class name:

      static const String LanguageModelXML::CLASS_NAME = L"LanguageModelXML";

- define the file parameters:

      static const String LanguageModelXML::DEF_PARAM = L"";

      static const long LanguageModelXML::DEF_LEVEL = -1;

      static const String LanguageModelXML::PARAM_GRAMMARS = L"grammars";

      static const String LanguageModelXML::PARAM_UNDERSCORE = L"_";

      static const long LanguageModelXML::DEF_TAG = 0;

      static const String LanguageModelXML::CONTEXT_LABEL_PREFIX = L"G_";

- define a default weight for arcs:

      static const float LanguageModelXML::DEF_WEIGHT = 1.0;

- define the special rules

      static const String LanguageModelXML::SPECIAL_GARBAGE = "GARBAGE";

      static const String LanguageModelXML::SPECIAL_VOID = L"VOID";

      static const String LanguageModelXML::SPECIAL_NULL = L"NULL";

- define the valid XMLToken values

      static const String LanguageModelXML::ONE_OF = L"one-of";

      static const String LanguageModelXML::ITEM = L"item";

      static const String LanguageModelXML::RULEREF = L"ruleref";

      static const String LanguageModelXML::RULE = L"rule";

      static const String LanguageModelXML::GRAMMAR = L"grammar";

- define the supported XMLToken attributes

      static const String LanguageModelXML::WEIGHT = L"weight";

      static const String LanguageModelXML::REPEAT = L"repeat";

      static const String LanguageModelXML::REPEAT_PROB = L"repeat-prob";

      static const String LanguageModelXML::ROOT = L"root";

      static const String LanguageModelXML::URI = L"uri";

      static const String LanguageModelXML::SPECIAL = L"special";

      static const String LanguageModelXML::ID = L"id";

- define delimiters found in attribute values

      static const String LanguageModelXML::RULEREF_URI_LOCAL_DELIM = L"#";

      static const String LanguageModelXML::REPEAT_RANGE_DELIM = L"-";

      static const String LanguageModelXML::REPEAT_LOOP_BACK = L"1-";

- define a default dummy symbol

      static const String LanguageModelXML::ISIP_DUMMY_NODE = L"isip_rule_null";

- define sof output constants

      static const String LanguageModelXML::XML_VERSION_TAG = L"";

      static const String LanguageModelXML::ROOT_ATTRIB = L"root";

      static const String LanguageModelXML::ID_ATTRIB = L"id";

      static const String LanguageModelXML::ALGORITHM = L"algorithm";

      static const String LanguageModelXML::XML = L"XML";

      static const String LanguageModelXML::IMPLEMENTATION = L"IHD";

      static const String LanguageModelXML::IHD = L"IHD";

      static const long LanguageModelXML::LOOP_BACK = 0;

      static const long LanguageModelXML::TERMINAL_INDEX = -2;

      static const long LanguageModelXML::START_INDEX = -1;

error codes:

- error code indicating LanguageModelXML class general error:

      static const long ERR = 100800;

- error code indicating that no symbol table is present when attempting to read context mapping:

      static const long ERR_CTXT_NO_SYM_TAB = 100801;

- error code indicating the symbol was not found in the graph name list:

      static const long ERR_CTXT_INV_LENGTHS = 100802;

- error code indicating the length of the symbol tables and number of sub-graphs do not agree:

      static const long ERR_CTXT_INV_GRAPH_ALIGN = 100803;

- error code indicating an invalid symbol was found :

      static const long ERR_CTXT_INV_SYM = 100804;

- error code indicating an error reading the symbol type:

      static const long ERR_READ_SYM_TYPE = 100805;

- error code indicating an error storing the context into a vector:

      static const long ERR_CTXT_SCV = 100806;

- error code indicating more than one symbol was found in a context graph:

      static const long ERR_CTXT_INV_SYM_TAB = 100807;

- error code indicating an error tokenizing the index of a context from the context symbol:

      static const long ERR_CTXT_INV_TOKEN_INDEX = 100808;

- error code indicating an error storing the vector of contexts into a HierarchicalDigraph:

      static const long ERR_CTXT_ESHG = 100809;

- error code indicating the same context was found twice with different indices:

      static const long ERR_CTXT_EXISTS_DIF_INDEX = 100810;

- error code indicating the same context was found twice with the same index:

      static const long ERR_CTXT_EXISTS_SAME_INDEX = 100811;

- error code indicating an error creating a temporary file:

      static const long ERR_DEBG_TEMP_FILE = 100812;

- error code indicating an attempt to diagnose the search methods with no grammar loaded:

      static const long ERR_DEBG_DSM_NOGR = 100813;

- error code indicating an attempt to diagnose the arc-target finding methods with no grammar loaded:

      static const long ERR_DEBG_DTM_NOGR = 100814;

- error code indicating an invalid token in a repeat tag attribute value:

      static const long ERR_REPT_IAV = 100815;

- error code indicating a missing grammar start tag:

      static const long ERR_GRAMMAR_NO_START = 100816;

- error code indicating a missing grammar end tag:

      static const long ERR_GRAMMAR_NO_END = 100817;

- error code indicating an empty grammar:

      static const long ERR_GRAMMAR_EMPTY = 100818;

- error code indicating an improperly formatted grammar:

      static const long ERR_GRAMMAR_FORMAT = 100819;

- error code indicating a rule reference to a rule which cannot be found:

      static const long ERR_RULEREF_NOT_FOUND = 100820;

- error code indicating an attemt to use a non-local rule reference:

      static const long ERR_RULEREF_NON_LOCAL = 100821;

- error code indicating an empty one-of tag:

      static const long ERR_ONE_OF_EMPTY= 100822;

- error code indicating an empty rule tag:

      static const long ERR_RULE_EMPTY = 100823;

- error code indicating that an invalid tag type has been declared as valid:

      static const long ERR_TOKEN_UNHANDLED = 100824;

- error code indicating an attempt to use a special rule which is not currently supported:

      static const long ERR_RULE_SPECIAL_INV = 100825;

- error code indicating an error creating a vertex:

      static const long ERR_VERTEX_STORE = 100826;

protected data:

- declare a long int to store the rule nesting level

      long rule_nesting_level_d;

- declare a Vector of XMLTokens for every digraph on every level

      Vector< Vector < Vector< XMLToken > > > grammars_d;

- declare an IHD to hold the level information we can't store as XMLTokens

      HierarchicalDigraph hg_d;

- declare a Vector of Strings to store the list of symbols for a single grammar

      Vector< String > symbol_table_d;

- declare a temporary Rule to be used during processing

      Rule temp_rule_d;

- declare a Vector of Rules to store a partitioned XML grammar

      Vector<Rule> rules_d;

- declare an XMLToken to store the grammar start tag

      XMLToken grammar_start_tag_d;

- declare an XMLToken to store the grammar end tag

      XMLToken grammar_end_tag_d;

required public methods:

- static methods:

      static const String& name();

      static boolean diagnose(Integral::DEBUG debug_level);

- debug methods:

      boolean debug(const unichar* msg) const;

- constructor:

      LanguageModelXML();

- i/o methods:

      boolean read(Sof& sof, long tag, const String& cname = CLASS_NAME);

      boolean readData(Sof& sof, const String& pname = DEF_PARAM, long size = SofParser::FULL_OBJECT, boolean param = true, boolean nested = false);

      boolean write(Sof& sof, long tag, const String& cname = CLASS_NAME) const;

      boolean writeData(Sof& sof, const String& pname = DEF_PARAM) const;

      long sofSize() const;

- assign methods:

      boolean assign(const LanguageModelXML& arg);

      boolean assign(const LanguageModelBase& arg);

- operator= methods:

      LanguageModelXML& operator=(const LanguageModelXML& arg);

- equality methods:

      boolean eq(const LanguageModelXML& arg) const;

      boolean eq(const LanguageModelBase& arg) const;

- operator== methods:

      LanguageModelXML& operator==(const LanguageModelXML& arg);

- clear methods:

      boolean clear(Integral::CMODE ctype);

class-specific public methods:

- input/output methods

      RuleModel getRuleModel();

      boolean setRuleModel(const RuleModel& rm_a);

      Pair< Vector< Vector< Rule > >, HierarchicalDigraph> getXMLModel();

      boolean setXMLModel(Pair< Vector< Vector< Rule > >, HierarchicalDigraph> xml_model_a);

- methods used when reading/writing:

      boolean readXMLGrammars( Sof& sof, const String& gname, long level, boolean use_context, Vector< SearchSymbol >* symbol_table, SofParser& parser )

      boolean parseXMLGrammar(String grammar_a, Vector< String >& sub_symbol_list_a, Vector< XMLToken >& token_vector_a, String& grammar_name_a);

      boolean alignGraphs(Vector< Vector< XMLToken > >& grammar_list_a, Vector< SearchSymbol >& symbol_table_a, Vector< String >& graph_name_list_a);

      boolean readSymbolType( Sof& sof, long level, const String& pname, Vector< SearchSymbol >& symbol_list, SofParser& parser );

      boolean readContextMapping(Sof& sof, const String& pname, long level, SofParser&parser);

class-specific protected methods:

- output methods:

      boolean addDummyItem(Vector< XMLToken >& token_vector_a, long& depth_a) const;

      boolean addEndBranchTag(Vector< XMLToken >& token_vector_a, long& depth_a) const;

      boolean addEndItemTag(Vector< XMLToken >& token_vector_a, long& depth_a) const;

      boolean addStartBranchTag(Vector< XMLToken >& token_vector_a, long& depth_a) const;

      boolean addStartItemTag(Vector< XMLToken >& token_vector_a, long& depth_a, float weight_a=1, boolean repeat_a=0, float repeat_prob_a=1) const;

      boolean writeGrammars(Sof& sof_a, long level_a, HierarchicalDigraph& h_digraph_a) const;

      boolean writeLevelTag(Sof& sof_a, long level_a, const String& tag_a) const;

      boolean addCDATA(Vector< XMLToken >& token_vector_a, const String& cdata_a, long& depth_a) const;

      boolean addStartRuleTag(Vector< XMLToken >& token_vector_a, String rule_name_a, long& depth_a) const;

      boolean addEndRuleTag(Vector< XMLToken >& token_vector_a, long& depth_a) const;

      boolean writeSymbols(Sof& sof_a, long level_a, const String& pname_a, Vector< SearchSymbol > symbol_list_a) const;

      long LanguageModelXML::constFindFollowingTagAtDepth(Vector< XMLToken >& token_vector_a, long start_index_a, long base_depth_a, XMLToken::TYPE type_a, String value_a) const;

      String tokensToString(Vector< XMLToken >& token_vector_a) const;

      Vector< XMLToken > removeRedundantItemTags(Vector& token_vector_a) const;

- BNF to XML conversion methods:

      Vector< XMLToken > convertBNFtoXML(ProductionRuleSet prset_a);

      ProductionRuleSet extractRule(const String& rule_name_a, ProductionRuleSet& prset_a) const;

      boolean addRule(ProductionRuleSet& prset_a, Vector< XMLToken >& token_vector_a) const;

      boolean addAlternative(ProductionRule pr_a, Vector< XMLToken >& token_vector_a, long depth_a) const;

- XML to ABNF conversion methods:

      ProductionRule convertXMLtoABNF(Vector< XMLToken > token_vector_a);

      boolean appendProduction(ProductionRule& prod1_a, ProductionRule prod2_a);

      Vector< XMLToken > getRange(long start_a, long end_a, Vector< XMLToken > vector_a);

      long findNextItem(long start_a, Vector< XMLToken > vector_a);

      boolean preProcessXMLGrammar(Vector& token_vector);

      boolean validateXMLGrammar();

      boolean partitionGrammar(Vector& token_vector);

      boolean expandXMLRules();

      boolean expandXMLRule(String rule_id);

      boolean handleStartAndEndElement (XMLToken xml_token);

      boolean handleStartElement (XMLToken xml_token);

      boolean handleEndElement (XMLToken xml_token);

      boolean handleCharacterData (XMLToken xml_token);

      boolean handleSpecialRules(Vector& token_vector);

      boolean handleRepeatRequests(Vector& token_vector);

      boolean addOptionalRepeatSections(Vector& token_vector, Long m);

      boolean addManditoryRepeatSections(Vector& token_vector, Long m);

- XML grammar search methods:

      long findFirstItemTagOfCDATA(long index);

      long findFirstItemTagOfCDATA(Vector& token_vector, long index);

      long findFollowingStructure(long index);

      long findFollowingStructure(Vector& token_vector, long index);

      long findForwardTargetStructure(long index);

      long findForwardTargetStructure(Vector& token_vector, long index);

      long findImmediateNesting(long index);

      long findImmediateNesting(Vector token_vector, long index);

      long findInternalStructure(long index);

      long findInternalStructure(Vector& token_vector, long index);

      long findMatchingStartTag(long index);

      long findMatchingStartTag(Vector& token_vector, long index);

      long findMatchingEndTag(long index);

      long findMatchingEndTag(Vector& token_vector, long index);

      long findPrecedingStructure(long index);

      long findPrecedingStructure(Vector& token_vector, long index);

      long findPrecedingTagAtDepth( long start_index, long base_depth, XMLToken::TYPE type = XMLToken::NULL_TAG, String value = String::EMPTY );

      long findPrecedingTagAtDepth( Vector& token_vector, long start_index, long base_depth, XMLToken::TYPE type = XMLToken::NULL_TAG, String value = String::EMPTY );

      long findFollowingTagAtDepth( long start_index, long base_depth, XMLToken::TYPE type = XMLToken::NULL_TAG, String value = String::EMPTY );

      long findFollowingTagAtDepth( Vector& token_vector, long start_index, long base_depth, XMLToken::TYPE type = XMLToken::NULL_TAG, String value = String::EMPTY );

- miscellaneous worker methods:

      boolean clearXMLGrammar();

      boolean displayVector(Vector& targets);

      boolean displayXMLModel();

      boolean displayDiGraph();

      boolean debugSearchMethod(long (LanguageModelXML::*func_ptr)( long ));

      boolean LanguageModelXML::debugTargetMethod(long (LanguageModelXML::*func_ptr)(long, Vector& ));

      boolean indexIsA(long index, String value);

      boolean indexIsA(Vector& token_vector, long index, String value );

      boolean indexIsA(long index, XMLToken::TYPE type );

      boolean indexIsA(Vector& token_vector, long index, XMLToken::TYPE type );

      boolean isInBounds(long index);

      boolean isInBounds(Vector& token_vector, long index);

      DiGraph getDiGraph();

      String getDummySymbol();

      String getGrammarName();

      Vector getHandledValues();

      float getRepeatProb(long index);

      String getRuleID(Rule rule);

      long getRuleIndexByID(String id);

      Vector getSymbolList()

      float getWeight(long index = Integral::NO_POS);

      boolean updateSymbolTable(XMLToken xml_token);

      boolean storeVertex(XMLToken xml_token, long index);

      Triple tokenizeRepeatValue(long index);

      Triple tokenizeRepeatValue(XMLToken xml_token);

- other methods required by interface with LanguageModelBase

      const String& className() const

examples:

- This example shows how to use a LanguageModelXML object to read a LanguageModelXML object from a file, and retrieve the parsed grammars in HierarchicalDigraph form.

      // declare an sof object to parse the file
      //
      Sof sof;

      // open the file for reading
      //
      sof.open("lm_xml_obj.sof", File::READ_ONLY);

      // declare a LMXML object to read the grammars and perform
      // the XML->BNF conversion
      //
      LanguageModelXML lmxml;

      // read the grammars
      //
      lmxml.read(sof, 0);

      // declare an HierarchicalDigraph object to store the grammars in DiGraph form
      //
      HierarchicalDigraph hg;

      // declare a LanguageModelIHD object to perform the BNF->IHD conversion
      //
      LanguageModelIHD lmihd;

      // perform conversion
      //
      lmihd.setRuleModel(lmxml.getRuleModel());

      // retrieve the grammars from the LMIHD object, and store them in the HD object
      //
      hg.assign(lmihd.getHDigraph());

notes: