name: JSGFParser
synopsis:

>     g++ [flags ...] file ... -l /isip/tools/lib/$ISIP_BINARY/lib_mmedia.a

>     #include <JSGFParser.h>

>     JSGFParser();
>     boolean setExpression(String& expression_a, long line_a);
>     boolean parseExpression(const String& graph_start_a, const String& graph_term_a);
>     DiGraph<String> getGraph();
>     Vector<String> getSymbolList();
>     String getGrammarName();

quick start:

>     JSGFParser jp;
>     String start(L"S");
>     String term(L"T");
>     String expression(L"public <rule> = /0/ A /0/ B /0/ C;");
>     jp.setExpression(expression, 0);
>     jp.parseExpression(start, term);

description:
JSGFParser is a parser used by the LanguageModel class to convert the platform-independent, vendor-independent textual JSGF grammar (Java Speech Grammar Format) to the ISIP internal DiGraph format. Digraph is used to represent grammar as a graph. This graph is used for training and decoding in the ISIP speech recognizer. For details on JSGF, see:

Java Speech Grammar Format Specification, http://java.sun.com/products/java-media/speech/forDevelopers/JSGF/index.html

dependencies:

- Queue
- Stack
- Vector
- DiGraph
- JSGFToken

public constants:

- define the class name:

      static const String CLASS_NAME = L"JSGFParser";

- define i/o related constants:

      static const long DEF_MAX_EXPRESSIONS = 500;

      static const long DEF_LINE_NUMBER = 0;

      static const String DEF_GRAPH_START = L"";

      static const String DEF_GRAPH_TERM = L"";

      static const String DEF_GRAMMAR_NAME = L"";

error codes:

- error code indicating JSGFParser class general error:

      static const long ERR = 50100;

protected data:

- JSGF expression lines:

      String expression_d[500];

- number of expression lines:

      long line_d;

- queues of JSGF tokens corresponding to the expressions:

      Queue<JSGFToken> queue_d[500];

- queue of JSGF tokens for final drawing arc:

      Queue<JSGFToken> final_d;

- graph converted from JSGF grammar:

      DiGraph<String> graph_d;

- grammar name:

      String grammar_name_d;

- symbol list of graph:

      Vector<String> symbol_list_d;

- graph starting and ending symbols:

      String graph_start_d;

      String graph_term_d;

- static memory manager:

      static MemoryManager mgr_d;

required public methods:

- static methods:

      static const String& name();

      static boolean diagnose(Integral::DEBUG level_a);

- debug methods:

      boolean debug(const unichar* msg_a) const;

- destructor/constructor(s):

      ~JSGFParser();

      JSGFParser();

- assign methods:

      boolean assign(const JSGFParser& arg_a);

- operator= methods:

      JSGFParser& operator=(const JSGFParser& arg_a);

- equality methods:

      boolean eq(const JSGFParser& arg_a) const;

- i/o methods:

      boolean read(Sof& sof, long tag, const String& cname = CLASS_NAME);

      boolean write(Sof& sof, long tag, const String& cname = CLASS_NAME) const;

- memory management methods:

      static void* operator new(size_t size);

      static void* operator new[](size_t size);

      static void operator delete(void* ptr);

      static void operator delete[](void* ptr);

      static boolean setGrowSize(long grow_size);

      boolean clear(Integral::CMODE ctype_a);

class-specific public methods:

- parse methods:

      boolean parseExpression(const String& graph_start_a, const String& graph_term_a);

- set methods:

      boolean setExpression(String& expression_a, long line_a);

- get methods:

      DiGraph<String> getGraph();

      Vector<String> getSymbolList();

      String getGrammarName();

private methods:

- method to tokenize JSGF expressions:

      boolean tokenize(const String& expression_a, Queue<JSGFToken>& queue_a);

      boolean splitRuleLine(const String& arg_a, Queue<JSGFToken>& queue_a);

- method to draw graph:

      boolean drawGraph();

- method to replace JSGF rule extensions:

      boolean replace(Queue<JSGFToken> queue_a, long public_index_a);

- method to search starting symbols in JSGF rule:

      JSGFToken searchStart(Queue<JSGFToken>& arg_a, Queue<JSGFToken>& head_a);

- method to search ending vertices in graph:

      boolean searchEnd(GraphVertex<String>* current_a, GraphVertex<String>* destination_a, float weight_a);

examples:

- This example shows how to use JSGFParser class to parse a JSGF grammar and convert it to a ISIP DiGraph.

      // isip include files
      //
      #include <JSGFParser.h>

      // declare an JSGFParser
      //
      JSGFParser jp;

      // define symbols for the ISIP DiGraph default starting and ending vertices
      //
      String start(L"S");
      String term(L"T");

      // declare a simple JSGF grammar
      //
      String exp(L"public <rule> = <ISIP_JSGF_1_0_START> /0/ boy ( /0.2/ run | /0.8/ walk ) /0/ <ISIP_JSGF_1_0_TERM>;");

      // set expression
      //
      jp.setExpression(exp, 0));

      // parse expression
      //
      jp.parseExpression(start, term));

      // get symbol list of the graph converted from the JSGF grammar
      //
      Vector<String> symbol_list = jp.getSymbolList();

      // get the graph converted from the JSGF grammar
      //
      DiGraph<String> graph = jp.getGraph();

      // open file to store the symbol list and the graph
      //
      String tmp_filename(L"example_output.sof");
      Sof sof;

      if(!sof.open(tmp_filename, File::WRITE_ONLY)) {
        return Error::handle(tmp_filename, L"open", Error::TEST, __FILE__, __LINE__);
      }

      // write the symbol list to the file
      //
      symbol_list.write(sof, 0);

      // write the graph to the file
      //
      graph.write(sof, 0);

      // close the file
      //
      sof.close();

      ---------------------------------------------------------------------------
      This is the output SOF file that contains the symbol list and graph, where
      a vector of the symbols is listed under the tag @ Vector<String> 0 @ and
      the vertices and arcs of the graph under the tag @ DiGraph<String> 0 @:

      @ Sof v1.0 @
      @ Vector<String> 0 @
      values = {
        "boy"
      }, {
        "run"
      }, {
        "walk"
      };
      @ DiGraph<String> 0 @
      weighted = true;
      vertices =
      	{0, {"boy"}},
      	{1, {"run"}},
      	{2, {"walk"}};
      arcs =
      	{S, 0, 0, true},
      	{0, 2, 0.8, true},
      	{0, 1, 0.2, true},
      	{1, T, 0, true},
      	{2, T, 0, true};

notes:

- We currently support a limited subset of the JSGF functionality. We have mentioned below a comprehensive list of all the rules that we currently support in this class. A simple example for each rule is provided for easy understanding.
  - Grammar names and grammar name declaration:
    Each grammar has a unique name that is declared in the grammar header. The grammar's name must be declared as the first statement of the grammar.

        	      grammar network.grammar.sentence;

  - Rulenames and rule definitions:
    A grammar is composed of a set of rules that together define what may be spoken. Rules are combinations of speakable text and references to other rules. Each rule has a unique rulename. A reference to a rule is represented by the rule's name in surrounding <> characters. A rule is defined once in a grammar.

        	      <rulename> = one two three;

  - Tokens (terminal symbols) and sequences of tokens:
    Token is the part of a grammar that defines what may be spoken and that cannot be expanded. Tokens may appear in isolation or as sequences of tokens separated by white-space characters.

        	      this is a test

  - Rule expansions:
    Rule expansion are a reference to a token or a reference to a rule.

        	      <rule> = <expansion1> <expansion2>;

  - Sequences composition:
    A rule can be defined by a sequence of expansions.

        	      <sentence> = this is a test;

  - Alternatives composition:
    A rule can be defined by a set of alternative expansions separated by vertical bar charaters and white-spaces.

        	      <color> = red | green | blue;

  - Weight:
    Not all ways of speaking a grammar are equally likely. Weights may be attached to the elements of a set of alternatives to indicate the likelihood of each alternative being spoken.

        	      <color> = /0.2/ red | /0.5/ green | /0.3/ blue;

  - () parentheses grouping:
    Any legal expansion may be explicitly grouped using matching parentheses. Grouping has high precedence and so can be used to ensure the correct interpretation of rules.

        	      <action> = please ( open | close) the door;

  - + plus operator:
    A token or rule expansion followed by the plus symbol indicates the token or expansion may be spoken one or more times.

        	      <action> = run+ without stop;

  - Rule recursion:
    Rule recursion is the definition of a rule in terms of itself.

        	      <action> = repeat <action>;

  - Comments:

        	      // this is a comment

  - Self-identifying header:
    Header identifies that the document contains JSGF and indicates the version of JSGF being used. The hash character # must be the first character in the document.

        	      #JSGF V1.0;

- The examples shown below are from our initial speech experiments that demonstrate the usage of the various JSGF rules supported in this class.
  - The first example is a loop grammar for TIDIGITS experiment. It shows how to combine expansions with alternatives:

               #JSGF V1.0;
               // Define the grammar name
               grammar network.grammar.sentence;

               // Define the rules
               public <sentence> = <ISIP_JSGF_1_0_START> /0/ SILENCE <extension> /0/ SILENCE /0/ <ISIP_JSGF_1_0_TERM>;
               <extension> = /0/ ONE | /0/ TWO | /0/ THREE | /0/ FOUR | /0/ FIVE | /0/ SIX | /0/ SEVEN | /0/ EIGHT | /0/ NINE | /0/ OH | /0/ ZERO;

  - The second example describes a five states left-to-right with self-loop HMM topology for word ONE. It shows how to combine sequence of expansions with terminal recursions.

               #JSGF V1.0;
               // Define the grammar name
               grammar network.grammar.one;

               // Define the rules

               public <one> = <ISIP_JSGF_1_0_START> /0/ <A> /0/ <B> /0/ <C> /0/ <D> /0/ <E> /0/ <ISIP_JSGF_1_0_TERM>;
               <A> = /0/ S_1+;
               <B> = /0/ S_2+;
               <C> = /0/ S_3+;
               <D> = /0/ S_4+;
               <E> = /0/ S_5+;

- JSGF grammar is defined for speakable units of language such as words that do not require the need for dummy terminal symbols like START and END. However, we have extended the JSGF usage to specify non-speakable units such as model-topology that require the usage of these terminal symbols. ISIP DiGraph defines two default vertices to start and end graph. Similarly, for JSGF format, we have reserved two symbols to represent the start and end graph vertices.

             	 <ISIP_JSGF_1_0_START>
      	 <ISIP_JSGF_1_0_TERM>