(r23) GrmThraxForum < Forum

Forum Web>GrmThraxForum (revision 23) (raw view)~~EditAttach~~
---+ [[GRM.Thrax][OpenGrm Thrax]] Forum

You need to be a  registered user to participate in the discussions. <br /> 
%IF{"context authenticated" then='%MAKETEXT{"You are logged in as [_1]." args="[[%WIKIUSERNAME%][%SPACEOUT{%WIKINAME%}%]]"}%%BR%%LOGOUT%' else='%LOGIN% %MAKETEXT{"or"}% [[%TWIKIWEB%.TWikiRegistration][%MAKETEXT{"Register"}%]]'}%

%TOC{depth="2"}%

You can start a new discussion here:
<blockquote>
%ICON{help}% You can use the formatting commands describes in TWiki.TextFormattingRules in your comment. <br /> 
%ICON{tip}% If you want to post some code, surround it with =&lt;verbatim&gt;= and =&lt;/verbatim&gt;= tags. <br /> 
%ICON{warning}% Auto-linking of TWiki.WikiWords is now disabled in comments, so you can type !VectorFst and it won't result in a broken link.<br /> 
%ICON{warning}% You now need to use =&lt;br&gt;= to force new lines in your comment (unless inside =verbatim= tags). However, a blank line will automatically create a new paragraph. </blockquote> 
%COMMENT{type="newtopic"}%



---++ utils/utils.cc &#39;close&#39; not declared? 
---+++ Main.StevenBedrick - 2014-02-17 - 18:01
<noautolink>
Hello, Richard et al.-

While compiling Thrax 1.1 (against OpenFST 1.3.4 on an Ubuntu 13.10 system), I&#39;m getting the following compilation error:

&#60;pre&#62;
...
/bin/bash ../../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H   -I./../include    -g -O2 -MT utils.lo -MD -MP -MF .deps/utils.Tpo -c -o utils.lo `test -f &#39;util/utils.cc&#39; || echo &#39;./&#39;`util/utils.cc
libtool: compile:  g++ -DHAVE_CONFIG_H -I./../include -g -O2 -MT utils.lo -MD -MP -MF .deps/utils.Tpo -c util/utils.cc  -fPIC -DPIC -o .libs/utils.o
util/utils.cc: In function &#39;bool thrax::Readable(const string&)&#39;:
util/utils.cc:139:13: error: &#39;close&#39; was not declared in this scope
   close(fdes);
             ^
make[3]: *** [utils.lo] Error 1
make[3]: Leaving directory `/home/steven/thrax-1.1.0/src/lib&#39;
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/home/steven/thrax-1.1.0/src&#39;
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/steven/thrax-1.1.0&#39;
make: *** [all] Error 2

&#60;/pre&#62;

Any ideas what might be going on here?
</noautolink>


---+++ Main.StevenBedrick - 2014-02-17 - 18:02
<noautolink>
OK, having wiki formatting trouble. Trying the code snippet again:

&#60;verbatim&#62;
/bin/bash ../../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H   -I./../include    -g -O2 -MT utils.lo -MD -MP -MF .deps/utils.Tpo -c -o utils.lo `test -f &#39;util/utils.cc&#39; || echo &#39;./&#39;`util/utils.cc
libtool: compile:  g++ -DHAVE_CONFIG_H -I./../include -g -O2 -MT utils.lo -MD -MP -MF .deps/utils.Tpo -c util/utils.cc  -fPIC -DPIC -o .libs/utils.o
util/utils.cc: In function &#39;bool thrax::Readable(const string&)&#39;:
util/utils.cc:139:13: error: &#39;close&#39; was not declared in this scope
   close(fdes);
             ^
make[3]: *** [utils.lo] Error 1
make[3]: Leaving directory `/home/steven/thrax-1.1.0/src/lib&#39;
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/home/steven/thrax-1.1.0/src&#39;
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/steven/thrax-1.1.0&#39;
make: *** [all] Error 2
&#60;/verbatim&#62;
</noautolink>

---+++ Main.StevenBedrick - 2014-02-17 - 18:03
<noautolink>
Third time&#39;s the charm?
&#60;!-- &#60;pre&#62; --&#62;
/bin/bash ../../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H   -I./../include    -g -O2 -MT utils.lo -MD -MP -MF .deps/utils.Tpo -c -o utils.lo `test -f &#39;util/utils.cc&#39; || echo &#39;./&#39;`util/utils.cc
libtool: compile:  g++ -DHAVE_CONFIG_H -I./../include -g -O2 -MT utils.lo -MD -MP -MF .deps/utils.Tpo -c util/utils.cc  -fPIC -DPIC -o .libs/utils.o
util/utils.cc: In function &#39;bool thrax::Readable(const string&)&#39;:
util/utils.cc:139:13: error: &#39;close&#39; was not declared in this scope
   close(fdes);
             ^
make[3]: *** [utils.lo] Error 1
make[3]: Leaving directory `/home/steven/thrax-1.1.0/src/lib&#39;
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/home/steven/thrax-1.1.0/src&#39;
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/steven/thrax-1.1.0&#39;
make: *** [all] Error 2
&#60;!-- &#60;/pre&#62; --&#62;
</noautolink>

---+++ Main.StevenBedrick - 2014-02-17 - 18:04
<noautolink>
OK, this is ridiculous. Click here to see a Gist:

https://gist.github.com/stevenbedrick/809dbe2c921d745fbcc6
</noautolink>
%COMMENT{type="mythreadmode"}% 



---++ compilation fails 
---+++ Main.KyleGorman - 05 Nov 2013 - 14:53
<noautolink>
Hi Richard (etc.), using Thrax 1.1.0 (and with OpenFst 1.3.4 already installed), compilation fails while making the file `ast/identifier-node.cc` due to an issue in the `include/thrax/compat/utils.h` header. Here's the error:

/bin/sh ../../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H   -I./../include    -g -O2 -MT identifier-node.lo -MD -MP -MF .deps/identifier-node.Tpo -c -o identifier-node.lo `test -f 'ast/identifier-node.cc' || echo './'`ast/identifier-node.cc
libtool: compile:  g++ -DHAVE_CONFIG_H -I./../include -g -O2 -MT identifier-node.lo -MD -MP -MF .deps/identifier-node.Tpo -c ast/identifier-node.cc  -fno-common -DPIC -o .libs/identifier-node.o
In file included from ast/identifier-node.cc:22:
./../include/thrax/compat/utils.h:119:8: error: field has incomplete type
      'char []'
  char buf[];
       ^

I presume this is because buf[] doesn't have a length defined (nor is it initialized with a string), and when I change the line to

char buf[1024];

compilation goes through. (I'm not sure this is a sensible default; I spent no time trying to understand what this code is doing.)

I'd include a patch but it's one line.

Kyle
</noautolink>


---+++ Main.RichardSproat - 05 Nov 2013 - 16:38
<noautolink>
Just remove that line: that variable is not used. Apparently it's a holdover from some earlier implementation, and I just forgot to update it.  I'll fix this in the next release.
</noautolink>
%COMMENT{type="mythreadmode"}% 



---++ TEST 
---+++ Main.RichardSproat - 13 Sep 2013 - 12:16
<noautolink>
This is a test. Please ignore.
</noautolink>

%COMMENT{type="mythreadmode"}% 



---++  Recommended way to obtain FST+symbols for use 
---+++ Main.JosefNovak - 10 Jun 2013 - 09:46
<noautolink>
Hi, 

I am currently using thrax to extend my some features of an alignment tool I wrote for my g2p system.  

The basic idea is that the user can specify some alignment correspondence rules and optional default penalties, and then these can be incorporated into the EM training process.  

At present I have kind of hacked the functionality of the thraxcompiler command tool to read in the grammar, and then return the desired FST+symbol table to the alignment program.  

EDIT: Maybe it makes more sense to just provide a couple of snippets:

*GetFstFromGrammar*
<verbatim>
template <typename Arc>
VectorFst<Arc> GetFstFromGrammar(const string& input_grammar, const string& rules_name) {
  GrmCompilerSpec<Arc> grammar;
  VectorFst<StdArc> rules;
  if ( grammar.ParseFile(input_grammar) && grammar.EvaluateAst() ) {
    const GrmManagerSpec<Arc>* manager = grammar.GetGrmManager();
    FstMap fsts = manager->GetFstMap();
    for( typename FstMap::const_iterator it=fsts.begin();
         it != fsts.end(); ++it ){
      cout << "Echo: " << it->first << endl;
    }
    rules = *fsts[rules_name];
    return rules;
  }

  return rules;
}
</verbatim>

*toy.grm*
<verbatim>
sy = SymbolTable['simple.syms'];

zero  = "0".sy : "zero".sy;
units = ( "these're".sy : ( "these're".sy | "[these]" | "[these]" "are".sy ) );
split = ( "[these]" "are".sy : "these're".sy );
sigma = "<sigma>".sy : "<sigma>".sy;
abc   = ( "a".sy "b c".sy : "a b b".sy );
export RULES = Optimize[ sigma* ( units | zero | abc ) sigma* ];
</verbatim>

Here the 'sigma' is used in combination with a specialized 1-state alignment transducer that relies on RHO and SIGMA matchers.

Is there an alternative or recommended way to do this?  It would be great if I could either specify the symbol table just once at the beginning, or automatically infer/generate the whole symbol table and return it - or even better modify the grammar from my C++ application to simply what the user is responsible for doing.

I went through the FAQ but did not notice any answers to these questions.

Thanks for your time.

UPDATE:
 I solved this by creating some bindings with [[https://code.google.com/p/pybindgen/][pybindgen]] and then writing a generator that interprets a simplified version of the Thrax grammar, then expands it to the versbose version with the extra quotes and symfile suffixes, etc.
</noautolink>


%COMMENT{type="mythreadmode"}% 



---++ Need some help, New to "Thrax" 
---+++ Main.GoudjilKamel - 03 Jan 2013 - 17:29
<noautolink>
compiling under unbuntu LTS 12.04 : got the msg below at linking
libtool: link: g++ -g -O2 -o .libs/thraxcompiler compiler.o  -L/usr/local/lib/fst -lm -ldl -lfst /usr/local/lib/fst/libfstfar.so ../lib/.libs/libthrax.so -Wl,-rpath -Wl,/usr/local/lib/fst -Wl,-rpath -Wl,/usr/local/lib
../lib/.libs/libthrax.so: undefined reference to `fst::IsSTList(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
../lib/.libs/libthrax.so: undefined reference to `fst::IsSTTable(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
collect2: ld returned 1 exit status
</noautolink>


---+++ Main.RichardSproat - 29 Aug 2013 - 11:47
<noautolink>
Did you compile the fst library with the far extension?
</noautolink>

---+++ Main.DanXu - 08 Jan 2014 - 02:55
<noautolink>
I also have encountered the same problem with v1.1.0(compile export/batch_test), and compiled thrax with far enable.
</noautolink>

---+++ Main.RichardSproat - 08 Jan 2014 - 09:06
<noautolink>
Yes, but did you also compile the fst library with far enabled?
</noautolink>

---+++ Main.DanXu - 09 Jan 2014 - 09:53
<noautolink>
yes (openfst 1.3.4 compiled with --enable-far and some other enable options ), thrax compiled successfully,but compilation fails while making the file `batch_test.c` (extracted form export.tgz), can you me some advice
</noautolink>

---+++ Main.RichardSproat - 10 Jan 2014 - 09:11
<noautolink>
I'd like to but first I need to understand what is going on. I can't reproduce your error (apparently) and I don't know what batch_test.c is since it's not part of the Thrax distribution. Is this your own code? If so then I need to see EXACTLY what you are doing, including probably your sending me a directory with all of the additional code.

If this *is* part of the Thrax distribution then please tell me where it is because I can't find it (nor do I remember such a file).
</noautolink>

---+++ Main.DanXu - 11 Jan 2014 - 09:18
<noautolink>
thank you for your reply.

in this page:

http://openfst.cs.nyu.edu/twiki/bin/view/Contrib/ThraxContrib,  

you can see

 Projects using the OpenGrm Thrax tools:
 export.tgz: Grammars and software developed as part of a text  normalization class taught at the Center for Spoken Language Understanding, Fall 2011. URL for the course: http://www.cslu.ogi.edu/~sproatr/Courses/TextNorm/

i download "export.tgz" .
there is a file called batch_tester.cc in batch_tester directory(extract from export.tgz)&#12290;
</noautolink>

---+++ Main.RichardSproat - 12 Jan 2014 - 09:08
<noautolink>
Ok that helps. Yes, I did write that, but it wasn't obvious from your query that this is what you were referring to. Please in future give all necessary information when reporting a bug.

In the meantime I will have a look. I do not know off the top of my head what the problem is.
</noautolink>

---+++ Main.RichardSproat - 12 Jan 2014 - 13:40
<noautolink>
Ok it's the usual nonsense about ordering of shared object libraries. If you do things in this order it should work:

 g++ -g -O2 -o batch_tester batch_tester.o  -L/usr/local/lib/fst -lm -ldl -lfst -lthrax   -Wl,--rpath -Wl,/usr/local/lib/fst -Wl,--rpath -Wl,/usr/local/lib/fst -Wl,--rpath -Wl,/usr/local/lib /usr/local/lib/fst/libfstfar.so 

Evidently there is a bug in the configuration of the distribution that was not causing problems before, but is now. I will look into that, but in the meantime, please try linking manually as above.
</noautolink>

---+++ Main.DanXu - 14 Jan 2014 - 03:47
<noautolink>
it's ok using above command you wrote,thanks!
</noautolink>

---+++ Main.RichardSproat - 14 Jan 2014 - 09:01
<noautolink>
Ok good, I'll update the tar file. Not sure why it worked before and not now, but I won't think about that.
</noautolink>
%COMMENT{type="mythreadmode"}% 



---++ Weight semiring 
---+++ Main.LauriLyly - 21 Nov 2012 - 00:34
<noautolink>
So far I find thrax a very neat piece of software but I have two questions...

Can I somehow use probability semiring as weights, because it seems Thrax only allows specifying log and tropical semirings? How about the other ones... Or should I somehow postprocess the generated far file?

Another question: I tried to use "fstdraw" on a far file, but got: ERROR: FstHeader::Read: Bad FST header: example.far

Is this a version mismatch?
</noautolink>


---+++ Main.LauriLyly - 29 Nov 2012 - 07:34
<noautolink>
Sorry, obviously my bad as it's a far and not an fst file :p Still not too familiar. But the weight question still applies ;)
</noautolink>

---+++ Main.RichardSproat - 29 Nov 2012 - 10:07
<noautolink>
Sorry, I missed the earlier comment -- for some reason I didn't get email about it.  

Unfortunately the restriction to Log and Tropical is due to a similar restriction in the fst library: the real semiring does not come predefined. The best suggestion would be to use Tropical and then just do the obvious e^-cost conversion.
</noautolink>
%COMMENT{type="mythreadmode"}% 

Access control:
   * Set ALLOWTOPICCHANGE = Main.AllAuthUsersGroup

-- Main.CyrilAllauzen - 13 Aug 2012
Topic revision: r23 - 2014-02-17 - StevenBedrick
Forum