OpenFst Forum

You need to be a registered user to participate in the discussions.
ALERT! Note some registrations before May 2018 were lost due to a system failure. If affected, please re-register.
Log In or Register

You can start a new discussion here:

Help You can use the formatting commands describes in TextFormattingRules in your comment.
Tip, idea If you want to post some code, surround it with <verbatim> and </verbatim> tags.
Warning, important Auto-linking of WikiWords is now disabled in comments, so you can type VectorFst and it won't result in a broken link.
Warning, important You now need to use <br> to force new lines in your comment (unless inside verbatim tags). However, a blank line will automatically create a new paragraph.
Subject
Comment
Log In

pair weights in weight

AliiiiRezaaa - 2018-11-04 - 08:08

Hi all I want to compile fst that have multiple weight and draw it. how can I do that? for example my fst.txt is: 0 1 311 0 0,20,51 0 2 361 0 0,51,68 0 247 0 1 -1.09863,51,150 0 3 864 0 0,71,91 0 4 885 0 0,94,118 0 5 545 0 0,118,150

but it error: fstcompile fst.txt FATAL: FstCompiler: Bad weight = "0,20,51", source = index.1.txt, line = 1 ERROR: FstHeader::Read: Bad FST header: standard input

best regards

AliiiiRezaaa - 2018-11-04 - 08:11

0 1 311 0 0,20,51 0 2 361 0 0,51,68 0 247 0 1 -1.09863,51,150 0 3 864 0 0,71,91 0 4 885 0 0,94,118 0 5 545 0 0,118,150 0 6 818 0 0,150,209 0 7 0 0 -2.2998,57,239
Log In

ComposeFst Memory Allocation

DaniloDeOliveira - 2018-08-24 - 16:04

Hi, I’m trying to manage the memory usage of a ComposeFst object for a lazy decoder. I monitor the size of the vector of known states in VectorCacheStore, as well as the composition state table, both not garbage collected, with the following methods:

*Vector of known states:*<br>

  • Inside VectorCacheStore:<br>
<verbatim>size_t GetExtraAllocBytes() { return state_vec_.size() * sizeof(State *) ; }</verbatim>

*State table:*<br>

  • Inside ComposeFstImpl:<br>
<verbatim>size_t GetStateTableSize(){ return GetStateTable()->Size()*sizeof(StateTuple) ; }</verbatim>

I use the sum of the output of these two methods along with the garbage collector limit in order to estimate the total memory allocated by ComposeFst and be able to re-initialize the ComposeFst object periodically. However, when I monitor memory usage via the top command, it doesn’t match this estimation, it says I’m using more memory than that. Am I missing anything?

MichaelRiley - 2018-08-24 - 22:41

Are you accounting for the size of each State that is pointed to (not fixed since it contains a vector of arcs)? Also for vector based components you'd expect a non-trivial difference between their size() and capacity(). Fragmentation could be another issue.

openfst-1.6.8.tar.gz checksum changed

SimonPodlesny - 2018-08-02 - 06:55

Hello,

Last time, when I was deploying OpenFst library in my project I did a checksum for download due to a insecure connection. Now, when I wanted to deploy project again, checksum is different. Is it possible that content of library was changed after it was made available for download? Last time when I downloaded library, current version was 1.6.8 and I copied sha256checksum from website (ba5a36662635eb68c202c0133d6137575342a5d507c2875fb0c859c5f199ead9), now sha256checksum is: af3f69ad3e32363e3a8c0e5953396ed35ee3130a0f27264b005879aeefd43236

KyleGorman - 2018-08-25 - 18:57

I don't know, but can you just use the more recent one that's up?
Log In

Implementing FST composition with "slop"

PiotrZelasko - 2018-06-29 - 11:40

Hi there!

Is there an option to perform FST composition with OpenFST which would allow some kind of "slop" parameter, similarly as in ElasticSearch (see the first example in attached link)? If not, what do you think would be the best approach to implement it? I'm thinking about lookahead matchers but perhaps there is a simpler way.

Reference: https://www.elastic.co/guide/en/elasticsearch/guide/current/slop.html

KyleGorman - 2018-07-12 - 00:09

I don't know how "slop" is implemented there but an obvious implementation of fuzzy matching with FSTs is described under FstExamples (see the section entitled "Edit Distance"). Here's an implementation using our Python wrappers:

https://github.com/kylebgorman/EditTransducer

That allows you to match between two strings of possibly infinite distance apart. If you only want to consider something a match when it's up to k characters (symbols) away, then there's an easier approach and it's in fact much faster at runtime: instead of composing with a cyclic edit transducer, you compose with a transducer that allows zero or one edits, and compose that with itself up to k times.

Log In

Binaries exit silently reading unknown FST weight type

KirillKatsnelson - 2018-05-30 - 08:26

Any binary that reads an .fst file (fstinfo, for example) just silently exits with exit code 1 when the weight type in the file is unknown. I think this started to happen after version 1.6.3.

The code in script/fst-class.cc checks if the returned reader function pointer is null, but for an unknown arc_type the returned pointer is to the NullReader function, which returns a null pointer when called.

The binary's main function then just exits without invoking the FSTERROR().

KyleGorman - 2018-06-13 - 20:49

Thanks, fixed.
Log In

Avoiding use of class values during backoff

JackNaru - 2018-05-22 - 15:07

This may be a simple question, but wanted to understand how we could avoid values from classes being used during backoff?

Log In

missing PROGRAM FLAGS in openfst-1.6.7 on cygwin

RolandSchwarz - 2018-05-18 - 07:30

Hi,

I've tried hard over the last days to compile openfst 1.6.7 on Cygwin and managed to get it to compile and link without error. However, when I try to use the command line tools, no 'PROGRAM FLAGS' are accessible. Here's (part of) the output from e.g. fstcompile:

fstcompile --help Creates binary FSTs from simple text format.

Usage: fstcompile [text.fst [binary.fst]]

PROGRAM FLAGS:

LIBRARY FLAGS:

Flags from: flags.cc --help: type = bool, default = false show usage information --helpshort: type = bool, default = false show brief usage information --tmpdir: type = string, default = "/tmp" temporary directory --v: type = int32, default = 0 verbosity level

Flags from: fst.cc --fst_align: type = bool, default = false Write FST data aligned where appropriate --fst_default_cache_gc: type = bool, default = true Enable garbage collection of cache --fst_default_cache_gc_limit: type = int64, default = 1048576 Cache byte size that triggers garbage collection --fst_read_mode: type = string, default = "read" Default file reading mode for mappable files --fst_verify_properties: type = bool, default = false Verify FST properties queried by TestProperties --save_relabel_ipairs: type = string, default = "" Save input relabel pairs to file --save_relabel_opairs: type = string, default = "" Save output relabel pairs to file

[...]

PROGRAM FLAGS are empty, preventing me to hand e.g. a symbol table to the binary. Interestingly, all flags that come from the library (fst.cc, symbol_table.cc, util.cc and weight.cc) are there.

Any help would be appreciated, I tried a lot already but can't make sense of it.

Thanks,

Roland

RolandSchwarz - 2018-05-18 - 16:12

From some preliminary debugging it seems that the registers which hold the flags change. For example if

auto bool_register = FlagRegister<bool>::GetRegister();

returns the bool FlagRegister at address X in fstcompile.cc, it also returns X in fstcompile-main.cc, but as soon as 'ShowUsage' in flags.cc is called the same command returns a different address Y.

So it seems the flags specific for fstcompile for example are put in a different register than the library flags and are never retrieved again.

KyleGorman - 2018-05-23 - 11:39

We are aware of this issue and it is fixed in an upcoming version.
Log In

why are we loosing 1 bit in expression of labels and states ?

PtiZoom - 2018-04-26 - 18:27

can someone explain why keys, indexes and states are not systematically unsigned ? thus we are loosing (MAX_UINT/2 -1) addressed data ? thx.

KyleGorman - 2018-05-23 - 11:43

The constants fst::kNoStateId and fst::kNoLabel, for instance, are both -1, and negative values should be treated as implementational details.

But the entire library is templated on a definition of arc so if you don't like this you could always write your own arc template like so:

template <class Weight> struct PreciseArcTpl { ssize_t ilabel; ssize_t olabel; Weight weight; ssize_t nextstate;

// yr constructor goes here. };

But IMO `int` tends to have enough precision for all but truly gigantic machines.

Log In

RandGen() with LogProbArcSelector question

KennethRBeesley - 2018-04-26 - 17:00

I have an FST with StdArc (Tropical Semiring). Using RandGen() with the default UniformArcSelector, I seem to get uniformly random results, as expected.

But when I try to use the LogProbArcSelector, e.g.

<verbatim> int seed = rand() % 57 ; // std::cout << "Rand: " << seed << std::endl; fst::LogProbArcSelector<fst::StdArc> selector(seed); fst::RandGenOptions< fst::LogProbArcSelector<fst::StdArc> > options(selector); fst::RandGen(fst, &random_path, options); </verbatim>

then it compiles and runs, but the results are not as expected. (I expect to get weighted-random results, biased by the relative weights of the paths in the FST.)

1. Am I thinking correctly about the LogProbArcSelector and what it's supposed to do? 2. Am I using it correctly? Does it work with StdArc FSTs? 3. Are there useful examples anywhere that I could study?

KennethRBeesley - 2018-04-26 - 17:10

I see that <verbatim>...</verbatim> didn't work very well. Here's another try with Markdown

I have an FST with StdArc (Tropical Semiring). Using RandGen() with the default UniformArcSelector, I seem to get uniformly random results, as expected. But when I try to use the LogProbArcSelector, e.g.

``` int seed = rand() % 57 ; // std::cout << "Rand: " << seed << std::endl; fst::LogProbArcSelector<fst::StdArc> selector(seed); fst::RandGenOptions< fst::LogProbArcSelector<fst::StdArc> > options(selector); fst::RandGen(fst, &random_path, options); ``` then it compiles and runs, but the results are not as expected. (I expect to get weighted-random results, biased by the relative weights of the paths in the FST.)

``` 1. Am I thinking correctly about the LogProbArcSelector and what it's supposed to do? 2. Am I using it correctly? Does it work with StdArc FSTs? 3. Are there useful examples anywhere that I could study? ```

Log In

building 1.6.7 problem

AlexanderRudnicky - 2018-03-20 - 10:50

I am getting the following error when I 'make' the project:

<verbatim> fst.cc:86:34: required from here ./../include/fst/util.h:202:3: error: unable to deduce 'const auto&' from '<expression error>' make[3]: * [fst.lo] Error 1 </verbatim>

I'm doing a vanilla build, with the following: <verbatim> wget http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.7.tar.gz

./configure CXX=g++47 --enable-static --enable-shared --enable-far --enable-ngram-fsts [also-->] ./configure CXX=g++47 make -j 4 [also-->] make </verbatim> The machine has: <verbatim> $cat /etc/*elease CentOS release 6.2 (Final) $g++47 -dumpversion 4.7.0 </verbatim>

I would want to assume that the distribution should have compiled w/o problem. I'm not a C++ person, so I'm not even sure what the error is about. Any ideas on what to try next? Thanks!

AlexanderRudnicky - 2018-03-20 - 11:01

hmm. The markup instructions above aren't clear. here's the error again: <verbatim> fst.cc:86:34: required from here = ./../include/fst/util.h:202:3: error: unable to deduce 'const auto&' from '<expression error>'= = make[3]: * [fst.lo] Error 1 = </verbatim>

KyleGorman - 2018-03-26 - 13:20

I haven't seen that before. I understand what it's saying (it's saying that some type deduction is failing on that line in util.h) but not why your compiler can't do type deduction there. Is it possible to upgrade your GCC beyond 4.7.0 (either by upgrading a package or your OS)? That version of GCC is 6 years old, which is a long time in C++ terms.

Successful cross-compilation of OpenFST, OpenGRM NGram and OpenGRM Thrax using MinGW

WincentBalin - 2018-03-15 - 16:31

Hello world! smile

I succeeded in compilation of the packages OpenFST/NGram/Thrax with MinGW using Docker. The will find the Git repository here: https://github.com/wincentbalin/compile-static-openfst The resulting MinGW binaries are static, both for win32 and for win64. You will find them in the list of releases: https://github.com/wincentbalin/compile-static-openfst/releases

I had to create a couple of patches, which I then put into the repository above. Some of them got obsolete already, and hence deleted. I hope that we might incorporate some of them into the main source code. I suppose it is much more feasible than trying to fork and adapt every single version to MSVC only to abandon it later. There are much too much of such repositories on GitHub.

I am looking forward to any question or opinion!

WincentBalin - 2018-03-15 - 16:34

Another question: may I post a link to this thread to the NGram and Thrax forums?

KyleGorman - 2018-05-23 - 11:45

Sure.
Log In

Linear FST error with other fst operation

MinseokKeum - 2018-03-07 - 03:36

Hello,

I tried to use linear fst extension.

I executed the below example as shown in http://www.openfst.org/twiki/bin/view/FST/FstExtensions.

fstlinear -vocab=vocab.txt -out=out.fst -start_symbol=NULL -end_symbol=NULL model1.txt model2.txt

It generated out.fst, but it cannot be accessed with fstinfo, or fstprint. It outputs error as below.

ERROR: GenericRegister::GetEntry: linear_tagger-fst.so: cannot open shared object file: No such file or directory ERROR: Fst::Read: Unknown FST type linear-tagger (arc type = standard): out.fst

I executed ldconfig, but cannot fix the problem. Is there any report on this problem?

KyleGorman - 2018-03-07 - 16:33

Find where linear_tagger-fst.so is located (probably something like /usr/local/lib/fst) and add it to your LD_LIBRARY_PATH. Then this should work. E.g.:

LD_LIBRARY_PATH=/usr/local/lib/fst fstinfo out.fst

MinseokKeum - 2018-03-08 - 01:31

Ok! Your advice solved the problem. There have been already a lot of LD_LIBRARY_PATH solution in this forum. Thank you.
Log In

Pynini and destructive operations

TanelAlumae - 2018-02-21 - 08:05

Hello,

I am trying to use Pynini in an NLP course for teaching FSTs. I find it great but I am a bit confused by the destructive operations in the Pynini API.

Let's say I have the following FSTs (inspired by Jason Eisner's homework at https://www.cs.jhu.edu/~jason/465/hw-ofst/hw-ofst.pdf):

import pynini as pn zero = pn.a("0") one = pn.a("1") bit = zero | one first = (zero + one + one + one + one.ques).optimize()

Now, a good way to rewrite 'first' seems to be something like:

first = (zero + one.closure(3,4)).optimize()

However, after executing this, 'one' is not any more /1/ but /1{3,4}/. I find this unintuitive. OK, I can use the copy method:

first = (zero + one.copy().closure(3,4)).optimize()

But is there a better way to do this, without using excessive calls to copy()?

KyleGorman - 2018-03-06 - 16:44

Hi Tanel, for all nearly all destructive methods of the form:

f.method(*args, **kwargs) # optionally returns `self`

there is an equivalent non-destructive form:

g = method(f, *args, **kwargs)

This just calls copy under the hood, of course.

BTW calling copy is not itself expensive. The major data classes in this library use reference-counting, copy-on-write semantics. The only time a deep copy happens is when you mutate something with a reference count > 1.

Log In

find N unique ShortestPaths for StdVectorFst using c++

VarunKumar - 2018-02-15 - 09:48

I am trying to find N unique shortest paths for a given StdVectorFst. What should be the value of "weight_threshold" for StdVectorFst in ShortestPathOptions?

fst::StdVectorFst* input = fst::StdVectorFst::Read(""); fst::StdVectorFst result;

//shortest path options fst::QueueType queue_type = fst::AUTO_QUEUE; const s::ShortestPathOptions shortest_path_opts(queue_type, n, true, fst::kDelta, weight_threshold, fst::kNoStateId);

KyleGorman - 2018-03-06 - 16:47

You should only set the weight threshold to something other than semiring zero if you also want to prune paths. See the documentation for Prune which goes over the associated semantics of pruning with a weight threshold.
Log In

weight type conversion in fst

IkeYuki - 2017-12-10 - 23:08

Is there a method to convert fst's weight? I want to get VectorFst<LogArc> from VectorFst<StdArc> such as HCLG.fst.

IkeYuki - 2017-12-15 - 07:19

I tried to use Arcmap, but an error occured. <verbatim> fst::MutableFst<fst::LogArc> *fst; fst::ArcMap<fst::LogArc,fst::WeightConvertMapper<fst::StdArc,fst::LogArc>>(*fst,fst::WeightConvertMapper<fst::StdArc,fst::LogArc>()); </verbatim>

<verbatim> No matching function for call to ArcMap(fst::MutableFst<fst::ArcTpl<fst::LogWeightTpl<float> > >&, fst::WeightConvertMapper<fst::ArcTpl<fst::TropicalWeightTpl<float> >, fst::ArcTpl<fst::LogWeightTpl<float> > >)' </verbatim>

Any ideas for this? Thanks for advance.

KyleGorman - 2018-01-16 - 16:48

Hi! Yes, such methods exist at several levels. At the template library level, try something like (these aren't tested but should work, with at most minor changes):

 
fst::VectorFst<StdArc> std_fst;
// Populates this...
fst::VectorFst<LogArc> log_fst;
fst::ArcMap(std_fst, &log_fst, fst::StdToLogMapper());

Or, you can use the binary interface:

fstmap --map_type=to_log std.fst log.fst

Or, from Python:

log_fst = pywrapfst.arcmap(std_fst, map_type="to_log")

Log In

Intersect operation gives error on symbol tables mismatch

SanJoshi - 2017-11-16 - 04:31

I want to check if a string exists in a set of strings. I created two FSTs

The first FST contains the set of all strings as shown here http://www.openfst.org/twiki/bin/view/Forum/FstForum#Making_an_FST_with_C_How_to_defi

The second SearchFST contains a single string (i.e. State1 -> a/x -> State2 -> b/y -> FinalState)

The input and output symbol tables for both FST are the same.

Now I sorted both the FSTs, the first on output labels, the second on input labels

ArcSort(&set_fst, StdOLabelCompare()) ArcSort(&search_fst, StdILabelCompare())

When I call StdIntersectFst output_fst(set_fst, search_fst)

I get this error WARNING: CompatSymbols: Symbol table checksums do not match. Table sizes are 3 and 3 FATAL: ComposeFst: Output symbol table of 1st argument does not match input symbol table of 2nd argument

Slide 28 here http://www.openfst.org/twiki/pub/FST/FstSltTutorial/part1.pdf seems to depict that both FSTs in an Intersect have the same symbol table.

What am I doing wrong ?

Log In

Get raw arc arrays from VectorFst or ArcIterator.

JustinLuitjens - 2017-11-09 - 16:45

Is there a method to get the raw arc arrays from either an Fst or an ArcIter? I'd like to be able to parallelism across an ArcIterator loop but to do that I need a way to get each arc in constant time.

KyleGorman - 2017-11-21 - 11:53

There's no guarantee that an FST type will have a precomputed arc array, so I'd just do something like:

std::vector<Arc> arcs; for (ArcIterator<Fst<Arc>> aiter(fst, state); aiter.Done(); aiter.Next()) { arcs.push_back(aiter.Value()); }

Not sure that answers your question though.

Log In

Error in fstconvert in openfst-1.6.5

EricWang - 2017-11-06 - 07:14

Hi, I have tried to run "./tools/openfst/openfst-1.6.5/bin/fstconvert --fst_type=olabel_lookahead --save_relabel_opairs=tmp.1 tmp.111> tmp.222" I got a error info as follows: ERROR: GenericRegister::GetEntry: lookup failed in shared object: olabel_lookahead-fst.so FATAL: Fst::Convert: Unknown FST type olabel_lookahead (arc type log)

But If I run above command using fstconvert in openfst-1.1.2.10, It is ok! I guess there some wrong in compiling fstconvert in openfst-1.6.5, but I don't know how to fix it. Could anyone give some helps? Thanks!

Eric

KyleGorman - 2017-11-21 - 11:55

You need to make sure you build lookahead FST SOs during compilation (`--enable-lookahead-fsts`) and that the generated SOs are in your LD_LIBRARY_PATH, or `fstconvert` won't be able to find them.

(1.1.2 is truly ancient---can't help you with why this would have worked 9 years ago.)

Log In

python minimize allow_nondet

DavidVanLeeuwen - 2017-11-03 - 12:05

Hello,

I think there is a small bug in pywrapfst, the parameter `allow_nondet` seems not to be transferred to the underlying c-call. If openfst's development were organized on github, I'd sent in a pull request.

Here is a diff that I think shows the bug, however, I can't get it working as I don't understand the build process. There is a pywrap.cc file that even has a "CYTHON_UNUSED" or something in a relevant place, but how the file is built is beyond my grasp. Maybe someone can have a look at this, that would be great.

Cheers,


david

* pywrapfst.pyx-orig 2017-11-03 14:43:48.072896578 +0100


pywrapfst.pyx 2017-11-03 14:44:04.829572772 +0100 ************* * 2019,2025 ** Returns: self. """ ! self._minimize(delta) return self

cpdef MutableArcIterator mutable_arcs(self, int64 state):


2019,2025 ---- Returns: self. """ ! self._minimize(delta, allow_nondet) return self

cpdef MutableArcIterator mutable_arcs(self, int64 state):

KyleGorman - 2017-11-03 - 15:35

Thanks, we'll fix that on our side. (A PR wouldn't be helpful because OpenFst's source is generated programmatically from proprietary code we develop within Google and I can't think of any serious alternative to that workflow that'd work for us.)

pywrapfst.cc is generated by Cython from pywrapfst.pyx, and then can be compiled normally. (We just build straight from the .cc though. Thus building this doesn't require that you have a copy of Cython, unless you want to hack the .pyx first.)

DavidVanLeeuwen - 2017-11-04 - 07:02

Thanks for the explanation.

I hack-fixed it for my own application now by changing a `(bool)0` into a `(bool)1` somewhere in the .cc.

I couldn't get an explicit determinize to work, the toy example

0 1 a a 0 2 a a 1 3 b b 2 3 b b 3

gives an empty result using cmdline openfst tools, and similarly for the pywrapfst interface.

Cheers,


david

KyleGorman - 2017-11-21 - 11:57

is the compiled FST from your toy example well-formed? (what does fstinfo say about it?)

Log In

python3 and configure

PavelDenisov - 2017-10-28 - 15:22

Running "./configure --enable-python" with python3 produces following error:

<verbatim> checking for a Python interpreter with version >= 2.7... python checking for python... /usr/bin/python checking for python version... 3.5 checking for python platform... linux checking for python script directory... ${prefix}/lib64/python3.5/site-packages checking for python extension module directory... ${exec_prefix}/lib64/python3.5/site-packages checking for python3.5... (cached) /usr/bin/python checking for a version of Python >= '2.1.0'... File "<string>", line 1 import sys, string; ver = string.split(sys.version)[0]; print ver >= '2.1.0' ^ SyntaxError: invalid syntax </verbatim>

It looks like python stuff in the configure script is not designed to work with python3. I would like to try to port it. Are there any reasons for not doing this?

KyleGorman - 2017-11-03 - 15:32

That error is coming from Python invoked by the autoconf macro, looks like. So we can fix that but maybe it's better to fix it upstream there.

Before submitting you may want to test Python 3 with the Python extension. We still develop on Python 2.7, exclusively, unfortunately, and it's not tested on Python 3.

PavelDenisov - 2017-11-04 - 12:25

OK, thank you. I'll keep that in mind.

ChristopherKermorvant - 2017-11-07 - 05:38

Hi Pavel, Do you have a version of ./configure working with python3 ? I am also trying to install pywrapfst with python3 Thank you

PavelDenisov - 2017-11-08 - 18:02

I was able to overcome the configure problem by taking ax_python_devel.m4 from fresh autoconf-archive distribution.

However other things must be taken care of before pywrapfst can be used with python3. At the moment I'm getting such error:

<verbatim>

pywrapfst.cc: In function 'void __Pyx_CppExn2PyErr()': pywrapfst.cc:2448:7: error: exception handling disabled, use -fexceptions to enable throw; ^~~~~ pywrapfst.cc:2450:40: error: 'exn' was not declared in this scope PyErr_SetString(PyExc_MemoryError, exn.what());

</verbatim>

NurL - 2018-03-14 - 09:27

After a lot of tweaks to the configuration file and Makefile I managed to successfully install the Python extension for Python 3. See this Dockerfile: https://gist.github.com/0xnurl/6f97eb39409ea48db31fe315fd1e208f
Log In

How recover state mapping after ShortestPath?

JeffToprak - 2017-09-17 - 12:41

I have the problem that I am extracting the shortest path of an FST via ShortestPath. However, the resulting FST has entirely new state IDs, and I would like have a map between the old state IDs and the new IDs (I have auxiliary information on the origin state IDs I want to carry over). How would I go about that? I guess I could try to "look up" the shortest path by recursively iterating through the original FST, matching ilabel/olabels , but is there a better way?

MichaelRiley - 2017-09-22 - 15:25

Assuming your FST is an FSA you could:

  1. make it an FST whose output label encoded the state/arc ID info OR
  2. You can intersect the FST with the input but pass the state table as an option first in IntersectFstOptions. The state table stores the mapping to and from the input state pairs and the output state.

Would be nice if Compose/IntersectFst offered GetStateTable() but issues with getting the right type of the return.

Log In

Pywrapfst API

AkuanLiu - 2017-09-13 - 06:41

Hi,

I'm having a bit of difficulty obtaining the API documentation for the Python wrapper of Openfst. I would like to use some more advanced methods such as epsilon removal and determinization, but am unsure as to how to call them.

Any help would be greatly appreciated. Thanks!

KyleGorman - 2017-10-12 - 17:25

Like most Python packages, they're documented extensively within. E.g.:

>>> import pywrapfst

>>> help(pywrapfst.determinize)

determinize(...)

determinize(ifst, delta=0.0009765625, det_type="functional", nstate=NO_STATE_ID, subsequential_label=0, weight=None, incremental_subsequential_label=False)

Constructively determinizes a weighted FST.

This operations creates an equivalent FST that has the property that no state has two transitions with the same input label. For this algorithm, epsilon transitions are treated as regular symbols (cf. `rmepsilon`).

Args:

ifst
The input FST.
delta
Comparison/quantization delta.
det_type
Type of determinization; one of: "functional" (input transducer is functional), "nonfunctional" (input transducer is not functional) and disambiguate" (input transducer is not functional but only keep the min of ambiguous outputs).
nstate
State number threshold.
subsequential_label
Input label of arc corresponding to residual final output when producing a subsequential transducer.
weight
A Weight or weight string indicating the desired weight threshold below which paths are pruned; if omitted, no paths are pruned.
increment_subsequential_label
Increment subsequential when creating several arcs for the residual final output at a given state.

Returns: An equivalent deterministic FST.

Raises:

FstArgError
Unknown determinization type.

See also: `disambiguate`, `rmepsilon`.

Log In

Python installation difficulties with both methods

PeterChoy - 2017-09-11 - 08:48

Hi all,

I am trying to use the python wrapper for openfst. I have tried both methods below, and cannot get either to work:

1. I configure using the configuration flag 'enable-python', run 'make' and then 'make install'. Running 'import pywrapfst' in Python gives me 'No module named pywrapfst'.

2. I run 'pip install openfst'. Running 'import pywrapfst' in Python gives me: 'ImportError: libfstfarscript.so.8: cannot open shared object file: No such file or directory'

I can't find anyone with the same issue: some help or ideas of how to troubleshoot would be deeply appreciated.

Thanks in advance! Peter

PeterChoy - 2017-09-11 - 08:58

P.S I am using Python 2.7, trying to install openfst-1.6.1

KyleGorman - 2017-09-11 - 10:09

If `make install` with enable-python ran, then there is somewhere a `pywrapfst.so` on your system. Find it, and make sure it's in your Python path (`sys.path` from within Python) for whatever Python installation you're using. (Many people have multiple installations of Python on their system.) You can do this by adjusting your Python path or moving pywrapfst.so into the existing path.

Using `pip` for this doesn't make much sense, and that's out of date anyways.

Log In

how to implement an on the fly fst replacement?

XingW - 2017-09-07 - 23:39

Dear all, I'm a newbie of openfst,nowadays I want to implement an on the fly fst replacement, could anyone teach me something about how to achieve the on the fly function in openfst? Thanks very much in advance!

KyleGorman - 2017-09-08 - 11:15

Construct a fst::ReplaceFst with your RTN specification, then pass it downstream to other algorithms which will expand it. Use the cache options struct (and/or command-line flags) to control how caching behaves.

XingW - 2017-09-08 - 23:07

Dear Kyle, thanks for your kindly reply, could you teach me something about on the fly? I know there is a command line named fstreplace, but I have no idea about how to implement the on the fly, is it means online?

KyleGorman - 2017-09-11 - 10:06

"on-the-fly" here means something similar to "lazy" in the sense of "lazy evaluation". It computes the FST corresponding to the RTN "as needed" by other operations; optionally, it stores recently visited states in the computation in a in-memory cache of size specified by the user. This is useful when the corresponding FST is large or only a small portion of it will be visited by subsequent operations.

You can't use lazy computation from the command-line, so `fstreplace` isn't really doing on-the-fly computation in any relevant sense. The only way to take advantage of lazy computation is through the lowest-level C++ API.

Log In

Error compiling OpenFst

LouNicotra - 2017-09-01 - 10:37

Hi, getting this error while running checks. Trying to install as a requisite for pyfst.

root@tiger21 openfst-1.6.3# ./configure --enable-python checking for a BSD-compatible install... /bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking for style of include used by make... GNU checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking whether gcc understands -c and -o together... yes checking dependency style of gcc... gcc3 checking for ar... ar checking the archiver (ar) interface... ar checking for g++... g++ checking whether we are using the GNU C++ compiler... yes checking whether g++ accepts -g... yes checking dependency style of g++... gcc3 checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking how to print strings... printf checking for a sed that does not truncate output... /bin/sed checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for fgrep... /bin/grep -F checking for ld used by gcc... /bin/ld checking if the linker (/bin/ld) is GNU ld... yes checking for BSD- or MS-compatible name lister (nm)... /bin/nm -B checking the name lister (/bin/nm -B) interface... BSD nm checking whether ln -s works... yes checking the maximum length of command line arguments... 1572864 checking whether the shell understands some XSI constructs... yes checking whether the shell understands "+="... yes checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop checking for /bin/ld option to reload object files... -r checking for objdump... objdump checking how to recognize dependent libraries... pass_all checking for dlltool... no checking how to associate runtime and link libraries... printf %s\n checking for archiver @FILE support... @ checking for strip... strip checking for ranlib... ranlib checking command to parse /bin/nm -B output from gcc object... ok checking for sysroot... no checking for mt... no checking if : is a manifest tool... no checking how to run the C preprocessor... gcc -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking for dlfcn.h... yes checking for objdir... .libs checking if gcc supports -fno-rtti -fno-exceptions... no checking for gcc option to produce PIC... -fPIC -DPIC checking if gcc PIC flag -fPIC -DPIC works... yes checking if gcc static flag -static works... no checking if gcc supports -c -o file.o... yes checking if gcc supports -c -o file.o... (cached) yes checking whether the gcc linker (/bin/ld -m elf_x86_64) supports shared libraries... yes checking whether -lc should be explicitly linked in... no checking dynamic linker characteristics... GNU/Linux ld.so checking how to hardcode library paths into programs... immediate checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes checking whether to build shared libraries... yes checking whether to build static libraries... no checking how to run the C++ preprocessor... g++ -E checking for ld used by g++... /bin/ld -m elf_x86_64 checking if the linker (/bin/ld -m elf_x86_64) is GNU ld... yes checking whether the g++ linker (/bin/ld -m elf_x86_64) supports shared libraries... yes checking for g++ option to produce PIC... -fPIC -DPIC checking if g++ PIC flag -fPIC -DPIC works... yes checking if g++ static flag -static works... no checking if g++ supports -c -o file.o... yes checking if g++ supports -c -o file.o... (cached) yes checking whether the g++ linker (/bin/ld -m elf_x86_64) supports shared libraries... yes checking dynamic linker characteristics... (cached) GNU/Linux ld.so checking how to hardcode library paths into programs... immediate checking for a Python interpreter with version >= 2.7... python checking for python... /var/local/miniconda2/bin/python checking for python version... 2.7 checking for python platform... linux2 checking for python script directory... ${prefix}/lib/python2.7/site-packages checking for python extension module directory... ${exec_prefix}/lib/python2.7/site-packages checking for python2.7... (cached) /var/local/miniconda2/bin/python checking for a version of Python >= '2.1.0'... yes checking for a version of Python >= '2.7'... yes checking for the distutils Python package... yes checking for Python include path... -I/var/local/miniconda2/include/python2.7 checking for Python library path... -L/var/local/miniconda2/lib/python2.7 -lpython2.7 checking for Python site-packages path... /var/local/miniconda2/lib/python2.7/site-packages checking python extra libraries... -lpthread -ldl -lutil checking python extra linking flags... -Xlinker -export-dynamic checking consistency of all components of python development environment... yes checking for dlopen in -ldl... yes checking that generated files are newer than configure... done configure: creating ./config.status config.status: creating Makefile config.status: creating src/Makefile config.status: creating src/include/Makefile config.status: creating src/lib/Makefile config.status: creating src/bin/Makefile config.status: creating src/test/Makefile config.status: creating src/extensions/Makefile config.status: creating src/extensions/compact/Makefile config.status: creating src/extensions/compress/Makefile config.status: creating src/extensions/const/Makefile config.status: creating src/extensions/far/Makefile config.status: creating src/extensions/linear/Makefile config.status: creating src/extensions/lookahead/Makefile config.status: creating src/extensions/mpdt/Makefile config.status: creating src/extensions/ngram/Makefile config.status: creating src/extensions/pdt/Makefile config.status: creating src/extensions/python/Makefile config.status: creating src/extensions/special/Makefile config.status: creating src/script/Makefile config.status: creating config.h config.status: config.h is unchanged config.status: creating src/include/fst/config.h config.status: src/include/fst/config.h is unchanged config.status: executing depfiles commands config.status: executing libtool commands

make gave some warnings eg: ./../../include/fst/string.h:207:59: warning: 'TokenType' is deprecated: Use fst::StringTokenType [-Wdeprecated-declarations] const SymbolTable *syms = nullptr)

Then checks fails

root@tiger21 openfst-1.6.3# make check Making check in src make[1]: Entering directory `/tmp/openfst-1.6.3/src' Making check in include make[2]: Entering directory `/tmp/openfst-1.6.3/src/include' make[2]: Nothing to be done for `check'. make[2]: Leaving directory `/tmp/openfst-1.6.3/src/include' Making check in lib make[2]: Entering directory `/tmp/openfst-1.6.3/src/lib' make[2]: Nothing to be done for `check'. make[2]: Leaving directory `/tmp/openfst-1.6.3/src/lib' Making check in script make[2]: Entering directory `/tmp/openfst-1.6.3/src/script' make[2]: Nothing to be done for `check'. make[2]: Leaving directory `/tmp/openfst-1.6.3/src/script' Making check in bin make[2]: Entering directory `/tmp/openfst-1.6.3/src/bin' make[2]: Nothing to be done for `check'. make[2]: Leaving directory `/tmp/openfst-1.6.3/src/bin' Making check in test make[2]: Entering directory `/tmp/openfst-1.6.3/src/test' make fst_test weight_test algo_test_log algo_test_tropical algo_test_minmax algo_test_lexicographic algo_test_power make[3]: Entering directory `/tmp/openfst-1.6.3/src/test' make[3]: `fst_test' is up to date. make[3]: `weight_test' is up to date. make[3]: `algo_test_log' is up to date. make[3]: `algo_test_tropical' is up to date. make[3]: `algo_test_minmax' is up to date. make[3]: `algo_test_lexicographic' is up to date. make[3]: `algo_test_power' is up to date. make[3]: Leaving directory `/tmp/openfst-1.6.3/src/test' make check-TESTS make[3]: Entering directory `/tmp/openfst-1.6.3/src/test' make[4]: Entering directory `/tmp/openfst-1.6.3/src/test' FAIL: fst_test FAIL: weight_test FAIL: algo_test_log FAIL: algo_test_tropical FAIL: algo_test_minmax FAIL: algo_test_lexicographic FAIL: algo_test_power make[5]: Entering directory `/tmp/openfst-1.6.3/src/test' make[5]: Nothing to be done for `all'. make[5]: Leaving directory `/tmp/openfst-1.6.3/src/test' ======================================================================== Testsuite summary for OpenFst 1.6.3 ======================================================================== # TOTAL: 7 # PASS: 0 # SKIP: 0 # XFAIL: 0 # FAIL: 7 # XPASS: 0 # ERROR: 0 ======================================================================== See src/test/test-suite.log Please report to help@www.openfst.org ======================================================================== make[4]: * [test-suite.log] Error 1 make[4]: Leaving directory `/tmp/openfst-1.6.3/src/test' make[3]: * [check-TESTS] Error 2 make[3]: Leaving directory `/tmp/openfst-1.6.3/src/test' make[2]: * [check-am] Error 2 make[2]: Leaving directory `/tmp/openfst-1.6.3/src/test' make[1]: * [check-recursive] Error 1 make[1]: Leaving directory `/tmp/openfst-1.6.3/src' make: * [check-recursive] Error 1 root@tiger21 openfst-1.6.3#

KyleGorman - 2017-09-08 - 11:17

That warning is harmless.

No clue about the errors in the tests. Can you run them manually and catch the first error they raise? (They're just executables.)

KyleGorman - 2017-09-11 - 10:23

BTW, the last I checked, pyfst doesn't support any version of OpenFst released in the last few years anyways, so you should check on their documentation for the supported version.
Log In

Pre-initialized composition

AhnK - 2017-08-31 - 22:07

Hi, The paper "Pre-initialized Composition for Large-Vocabulary Speech Recognition (Cyril Allauzen, et al, 2013)" mentioned the pre-initialized composition was implemented in the OpenFst toolkit. How can I use this feature? I can't find any explicit explanations on this.

AlexandreCaulier - 2017-12-20 - 11:46

Hello! I'm interested as well. Any idea where it is implemented ?

KyleGorman - 2018-05-24 - 01:33

"Implemented in OpenFst" in this case doesn't mean it's part of OpenFst, just that it was built on top of OpenFst. I'm not aware of an OSS implementation.
Log In

trouble installing Pynini on Mac OS X

AnnaKazantseva - 2017-08-21 - 17:34

Hello,

I am having a bit of trouble installing Pynini on a Mac OS X (El Capitan). I installed OpenFST and re2 from sources and I am using Anaconda python 2.7. When I run “sudo python setup.py install” everything goes smoothly. But when I run “python setup.py test” I keep getting this error:

<verbatim> sudo python setup.py test running test running egg_info writing pynini.egg-info/PKG-INFO writing top-level names to pynini.egg-info/top_level.txt writing dependency_links to pynini.egg-info/dependency_links.txt reading manifest file 'pynini.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'pynini.egg-info/SOURCES.txt' running build_ext copying build/lib.macosx-10.6-x86_64-2.7/pywrapfst.so -> copying build/lib.macosx-10.6-x86_64-2.7/pynini.so -> Traceback (most recent call last): File "setup.py", line 88, in <module> test_suite="pynini_test") File "/Users/kazantsevaa/anaconda/envs/python2/lib/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/Users/kazantsevaa/anaconda/envs/python2/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/Users/kazantsevaa/anaconda/envs/python2/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/Users/kazantsevaa/anaconda/envs/python2/lib/python2.7/site-packages/setuptools/command/test.py", line 172, in run self.run_tests() File "/Users/kazantsevaa/anaconda/envs/python2/lib/python2.7/site-packages/setuptools/command/test.py", line 193, in run_tests testRunner=self._resolve_as_ep(self.test_runner), File "/Users/kazantsevaa/anaconda/envs/python2/lib/python2.7/unittest/main.py", line 94, in init self.parseArgs(argv) File "/Users/kazantsevaa/anaconda/envs/python2/lib/python2.7/unittest/main.py", line 149, in parseArgs self.createTests() File "/Users/kazantsevaa/anaconda/envs/python2/lib/python2.7/unittest/main.py", line 158, in createTests self.module) File "/Users/kazantsevaa/anaconda/envs/python2/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames suites = [self.loadTestsFromName(name, module) for name in names] File "/Users/kazantsevaa/anaconda/envs/python2/lib/python2.7/unittest/loader.py", line 91, in loadTestsFromName module = __import__('.'.join(parts_copy)) File "/Users/kazantsevaa/Software/pynini-1.6/pynini_test.py", line 16, in <module> from pynini import * ImportError: dlopen(/Users/kazantsevaa/Software/pynini-1.6/pynini.so, 2): Symbol not found: __ZN3re23RE213GlobalReplaceEPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKS0_RKNS_11StringPieceE Referenced from: /Users/kazantsevaa/Software/pynini-1.6/pynini.so Expected in: flat namespace in /Users/kazantsevaa/Software/pynini-1.6/pynini.so </verbatim>

Any ideas how to fix this? Has anyone else run into this? I reinstalled openFST and re2 but it does not go away.

Thanks for any tips!

Anna

Log In

Error installing OpenFST under Ubuntu

KeesKoenen - 2017-08-17 - 14:02

Want to install OpenFST 1.6.2. Ran ./configure, make, make check. I get this output. Please advice. Thanks. PS Kind of a newbie at this, pls be gentle wink

<verbatim>fst_test.o: In function `fst::internal::CompactFstImpl<fst::ArcTpl<fst::TropicalWeightTpl<float> >, fst::(anonymous namespace)::CustomCompactor<fst::ArcTpl<fst::TropicalWeightTpl<float> > >, unsigned short, fst::DefaultCompactStore<std::pair<int, fst::TropicalWeightTpl<float> >, unsigned short>, fst::DefaultCacheStore<fst::ArcTpl<fst::TropicalWeightTpl<float> > > >::Init(fst::Fst<fst::ArcTpl<fst::TropicalWeightTpl<float> > > const&, std::shared_ptr<fst::DefaultCompactStore<std::pair<int, fst::TropicalWeightTpl<float> >, unsigned short> >)': fst_test.cc:(.text+0x3c35f): undefined reference to `fst::Int64ToStr(long, std::string*)

make[3]: Entering directory '/home/kkoenen/stt/kaldi-master/tools/openfst-1.6.2/src/test' /bin/bash ../../libtool --tag=CXX --mode=link g++ -std=c++11 -o fst_test fst_test.o ../lib/libfst.la -lm -ldl libtool: link: g++ -std=c++11 -o .libs/fst_test fst_test.o ../lib/.libs/libfst.so -lm -ldl -Wl,-rpath -Wl,/home/kkoenen/stt/kaldi-master/tools/openfst-1.6.2/lib Makefile:645: recipe for target 'fst_test' failed make[3]: Leaving directory '/home/kkoenen/stt/kaldi-master/tools/openfst-1.6.2/src/test' Makefile:1054: recipe for target 'check-am' failed make[2]: Leaving directory '/home/kkoenen/stt/kaldi-master/tools/openfst-1.6.2/src/test' Makefile:358: recipe for target 'check-recursive' failed make[1]: Leaving directory '/home/kkoenen/stt/kaldi-master/tools/openfst-1.6.2/src' Makefile:414: recipe for target 'check-recursive' failed </verbatim>

Same problem with openfst-1.6.3.

KyleGorman - 2017-08-17 - 17:06

We can fix this on our side but you don't need to run "make check", just "sudo make install" should suffice.

KyleGorman - 2017-08-18 - 15:03

Hi Kees, the error doesn't make a ton of sense to me. There is no function fst::Int64ToStr in OpenFst 1.6.3, as far as I can tell.

You may want to try doing a clean install (i.e., go to where OpenFst headers and source are installed and remove them, then install again).

You may want to try this outside of the Kaldi framework for installing OpenFst, since we don't maintain that, the Kaldi people do, and I don't know anything about it.

Log In

patch for failed cross-compile on openfst-1.6.3

BoonPangLim - 2017-08-03 - 14:02

Hi, I get an error linking when trying to compile using a cross-compiling toolchain:

<pre> make[4]: Entering directory '/scratch/bplim/crossbuild2/tmp/openfst-1.6.3/src/extensions/far' /bin/sh ../../../libtool --tag=CXX --mode=link armv8rpi3-novumind-linux-gnueabihf-g++ -std=c++11 -o farcompilestrings farcompilestrings.o libfstfarscript.la ../../script/libfstscript.la ../../lib/libfst.la -lm -ldl libtool: link: armv8rpi3-novumind-linux-gnueabihf-g++ -std=c++11 -o .libs/farcompilestrings farcompilestrings.o ./.libs/libfstfarscript.so ../../script/.libs/libfstscript.so ../../lib/.libs/libfst.so -lm -ldl -Wl,-rpath -Wl,/scratch/bplim/crossbuild2/tmp/build/rpi3/openfst/lib /home/share/crosstool/rpi3/lib/gcc/armv8rpi3-novumind-linux-gnueabihf/6.3.0/../../../../armv8rpi3-novumind-linux-gnueabihf/bin/ld: warning: libfstfar.so.8, needed by ./.libs/libfstfarscript.so, not found (try using -rpath or -rpath-link) ./.libs/libfstfarscript.so: undefined reference to `fst::IsSTList(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' ./.libs/libfstfarscript.so: undefined reference to `fst::IsSTTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' </pre>

I'm not sure if this is completely correct but this patch (followed by rerunning aclocal && automake && autoreconf) seems to fix it:

<pre>


openfst-1.6.3/src/extensions/far/Makefile.am 2017-06-27 13:12:19.000000000 -0700 +++ openfst-1.6.3/src/extensions/far/Makefile.new 2017-08-02 19:24:37.034355239 -0700

@@ -23,7 +23,7 @@ bin_PROGRAMS = farcompilestrings farcreate farequal farextract farinfo farisomorphic farprintstrings

-LDADD = libfstfarscript.la ../../script/libfstscript.la +LDADD = libfstfarscript.la libfstfar.la ../../script/libfstscript.la ../../lib/libfst.la -lm $(DL_LIBS)

farcompilestrings_SOURCES = farcompilestrings.cc </pre>

BoonPangLim - 2017-08-03 - 14:05

Update with cleaner versions:

ERROR: <verbatim> make[4]: Entering directory '/scratch/bplim/crossbuild2/tmp/openfst-1.6.3/src/extensions/far' /bin/sh ../../../libtool --tag=CXX --mode=link armv8rpi3-novumind-linux-gnueabihf-g++ -std=c++11 -o farcompilestrings farcompilestrings.o libfstfarscript.la ../../script/libfstscript.la ../../lib/libfst.la -lm -ldl libtool: link: armv8rpi3-novumind-linux-gnueabihf-g++ -std=c++11 -o .libs/farcompilestrings farcompilestrings.o ./.libs/libfstfarscript.so ../../script/.libs/libfstscript.so ../../lib/.libs/libfst.so -lm -ldl -Wl,-rpath -Wl,/scratch/bplim/crossbuild2/tmp/build/rpi3/openfst/lib /home/share/crosstool/rpi3/lib/gcc/armv8rpi3-novumind-linux-gnueabihf/6.3.0/../../../../armv8rpi3-novumind-linux-gnueabihf/bin/ld: warning: libfstfar.so.8, needed by ./.libs/libfstfarscript.so, not found (try using -rpath or -rpath-link) ./.libs/libfstfarscript.so: undefined reference to `fst::IsSTList(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' ./.libs/libfstfarscript.so: undefined reference to `fst::IsSTTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' collect2: error: ld returned 1 exit status

</verbatim>

PATCH: <verbatim>


openfst-1.6.3/src/extensions/far/Makefile.am 2017-06-27 13:12:19.000000000 -0700 +++ openfst-1.6.3/src/extensions/far/Makefile.new 2017-08-02 19:24:37.034355239 -0700

@@ -23,7 +23,7 @@ bin_PROGRAMS = farcompilestrings farcreate farequal farextract farinfo farisomorphic farprintstrings

-LDADD = libfstfarscript.la ../../script/libfstscript.la +LDADD = libfstfarscript.la libfstfar.la ../../script/libfstscript.la ../../lib/libfst.la -lm $(DL_LIBS)

farcompilestrings_SOURCES = farcompilestrings.cc </verbatim>

Log In

Pre-determinization

MarinaPetkovic - 2017-07-31 - 04:52

Hi, I was wondering if there is an openfst implementation of an algorithm described in the paper "An Optimal Pre-Determinization Algorithm for Weighted Transducers" by Allauzen and Mohri?

KyleGorman - 2017-08-14 - 13:47

No sorry, but you could make your own?

There is (was?) an implementation in the old AT&T FSM tools.

Log In

How to properly link my own program with openfst's libfst.so?

WeiyiZheng - 2017-07-26 - 19:31

I encountered this problem when using fst/fst.h and libfst.so with my own c++ code. not sure what I did wrong, does anyone have a hint?

And this is the error <verbatim>

E0726 16:23:20.319480 640899 generic-register.h:71] GenericRegister::GetEntry: vector-fst.so: cannot open shared object file: No such file or directory E0726 16:23:20.319733 640899 fst.h:246] Fst::Read: Unknown FST type vector (arc type = standard): <unspecified> </verbatim>

JigarGada - 2017-08-02 - 20:31

Can you try using <fst/fstlib.h> istead?
Log In

How to static link openFST with -fPIC?

AdamZahran - 2017-07-18 - 06:30

Hi, I'd like to compile openFST to produce a static link that would later be linked into a shared library. So I need to compile it with the -fPIC flag, but automake is really confusing and I don't know what to edit to accomplish that. Halp! BTW thank you for the amazing library <3 big grin

Log In

Label IDs for matchers

JigarGada - 2017-07-10 - 19:04

I want to use sigma matcher and add it to isyms.txt. Is there a special integer reserved for all the matchers like we have 0 reserved for epsilon?

KyleGorman - 2017-07-24 - 12:01

No, it just needs to not be 0 and to not overlap any other non-"special" labels. Sometimes I use 1, 2, 3 for phi, sigma, rho. sometimes I use -1, -2, -3 (though we don't do much testing for negative labels so bad things could happen).

KyleGorman - 2017-07-24 - 12:01

BTW, if you're working with fstspecial, there are specific flags for setting the special labels for the different types of matchers (you also have to select the matcher to convert to).

JigarGada - 2017-08-02 - 20:29

I selected the fst::RhoMatcher as the matcher type and it works with whatever label I provide for <rho>. Thanks Kyle

BillByrne - 2017-08-29 - 08:19

Hi, Can all this be done at the command line? I have a simple example with two acceptors: > fstprint --acceptor a.fst 0 1 1 1 > fstprint --acceptor b.fst 0 1 10 1 and I wish to use the 10 in b.fst as a sigma . I run `fstspecial --sigma_fst_sigma_label=10 b.fst bs.fst ` but then composition `fstcompose a.fst bs.fst ` fails. Am I missing a step? Do I need to set the matcher somehow? Thanks, Bill

BillByrne - 2017-08-29 - 17:32

but found https://github.com/shiranD/specializer

KyleGorman - 2017-09-08 - 11:27

Here's the full story. fstspecial is statically linked against phi, rho, sigma FST SOs, for uninteresting reasons. All other OpenFst binaries (including fstcompose) will try to dynamically load the relevant SOs before reading the FSTs in. To work around that, you just need to set your LD_LIBRARY_PATH so that dlopen looks in the directory where the SOs are stored. For me that looks like:

LD_LIBRARY_PATH=/usr/local/lib/fst

Or you could use Shiran's library to make this all work in Python. I've added a link to it on the Contrib page.

Log In

Pynini installation difficulties

LeahVelleman - 2017-07-08 - 20:49

I've successfully built and installed OpenFST. The cython commands listed here do not run successfully for me, giving the following errors:

> cython -3 --cplus -o pynini.cc pynini.pyx

Error compiling Cython file:


... if self._mode == b"r": return self._reader.far_type() elif self._mode == b"w": return self._writer.far_type() else: return "closed" ^

pynini.pyx:2426:13: Unicode literals do not support coercion to C types other than Py_UNICODE/Py_UCS4 (for characters) or Py_UNICODE* (for strings).

> cython -3 --cplus -o pywrapfst.cc pywrapfst.pyx

Error compiling Cython file:


... # Prints FST to stringstream, then returns resulting string. cdef fst.SymbolTable *ssymbols_ptr = NULL if ssymbols is not None: ssymbols_ptr = ssymbols._table cdef stringstream sstrm fst.PrintFst(deref(self._fst), sstrm, "<pywrapfst>", ^

pywrapfst.pyx:1717:42: Unicode literals do not support coercion to C types other than Py_UNICODE/Py_UCS4 (for characters) or Py_UNICODE* (for strings).

If I attempt to install Pynini anyway, `python setup.py install` runs without errors or warnings, but then attempting to use the code (either by importing it in python or by running `python setup.py test`) gives this error:

> python setup.py test running test running egg_info writing pynini.egg-info/PKG-INFO writing dependency_links to pynini.egg-info/dependency_links.txt writing top-level names to pynini.egg-info/top_level.txt reading manifest file 'pynini.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'pynini.egg-info/SOURCES.txt' running build_ext copying build/lib.macosx-10.12-x86_64-3.6/pywrapfst.cpython-36m-darwin.so -> copying build/lib.macosx-10.12-x86_64-3.6/pynini.cpython-36m-darwin.so -> pynini_test (unittest.loader._FailedTest) ... ERROR

================================================================== ERROR: pynini_test (unittest.loader._FailedTest)


ImportError: Failed to import test module: pynini_test Traceback (most recent call last): File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 153, in loadTestsFromName module = __import__(module_name) File "/Users/leahvelleman/Downloads/pynini-1.6/pynini_test.py", line 16, in <module> from pynini import * ImportError: dlopen(/Users/leahvelleman/Downloads/pynini-1.6/pynini.cpython-36m-darwin.so, 2): Symbol not found: __ZN3re23RE23Arg12parse_stringEPKcmPv Referenced from: /Users/leahvelleman/Downloads/pynini-1.6/pynini.cpython-36m-darwin.so Expected in: flat namespace in /Users/leahvelleman/Downloads/pynini-1.6/pynini.cpython-36m-darwin.so


Ran 1 test in 0.001s

FAILED (errors=1) Test failed: <unittest.runner.TextTestResult run=1 errors=1 failures=0> error: Test failed: <unittest.runner.TextTestResult run=1 errors=1 failures=0>

This is on MacOS 10.12.5, with XCode 8.3.3, and with python 3.6.1 and cython installed through macports.

LeahVelleman - 2017-07-08 - 21:18

Update: replacing the relevant "..." strings in the relevant .pyx files with b"..." strings allows cython to run without error messages, but after doing this and reinstalling pynini, `python setup.py test` gives the same error as before.

LeahVelleman - 2017-07-08 - 21:45

In fact, it looks like the cython errors and the conversion to python 3.6 is a red herring. When I do the whole process under python 2.7 (with the original, unmodified versions of those .pyx and .cc files) I get essentially the same error:

Traceback (most recent call last): File "setup.py", line 88, in <module> test_suite="pynini_test") File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/command/test.py", line 211, in run self.run_tests() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/command/test.py", line 234, in run_tests **exit_kwarg File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/main.py", line 94, in init self.parseArgs(argv) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/main.py", line 149, in parseArgs self.createTests() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/main.py", line 158, in createTests self.module) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames suites = [self.loadTestsFromName(name, module) for name in names] File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 91, in loadTestsFromName module = __import__('.'.join(parts_copy)) File "/Users/leahvelleman/Downloads/pynini-1.6/pynini_test.py", line 16, in <module> from pynini import * ImportError: dlopen(/Users/leahvelleman/Downloads/pynini-1.6/pynini.so, 2): Symbol not found: __ZN3re23RE23Arg12parse_stringEPKcmPv Referenced from: /Users/leahvelleman/Downloads/pynini-1.6/pynini.so Expected in: flat namespace in /Users/leahvelleman/Downloads/pynini-1.6/pynini.so

LeahVelleman - 2017-07-08 - 22:45

OK, got it sorted I think. The solution was to

1) Rebuild and reinstall re2 from source. 2) Switch a few "..." strings out for b"..." strings as described above. 3) Run cython to regenerate the files as described in README.rst. 4) Install and test Pynini.

I think step 2 would have been necessary for anyone trying to get this to work with python 3.6, but step 1 may have been due to some idiosyncrasy of my own setup. In any case, leaving this here in case someone else hits the same errors.

KyleGorman - 2017-07-24 - 11:58

Thanks Leah for the report there, I'll update the documentation for the next release. I wonder if I can simply eliminate all the "b"" literals and see if that works on both Python 2 and 3? (I love Python 3 but I develop mostly with 2.7 for ...employer reasons).

Bad fst header error in fstcompose

NatashaNayga - 2017-06-23 - 17:04

I have an fst file, but fstcompose doesn't like it. I made the fst file using fstinvert, but this returns a bad fst header error: ERROR: FstHeader::Read: Bad FST header: inverselexicon.fst<br> I currently have no experience with openfst, so any help would be appreciated.<br> Thanks!

KyleGorman - 2017-06-27 - 16:49

Where did the FST file come from?

Note that OpenFst data files are not guaranteed to be compatible across platforms, since some file formats are byte order-dependent. (And this is so we can memory-map them, which makes IO much faster.) If you want to share files across platforms, share the arc list produced by fstprint instead, and then use fstcompile to turn that into a platform-specific FST data file.

MattHawthorn - 2018-01-26 - 03:02

I'm getting the same error as @LeahVelleman on Mac OSX 10.12.6. I'm attempting to install pynini into an anaconda environment with python at version 3.6.4.

Of note is the fact that I'm using the Mac OSX-shipped c/c++ compilers to build re2: "Apple LLVM version 9.0.0 (clang-900.0.39.2)" and a homebrew-installed gcc/g++ to compile openfst: "gcc-7 (Homebrew GCC 7.3.0) 7.3.0"

I don't know enough about C++ compilers to know if that's simply an irreconcilable error, but I do know that both of my compilers were inadequate by themselves to compile and successfully test both libraries - the standard lib headers of the Mac OS version were out of date for openfst (specifically 'type_traits' was missing), and while the newer g++ compiled re2, some of the dynamic tests failed to pass. This is all using `make CC=gcc-7 CXX=g++-7 install/test/check`

`setup.py test` produces this output:

``` ================================================================== ERROR: pynini_test (unittest.loader._FailedTest)


ImportError: Failed to import test module: pynini_test Traceback (most recent call last): File "/Users/matthew_hawthorn/anaconda3/lib/python3.6/unittest/loader.py", line 153, in loadTestsFromName module = __import__(module_name) File "/Users/matthew_hawthorn/Desktop/src/pynini-1.8/pynini_test.py", line 15, in <module> from pynini import * ImportError: dlopen(/Users/matthew_hawthorn/Desktop/src/pynini-1.8/pynini.cpython-36m-darwin.so, 2): Symbol not found: __ZN3re23RE213GlobalReplaceEPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKS0_RKNS_11StringPieceE Referenced from: /Users/matthew_hawthorn/Desktop/src/pynini-1.8/pynini.cpython-36m-darwin.so Expected in: flat namespace in /Users/matthew_hawthorn/Desktop/src/pynini-1.8/pynini.cpython-36m-darwin.so ```

MattHawthorn - 2018-01-26 - 03:39

Udate: following the same procedure as LeahVelleman above (recompile re2, then openfst, then `python setup.py install` from the pynini root) but this time using gcc-7/g++-7 seems to have resulted in at least a partially-functional python installation. when I run `python setup.py test` a large number of tests pass, but a large number also fail. The cheeses example at https://www.oreilly.com/ideas/how-to-get-superior-text-processing-in-python-with-pynini appears to be running fine though. If you would like the full output of the test suite, let me know and I'll send it to you - don't want to clutter the forum here with it since it's quite long.

XiaofengWu - 2018-10-10 - 12:02

fstcompose failed with no complain... Hi

Could any help me with this? I used fstcompose to compose two toy fst, one is an acceptor, the other is a G.fst... if I just delete some line in the arpaLM which is used to create the G.fst, the composition will fail with no complain. Tried to debug it but its too hard for me. the code is ``` ARPALM=lm.arpa WORDS=words.txt GFST=G.toytwosent.2gm.2.fst

##get he arapLM cat <<EOF | ngram-count -lm $ARPALM -order 2 -text - 2>/dev/null show me pants in size thirty two show me pants in size thirty too EOF #delete the following line , it will fail #-0.30103 in size thirty two </s> #sed -i.bak -e '57d' $ARPALM ###get G.fst get_G(){ cat $1 | #-0.30103 in size thirty two </s> #sed -i.bak -e '27d' $ARPALM ngram -lm $ARPALM -write-lm $ARPALM.pruned -order 2 -prune-lowprobs ARPALM=$ARPALM.pruned

get_G(){ cat $1 | grep -v '<s> <s>' | grep -v '</s> <s>' | grep -v '</s> </s>' | arpa2fst - | fstprint | ${SCRIPT_DIR_FST}/eps2disambig.pl | ${SCRIPT_DIR_FST}/s2eps.pl | fstcompile --isymbols=${WORDS} --osymbols=${WORDS} --keep_isymbols=false --keep_osymbols=false | fstrmepsilon > $2 }

get_G $ARPALM $GFST 2>/dev/null

fstdraw --isymbols=${WORDS} --osymbols=${WORDS} $GFST $GFST.dot dot -Tpdf $GFST.dot -o $GFST.dot.pdf #a sentence #get the best score of this sentence in G.fst echo "************" echo "the best score in $GFST is " cat <<EOF | fstcompile --acceptor=true --isymbols=$WORDS |tee smallg.fst| fstcompose --compose_filter=auto --connect=false --v=5 - $GFST| tee gG.fst | fstshortestpath | fstpush --to_final=true --push_weights=true | fstprint | awk '{if(NF==2){print -$2;}}' 1 2 show 2 3 me 3 4 pants 4 5 in 5 6 size 6 7 thirty 7 8 too 8 EOF ```

Log In

Incorrect C++ code

PaulHewlett - 2017-06-12 - 11:51

I have just compiled the library on alpine-linux and the gcc compiler is giving a lot of warnings at line 174 of arc-map.h.

The warning points out that some code is incorrectly indented inside an if statement giving the impression that it is wrapped by the if when - because no curly braces are present - it is not. My reading of the rest of arc-map.h implies that this is in fact an error and curly braces should be added. The offending call to fst->SetFinal() is in the MAP_REQUIRE_SUPERFINAL clause.

If not an error fst->SetFinal() should be suitably outdented

PaulHewlett - 2017-06-12 - 11:53

<verbatim> case MAP_REQUIRE_SUPERFINAL: { if (s = superfinal) { A final_arc = (*mapper)(A(0, 0, fst->Final(s), kNoStateId)); if (final_arc.ilabel = 0 || final_arc.olabel = 0 || final_arc.weight = Weight::Zero()) fst->AddArc(s, A(final_arc.ilabel, final_arc.olabel, final_arc.weight, superfinal)); fst->SetFinal(s, Weight::Zero()); } break;

</verbatim>

KyleGorman - 2017-06-27 - 16:52

That appears to be fixed internally, and should be correct in the next release. Thanks for the report.
Log In

How can I get SFST Library?

VeljkoMiljanic - 2017-05-08 - 17:25

I would like to use NGramApprox operation from SFst library but I couldn't locate download link.

KyleGorman - 2017-05-30 - 14:36

I don't believe it's ready for release yet.

KyleGorman - 2018-07-17 - 22:34

It's now available at opengrm.org
Log In

Non-functional FST at HCLG stage

SurbhiSinghal - 2017-05-02 - 17:35

For a simple corpus that has 2 sentences A B CD AB C D

I used kaldi's recipe till CLG FST compilation and then my code because my Acoustic Model is Sphinx format. For a simple case like the one I mentioned above, I get the following error at the determinization step: <verbatim>FATAL: StringWeight::Plus: unequal arguments (non-functional FST?)</verbatim> However, if I use simple work around of Encode->Determinize->Decode using Thrax's Optimize function , it finishes compilation. I wonder what trick (that probably kaldi uses) in taking care of such kinds of trigram pronunciations.(that it maps the 2 same trigram pronunciations to different acoustic ID.) Thanks,

I am not sure

SurbhiSinghal - 2017-05-02 - 17:47

I apologize, my corpus was : A B CD AB C D

KyleGorman - 2017-05-30 - 14:37

You'd have to ask at the Kaldi forums, but probably the same trick? It's a relatively well-known one.
Log In

Optimal use of ReplaceFst

PafnoutiT - 2017-04-26 - 12:25

I have a reasonably sized graph (about 100MB) on which I want to run a replace with a small one (100kB) on an output label that appears 3 times.

The baseline runtime is 16 seconds. `fstreplace` takes 4 seconds, so if I pipe this into my binary it takes a total of 20 seconds. If I use ReplaceFst in the C++ the total runtime is 22 seconds.

Is there a way of making ReplaceFst more efficient? One of the issues of using `fstreplace` is that on short tasks with a baseline time one 1 second, it adds a flat time of 4 seconds, whereas there is almost no overhead with ReplaceFst.

KyleGorman - 2017-05-30 - 14:40

fstreplace uses the scripting interface, which means that functions and classes have to be looked up in static registries, several layers of pointer dereferencing, and virtual dispatch, so that might be the additional time spent? ReplaceFst is also "on-the-fly" (lazy) whereas there is no reasonable way to use that from within a binary. So if you build a ReplaceFst but don't visit it all in the next step, it may do less work.
Log In

Installation Problem

YangYu - 2017-04-19 - 09:32

I am using CentOs6.7; gcc version 4.8.2; When I installed openfst 1.6.0 or 1.6.2: when I put ./configure , everything runs smoothly when I put make, it runs for a while and some error occurs:

Here are the errors:

libtool: link: g++ -std=c++11 -o .libs/fstarcsort fstarcsort.o ../script/.libs/libfstscript.so ../lib/.libs/libfst.so -lm -ldl -Wl,-rpath -Wl,/usr/local/lib g++: ../script/.libs/libfstscript.so: No such file or directory g++: ../lib/.libs/libfst.so: No such file or directory

KyleGorman - 2017-05-30 - 14:42

None of us have old CentOS machines so it's difficult to replicate this. Could you just try with a more recent GCC? (That's quite old by Linux standards.)
Log In

About an error

BurakBOELEK - 2017-03-29 - 04:13

Hello;

I just took over a project and I am trying to run a speech recognition. I am not sure if this is the correct place to ask, But I will try my chance...

My code is here :

<verbatim> commands.add("/bin/sh"); commands.add("-c"); commands.add(execPath + "add-deltas ark:" + tempKlasor + "/stereo.ark ark:- | " + execPath + "nnet-forward --feature-transform=" + modelPath + "/word_model/final.feature_transform " + "--no-softmax=true --class-frame-counts=" + modelPath + "/word_model/ali_train_pdf.counts " + "--use-gpu=no " + modelPath + "/word_model/final.nnet ark:- ark:- | " + execPath + "/decode-faster-mapped --acoustic-scale=0.1 --allow-partial=true --max-active=1000 " + "--beam=7 " + modelPath + "/word_model/final.mdl " + modelPath + "/phone_model/HCLG.fst ark:- ark,t:" + tempKlasor + "/stereo.hyp ark:- | " + execPath + "ali-to-phones --per-frame=true " + modelPath + "/word_model/final.mdl ark:- ark,t:" + tempKlasor + "/stereo.phones"); <verbatim>

I expect this code to create stereo.phones file. But it did not create the file. I checked the file permissions. They all seem ok.

I am not experienced in speech recognition frown I hope you can help me...

Thanks

KyleGorman - 2017-05-30 - 14:41

This is the wrong place to ask about this---try at the forum of whatever speech recognition package (is that Kaldi?) you're using.
Log In

mutable versions of classes in pywrapfst

DavidVanLeeuwen - 2017-03-28 - 08:53

Hello,

I am working with pywrapfst to make my fsts. When I have used a more complicated function, such as `replace()`, the fst (or the symbol tables) are no mutable anymore. I still need to add arcs to the fst. How can I create a fully mutable copy from an unmutable fst? SHould I re-create one arc-by-arc?

Cheers,


david

KyleGorman - 2017-05-30 - 14:58

Hi David, I can't replicate this. The return value of replace is pywrapfst._MutableFst (underlyingly it's an VectorFst), which supports the .add_arc(state, arc) method.

As for symbol table access, if you want to mutate the symbol tables attached to an FST in-place, use .mutable_input_symbols() and .mutable_output_symbols() methods to get mutable references into the underlying tables.

Or, somewhat more slowly, make copies of the current symbol tables with .isymbols().copy() or .osymbols().copy(), mutate these, and then assign them back using .set_input_symbols(syms) and .set_output_symbols(syms).

The only way for an FST to not be mutable in pywrapfst is if it was read from disk and it wasn't mutable when it was on disk either.

Log In

Compilation of Openfst 1.6.2 fails with Intel Compiler

PeterSmit - 2017-03-23 - 16:07

OpenFST compilation with intel compiler fails with the following error:

libtool: compile:  icpc -DHAVE_CONFIG_H -I./../include -fno-exceptions -funsigned-char -ipo -O2 -xSSE4.2 -fp-model precise -fp-model source -ansi-alias -std=c++11 -MT mapped-file.lo -MD -MP -MF .deps/mapped-file.Tpo -c mapped-file.cc -o mapped-file.o &#62;/dev/null 2&#62;&1
./../include/fst/vector-fst.h(693): error: a function cannot be both constexpr and virtual
    constexpr uint32 Flags() const final { return kArcValueFlags; }
                     ^
          detected during:
            instantiation of class &#34;fst::MutableArcIterator&#60;fst::VectorFst&#60;Arc, State&#62;&#62; [with Arc=fst::LogArc, State=fst::VectorState&#60;fst::LogArc, std::allocator&#60;fst::LogArc&#62;&#62;]&#34; at line 707
            instantiation of &#34;void fst::VectorFst&#60;A, S&#62;::InitMutableArcIterator(fst::VectorFst&#60;A, S&#62;::StateId, fst::MutableArcIteratorData&#60;fst::VectorFst&#60;A, S&#62;::Arc&#62; *) [with A=fst::LogArc, S=fst::VectorState&#60;fst::LogArc, std::allocator&#60;fst::LogArc&#62;&#62;]&#34; at line 443
            implicit generation of &#34;fst::VectorFst&#60;A, S&#62;::~VectorFst() [with A=fst::LogArc, S=fst::VectorState&#60;fst::LogArc, std::allocator&#60;fst::LogArc&#62;&#62;]&#34; at line 443
            instantiation of class &#34;fst::VectorFst&#60;A, S&#62; [with A=fst::LogArc, S=fst::VectorState&#60;fst::LogArc, std::allocator&#60;fst::LogArc&#62;&#62;]&#34; at line 443
            instantiation of &#34;fst::VectorFst&#60;A, S&#62;::VectorFst() [with A=fst::LogArc, S=fst::VectorState&#60;fst::LogArc, std::allocator&#60;fst::LogArc&#62;&#62;]&#34; at line 77 of &#34;./../include/fst/register.h&#34;
            instantiation of &#34;fst::FstRegisterer&#60;FST&#62;::FstRegisterer() [with FST=fst::VectorFst&#60;fst::LogArc, fst::VectorState&#60;fst::LogArc, std::allocator&#60;fst::LogArc&#62;&#62;&#62;]&#34; at line 41 of &#34;fst.cc&#34;

Could this be fixed?

DanielRichardG - 2017-08-07 - 18:44

Hello PeterSmit,

Yes, I've encountered this too. The Intel C++ compiler appears to be a bit more strict on C++ compared to g++.

I researched this a bit at the time and found that this use of constexpr is incorrect. The following patch fixes the issue:

diff -ru openfst-1.6.2/src/include/fst/complement.h openfst-1.6.2/src/include/fst/complement.h
--- openfst-1.6.2/src/include/fst/complement.h   2017-03-13 15:54:39.000000000 -0400
+++ openfst-1.6.2/src/include/fst/complement.h   2017-06-08 19:15:38.128407003 -0400
@@ -246,7 +246,7 @@
     pos_ = a;
   }
 
-  constexpr uint32 Flags() const final { return kArcValueFlags; }
+  /*constexpr*/ uint32 Flags() const final { return kArcValueFlags; }
 
   void SetFlags(uint32, uint32) final {}
 
diff -ru openfst-1.6.2/src/include/fst/vector-fst.h openfst-1.6.2/src/include/fst/vector-fst.h
--- openfst-1.6.2/src/include/fst/vector-fst.h   2017-03-13 15:54:43.000000000 -0400
+++ openfst-1.6.2/src/include/fst/vector-fst.h   2017-06-08 18:10:12.585208111 -0400
@@ -690,7 +690,7 @@
                     kNoOEpsilons | kWeighted | kUnweighted;
   }
 
-  constexpr uint32 Flags() const final { return kArcValueFlags; }
+  /*constexpr*/ uint32 Flags() const final { return kArcValueFlags; }
 
   void SetFlags(uint32, uint32) final {}
 
Log In

KyleGorman - 2017-05-30 - 15:35

@PeterSmit we can fix this in the next version. It does appear to be against the spec.
Log In

Do not link against python libraries on mac

MichkaPopoff - 2017-03-21 - 18:42

Hi,

I am one of the maintainers of the homebrew package manager. We have a check for libraries that link against specific python libraries. This does not allow to run openfst with different Python versions on Mac (for example system Python 2 and brewed Python 2).

The fix is pretty easy, see the explanation the error message from our tests:

<verbatim> homebrew/science/openfst: * python modules have explicit framework links These python extension modules were linked directly to a Python framework binary. They should be linked with -undefined dynamic_lookup instead of -lpython or -framework Python. /usr/local/opt/openfst/lib/python2.7/site-packages/pywrapfst.so </verbatim>

See also the related discussion here: https://github.com/Homebrew/homebrew-science/pull/3010

I hope this can be fixed. If you need any help I can also have a look at the code. Thanks in advance.

KyleGorman - 2017-05-30 - 14:48

HI Michka, sorry we're just seeing this! I'd be glad to fix this but I'm not sure I understand the error. Or rather, I don't know what causes -lpython to be used, since this compilation is all done inside of several layers of autoconf magic (see openfst-1.6.2/src/extensions/python/Makefile.am). Could you perhaps suggest a patch to that file to address this problem? (please feel free to email me directly: my email address is in the README.)
Log In

Python extension error

WeijianLin - 2017-03-07 - 20:42

I successfully installed openfst It also passed 'make check'.

And this is what i got when i tried to install python extension: Collecting openfst Downloading openfst-1.6.1.tar.gz (223kB) 100% |████████████████████████████████| 225kB 1.7MB/s Installing collected packages: openfst Running setup.py install for openfst ... error Complete output from command /Users/weijian/anaconda2/bin/python -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-3hgZ8Q/openfst/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-MksT_q-record/install-record.txt --single-version-externally-managed --compile: running install running build running build_ext building 'pywrapfst' extension creating build creating build/temp.macosx-10.6-x86_64-2.7 gcc -fno-strict-aliasing -I/Users/anaconda2/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/anaconda2/include/python2.7 -c pywrapfst.cc -o build/temp.macosx-10.6-x86_64-2.7/pywrapfst.o -std=c++11 -Wno-unneeded-internal-declaration -Wno-unused-function In file included from pywrapfst.cc:461: /usr/local/include/fst/util.h:16:10: fatal error: 'type_traits' file not found #include <type_traits> ^ 1 error generated. error: command 'gcc' failed with exit status 1 I tried both --enable-python and --enable-far during configuration, still got the same error.

Any thoughts?

KyleGorman - 2017-05-30 - 15:36

I'd guess your compiler doesn't have full C++11 support, since it doesn't have the type_traits header. Upgrade and try again?
Log In

KyleGorman - 2017-03-17 - 14:43

There's no way to store a string inside of a "label", but if you want to keep track of which label integers go with which strings, use the fst::SymbolTable class. Say you've already populated the symbol table `syms`, then you can do something like:

f.AddArc(state, StdArc(syms.Find(istring), syms.Find(ostring), TropicalWeight::One, state));

Log In

Compilation on Ubuntu 16.10 (yakkety)

MoiriGamboni - 2017-02-28 - 03:59

Hello,

I am getting the following error when compiling version 1.6.1 on Ubuntu 16.10:

<verbatim> In file included from ./../include/fst/fstlib.h:72:0, from ./../include/fst/script/arciterator-class.h:10, from arciterator-class.cc:4: ./../include/fst/shortest-path.h: In function 'void fst::internal::SingleShortestPathBacktrace(const fst::Fst<Arc>&, fst::MutableFst<Arc>*, const std::vector<std::pair<typename Arc::StateId, long unsigned int> >&, typename Arc::StateId)': ./../include/fst/shortest-path.h:77:8: error: inconsistent deduction for 'auto': 'auto' and then 'int' for (auto state = f_parent, d = kNoStateId; state = kNoStateId; ^~~~ </verbatim>

Version 1.6.0 gives me a similar error at another location:

<verbatim> In file included from ./../include/fst/state-map.h:15:0, from ./../include/fst/arcsort.h:14, from ./../include/fst/accumulator.h:15, from ./../include/fst/label-reachable.h:14, from ./../include/fst/lookahead-matcher.h:17, from ./../include/fst/matcher-fst.h:14, from fst.cc:14: ./../include/fst/arc-map.h: In member function 'fst::GallicToNewSymbolsMapper<A, G>::ToArc fst::GallicToNewSymbolsMapper<A, G>::operator()(const FromArc&)': ./../include/fst/arc-map.h:968:14: error: inconsistent deduction for 'auto': 'int' and then 'auto' for (auto i = 0, p = state_; i < w1.Size(); ^~~~ </verbatim>

Version 1.5.4 seems to work correctly. Any ideas about how to fix this?

Thank you

MoiriGamboni - 2017-02-28 - 08:22

I changed auto to int and compilation finished without problems.

KyleGorman - 2017-03-17 - 14:45

This is a known compiler bug. We've worked around it in 1.6.2, BTW.

(BTW, not always safe to assume that `int` is the same as StateId. Use StateId instead.)

Log In

Nontrivial examples of the C++ interface

VictorMiller - 2017-02-02 - 16:19

Are there pieces of example code using the C++ interface to the library for non-trivial FSMs?

Also, how do I just create an acceptor (as opposed to a transducer) -- the StdArc takes output symbols and weights as arguments. Is there a variant which just takes input symbol and destination state? I noticed that there is no boolean semiring described. Do you just use the TropicalSemiring in that case?

VictorMiller - 2017-02-03 - 11:01

I've created a transducer and I want to minimize it. However, when I call Minimize I get the message: FATAL: Refusing to minimize a non-deterministic FST with allow_nondet = false

What is allow_nondet, and how to I set it to true?

KyleGorman - 2017-02-22 - 11:45

Hi Victor, yes we just use the tropical semiring, usually, though you could write your own.

You almost surely do not want to use allow_nondet (though you can set it with a flag invocation like --allow_nondet). Rather you want to (attempt to) determinize first. For some information on generic FST optimization, check out this snippet here:

http://openfst.cs.nyu.edu/doxygen/thrax/html/algo_2optimize_8h_source.html

Log In

compilation on iOS?

EstherJudd - 2017-01-11 - 04:53

Has anybody tried compiling openFST for iOS and using it in an app? I know this is Google code, so maybe not so likely ;). The compilation seems to go ok but when we use fst::Read to try and read in an FST we get an error:

ERROR: Fst::Read: Unknown Fst type const (arc type = standard): /Users/<username>/Library/Developer/CoreSimulator/Devices/56CC2534-29C7-417E-8C84-602DAE8714DC/data/Containers/Bundle/Application/09BD3339-3291-4D56-AFBB-28C8CFDCD1F4/demo3.app/model.fst

It happens regardless of whether we have a vector FST or change it to a const FST using fstconvert. Do the models also have to be created using the same platform, or can we create them on linux and use on MacOS / iOS?

KyleGorman - 2017-01-23 - 12:57

Hi Esther, we do strive to have OpenFst compile on any platform with POSIX, C99, and C++11 support and that ought to include iOS. (I also note, without actually knowing anything for a fact, that Google.app for iOS supports voice search, and that probably means that OpenFst is inside it.) We do occasionally get patches for OpenFst on non-Linux platforms and incorporate them, too.

That error would appear to suggest it was unable to `dlopen` the SO for const FST support (libfst-const.so on my system). That usually occurs in turn when the LD_LIBRARY_PATH does not include whatever directory this is found in. To debug, you could try simply finding that SO and then making sure dlopen can find it.

(Out of curiosity, what happens if you

`#include <fst/const-fst.h>`

then use `ConstFst::Read` ?)

KyleGorman - 2017-01-23 - 12:59

Note also that we use memory-mapped IO in OpenFst (for efficiency reasons), which means that serialized FSTs are not portable across platforms due to differing byte orderings; however your error suggests that this isn't an issue here.
Log In

Looking for "Runtime Code" to simulate the application of a transducer

KennethRBeesley - 2017-01-09 - 20:00

Could some kind soul point me to any algorithmic "runtime code" that takes an OpenFST FST and an input string (basically a sequence of integers), and simulates the application of the FST to the input string and returns any results?

That is, straightforward application can be done by building the input string into an FST, composing the input-FST on the input side of the main FST, and then extracting the output projection of the resulting FST. But I'm looking for C++ runtime code that takes an input string, walks through the main FST algorithmically, without modifying it, collecting matching output strings, and finally returning the set of output strings related to the original input string.

Any help would be much appreciated.

KyleGorman - 2017-01-23 - 12:19

That sounds pretty difficult (seems like you'd have to reinvent large portions of composition?), but I too would love to see a worked example.

KennethRBeesley - 2017-01-25 - 18:07

Difficult but not impossible. Here's a rough outline of what would need to be done. Assume that the arcs of the FST are marked with ints representing Unicode code point values (both on the input side and the output side of each arc label). And assume that the input is a single string, coded as a sequence of ints representing Unicode code point values. And assume that the FST is determinized, with Tropical Semiring weighting. The algorithm would start at the Start state of the FST; let that be the initial Current State; and at the beginning of the Input Sequence. Let the Current Weight be initialized to zero, and let the Current Output Sequence be an empty sequence of int. In the simplest case. where the FST has no epsilons (zeros) on the input side, you would take the first int of the input sequence (let that be the Current Input Symbol) and use a Matcher object to select the set of exit arcs leading from the Current State that match the Current Input Symbol on the input side. If there is just one such matching exit arc, "consume" the input symbol, collect the corresponding output int (output symbol) on the matching arc and append it to the Current Output Sequence, and add the weight of the matching arc to the Current Weight, and let the destination state of the arc be the new Current State. (If there's no match, then Fail.) Proceed like that until you simultaneously run out of input symbols and reach a Final State---that's a Success. Then collect or output the Current Output Sequence and its weight. Complications: at any Current State, if there are exit arcs with Epsilon on the input side, they need to be "explored" without consuming the Current Input Symbol. At any Current State, if there are multiple exit arcs that match the Current Input Symbol, or multiple epsilon arcs, or at least one epsilon arc and at least one matching arc, then you have a Branch State, and the algorithm would need to store the current State (Current State, position in the Input Sequence, Current Output Sequence, Current Weight) for future backups, and iterate through those exit arcs, exploring each one individually. After any path being explored succeeds or fails, the algorithm would need to backup to explore other yet-unexplored exit arcs at Branch States, restoring the State at the Branch as appropriate and exploring all remaining exit paths. Other complications: input-side epsilon loops; epsilon arcs leading from Final States where input succeeded. The algorithm could be implemented as Depth-First or Breadth-First. (Any corrections and suggestions would be appreciated.)

Such algorithmic "runtime code" that simulates the composition of an input string and the extraction of the output-projection language was implemented in the Xerox Finite State Toolkit (lexc and xfst) and in Foma, a very good clone. And it was deemed in both projects to be much more efficient than doing the straightforward composition and output-projection extraction. The Foma runtime code is available, and I'm trying to get a handle on it. I worked at Xerox for years but never down at such low levels. Both Xerox and Foma implemented unweighted FSTs, so the code for OpenFST FSTs would need to be modified slightly to collect the weight along each path.

I had hoped that such algorithmic runtime code might already be available for OpenFST FSTs. Hence my posting. Any pointers to such existing code would still be much appreciated.

[After finding or writing such runtime code for lookup of a single input string in a single FST, the next step would be to allow multiple FSTs and algorithmically simulate the composition of those FSTs while looking up the input sequence. This would be useful in cases where the multiple FSTs, if straightforwardly composed, would blow up in size.]

FrankieRobertson - 2017-09-21 - 11:04

This last thing mentioned is already implemented in OpenFST. This is what the delayed versions of each operation does, e.g. ComposeFst (as opposed to Compose). This has also been refered to as building a virtual transducer. I assume using this in combination with lazy output projection (ProjectFst) should be almost as fast as a method specialised to input strings.
Log In

Delayed composition via Python API

KilianGebhardt - 2017-01-06 - 15:57

I have a rather large FST F and want to compose it with an input string w. From the result I want to find the shortest path. For this I use in pynini: shortestpath(acceptor(w) * F) which takes long. I suspect that the composition is fully computed before the shortest path is computed. Is there some way to enforce delayed composition from Python?

KyleGorman - 2017-01-23 - 12:02

No, sorry, no easy way to expose delayed composition at the moment.

Just to play the devil's advocate:

* do you know for a fact that this would be faster with delayed computation of the cascade? (i.e., is it faster when you do it like that in C++?) I would not assume that it is necessarily so---just that it takes up less memory under the default garbage collection / cache settings; it depends. * Have you confirmed that it's the composition and not the shortest-path operation that is responsible for the slow computation? (i.e., by assuming the composition result and then doing shortestpath on that?) * Would calling `.optimize()` on `F` (or manually optimizing with epsilon-removal, determinization, minimization, etc.) help?

Log In

FST determinizability

MariyaCelinTA - 2016-12-21 - 00:33

I am working with WFST for modelling the pronunciation variations for dysarthric speakers. For my work I need to add multiple pronunciations for a single dysarthric speaker. I have tried them with HTK by adding multiple pronunciations in the dictionary, however the dictionary size was very large and it ended up in a poor performance. So, I thought of using weighted finite state transducers for modelling the pronunciation variations. I have added such pronunciation variations in my Lfst.txt. However during fstdeterminizestar, the L.fst and G.fst (bigram language model) are not composed because L.fst is not determinizable. So, I made the cyclic fst as an acyclic, then they got determinized, but during composition with H.fst (monophone acoustic model) I face the same issue. I get the following error doing fstdeterminize for my fst: FATAL: StringWeight::Plus: unequal arguments (non-functional FST?) w1 = 616 w2 = 719

I doubt whether, the multiple transitions introduced by me (to take care of the multiple pronunciation) at each state will be an issue? However, the multiple transitions have different outputs. I don't understand where did it go wrong. I always had a doubt that, Is HTK (by adding multiple pronunciations in dictionary) and WFST is the same. Or is WFST only a representation for speech recognition, is there any difference between HTK and WFST?

Can anyone please provide me assistance in adding the multiple pronunciation issues with Openfst.

KyleGorman - 2017-01-23 - 11:31

Hi Mariya,

Having multiple pronunciation variants in one's lexicon is a known problem in the literature on WFST speech recognition and OpenFst implements some solutions. FST disambiguation (http://www.openfst.org/twiki/bin/view/FST/DisambiguateDoc) is an algorithm specifically for making L \circ G determinizable and would seem to be applicable here.

The error you're getting from `fstdeterminize` indicates that the FST to be determinized is non-functional (in the sense that it is a one-to-many relation). `fstdeterminize` does support non-functional determinization, though this can be very slow; in general, determinization has poor guarantees: it is not guaranteed to terminate (the FST may have the "twins property": http://www.cs.nyu.edu/~mohri/pub/twins.pdf) and even for determinizable FSTs the complexity may be exponential in the size of the input.

If disambiguation doesn't help, here, you may want to see if you can use another trick: determinizing a weighted transducer as if it were an unweighted acceptor (http://www.cs.nyu.edu/~mohri/pub/dmk.pdf). This technique is used by `fst::Optimize`, a C++ function distributed with Pynini (pynini.opengrm.org) and Thrax (thrax.opengrm.org).

Both HTK and OpenFst use WFSTs, but I don't know much about the HTK internal representation thereof or how HTK handles multiple pronunciation variants.

Log In

Check if FST is empty?

EstherJudd - 2016-12-09 - 04:55

What is the best way to check in c++ whether the output of a composition between two FSTs results in a non-empty FST?

I tried with fst::Verify but it does not seem to give me the right answer.

EstherJudd - 2016-12-12 - 11:01

Another option might be to check if there is more than one state in the FST? I really need to find a fallback option when a composition between a pronunciation FST and a constraint grammar yields an empty FST.

EstherJudd - 2016-12-13 - 03:32

I found the solution. I can just check whether fst.NumStates() == 0.

KyleGorman - 2017-01-23 - 11:18

Hi Esther, that works for any subclass of ExpandedFst. Delayed FSTs (like ComposeFst) do not have a NumStates method, since we do not necessarily know the answer yet. A solution that works for all FST types (all subclasses of Fst) is to check if the Start() == kNoStateId. (which is -1).
Log In

changes in reader->GetFst from 1.5.2 to 1.5.3?

EstherJudd - 2016-09-29 - 06:10

I used to be able to get an FST from a FAR archive in c++ using

<verbatim>const StdFst &fst = reader->GetFst();</verbatim>

However with version 1.5.3 I get a compilation error and it works if I change to *fst but that seems to be incompatible with the older version that we still have running on our server. What is the best way to solve this apparent change (besides upgrading our server version)?

KyleGorman - 2016-10-11 - 18:52

This returns a pointer, not a reference, now, simply for cross-API consistency (other functions of the form "GetFst" all return a pointer as well). You're welcome to upgrade to any version >= 1.5.3 to get a version that returns a pointer. I can't think of a reasonable way to write code that can handle both, sorry.

EstherJudd - 2016-10-18 - 04:43

Ok, no problem. I will tell our software coders to update to the latest version of openfst.
Log In

Showing characters in the input label in PyFST

JoseOrtiz - 2016-09-28 - 19:41

I'm using the method linear_chain to accept a String. When I convert it into a fst binary to then into a DOT format, I get integers instead of the characters. Also, I have a SymbolTable for each of the corresponding letters being read.

What I need is to show the characters instead, be by the command line or by coding directly into Python. Sorry if this is not the place to ask. Any help or reference would be greatly appreciated.

EstherJudd - 2016-09-29 - 03:51

I have done it on the command line by typing:

fstdraw --isymbols=isyms.txt --osymbols=osyms.txt binary.fst binary.dot

I haven't been using this in python but maybe you can just do this?:

draw("binary.dot",input_symbols="isyms.txt",output_symbols="osyms.txt")

KyleGorman - 2016-10-11 - 18:55

The OpenFst developers don't contribute to pyfst, so you might be better off asking at their GitHub site: https://github.com/vchahun/pyfst

(Though note that project appears to have been abandoned: the last commit was 3 years ago, and it's unlikely to compile against any reasonable recent version of OpenFst).

JoseOrtiz - 2016-10-18 - 17:45

Thank you both. The suggestion by Esther worked perfectly. I started using PyFST instead of pywrapfst because I was unable to install it correctly in my laptop, however, I still wish to install it. Is the pywrapfst still being maintained?

Once again, thank you for taking the time to answer.

KyleGorman - 2017-01-23 - 17:24

pywrapfst is very much maintained; we use it every day. If you're having installation problems, does it compile?
Log In

+= and *= equivalence functions for weights

MaartenVanCasteren - 2016-08-08 - 11:17

Would it be possible to add functions to the weight templates that work equivalent to += and *= in C++, to avoid unnecessary copying? They could be .Times(const weight& w) and .Plus(const weight& w). The normal Times() and Plus() could then potentially be implemented using these.

KyleGorman - 2016-10-11 - 19:07

That certainly could be done. (Nit: you actually need separate LeftTimes and RightTimes methods, since some semirings are non-commutative as mentioned in http://www.openfst.org/twiki/bin/view/FST/FstWeightRequirements.) I'm not sure this is going to be worth much for the float weight types since everything ought to fit in CPU registers, and for composite weight types, I wonder if there's copy elision here. Do you know for a fact there's not? (I.e., what happens if you compile with -fno-elide-constructors on your system?)
Log In

Label with String

JoseOrtiz - 2016-07-15 - 12:23

Greetings,

I am currently trying to set the Labels to more than one character per Label. Would this be possible?

EstherJudd - 2016-08-23 - 08:33

I have a SymbolTable where the symbols sometimes consist of two characters, so this shouldn't be a problem. I would advise you to work with a SymbolTable that maps each label to a number and train the FST with the numbers instead of the labels.
Log In

pywrapfst SymbolTable allow_negative_labels flag?

LouisaConwill - 2016-07-11 - 15:52

Hello,

I am writing a Python application to traverse FSTs. I want to read in a binary FST (which does not have a symbol table associated with it), read in a symbol table, and set it as the binary FST's symbol table. The code that I've written looks like this:

fst = Fst.read(args.fst) isyms = SymbolTable.read_text(args.sym_table) fst.set_input_symbols(isyms)

This code works great when the symbol table does not have negative labels; however, it does not work when the symbol table has negative labels. The error I get is: "ERROR: SymbolTable::ReadText: Bad non-negative integer "-4"" (the symbol table I'm trying to read in has -4 as one if its labels).

Is there a way to set allow_negative_labels to true using either SymbolTable.read or SymbolTable.read_text? I know this is possible in C++ and when using the openfst command line tool, but I'm struggling to figure out how to do this in Python.

Thank you!

KyleGorman - 2016-10-11 - 19:16

There is a flag --allow_negative_labels for the binaries, but the ability to use them is not exposed to pywrapfst yet. I can do so, but I really wouldn't recommend using negative labels if you can possibly avoid it. I'll update the thread once/if I add this support.

KyleGorman - 2016-10-13 - 20:37

Okay, I've added the ability to specify that you want to permit negative labels for pywrapfst.SymbolTable.read_text. That'll be in the next release. But l I still strongly discourage using negative labels.
Log In

openfst 1.5.3 causes ld to segfault with gcc 4.7.2

DanielPovey - 2016-06-29 - 15:12

On our grid in JHU, when compiling OpenFst 1.5.3, we get a segfault in "ld". I have never heard of "ld" segfaulting before so something pretty funky must be going on. What compiler versions do you think it should work on?

jtrmal@a14 ~/soft/openfst-1.5.3/src/script $ /bin/sh ../../libtool --verbose --tag=CXX --mode=link g++ -std=c++11 -version-info 4:0:0 -o libfstscript.la -rpath /usr/local/lib arciterator-class.lo arcsort.lo closure.lo compile.lo compose.lo concat.lo connect.lo convert.lo decode.lo determinize.lo difference.lo disambiguate.lo draw.lo encode.lo encodemapper-class.lo epsnormalize.lo equal.lo equivalent.lo fst-class.lo info.lo intersect.lo invert.lo isomorphic.lo map.lo minimize.lo print.lo project.lo prune.lo push.lo randequivalent.lo randgen.lo relabel.lo replace.lo reverse.lo reweight.lo rmepsilon.lo script-impl.lo shortest-distance.lo shortest-path.lo stateiterator-class.lo synchronize.lo text-io.lo topsort.lo union.lo weight-class.lo verify.lo ../lib/libfst.la -lm -ldl libtool: link: rm -fr .libs/libfstscript.so.4.0.0 libtool: link: g++ -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.7/crtbeginS.o .libs/arciterator-class.o .libs/arcsort.o .libs/closure.o .libs/compile.o .libs/compose.o .libs/concat.o .libs/connect.o .libs/convert.o .libs/decode.o .libs/determinize.o .libs/difference.o .libs/disambiguate.o .libs/draw.o .libs/encode.o .libs/encodemapper-class.o .libs/epsnormalize.o .libs/equal.o .libs/equivalent.o .libs/fst-class.o .libs/info.o .libs/intersect.o .libs/invert.o .libs/isomorphic.o .libs/map.o .libs/minimize.o .libs/print.o .libs/project.o .libs/prune.o .libs/push.o .libs/randequivalent.o .libs/randgen.o .libs/relabel.o .libs/replace.o .libs/reverse.o .libs/reweight.o .libs/rmepsilon.o .libs/script-impl.o .libs/shortest-distance.o .libs/shortest-path.o .libs/stateiterator-class.o .libs/synchronize.o .libs/text-io.o .libs/topsort.o .libs/union.o .libs/weight-class.o .libs/verify.o -Wl,-rpath -Wl,/home/jtrmal/soft/openfst-1.5.3/src/lib/.libs ../lib/.libs/libfst.so -ldl -L/usr/lib/gcc/x86_64-linux-gnu/4.7 -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.7/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/crtn.o -Wl,-soname -Wl,libfstscript.so.4 -o .libs/libfstscript.so.4.0.0 collect2: error: ld terminated with signal 11 [Segmentation fault]

DanielPovey - 2016-06-29 - 19:40

BTW, this is on debian 7.10. Apparently OpenFst 1.5.3 says it should work from gcc version 4.7 which is what we are using.

DanielPovey - 2016-06-29 - 20:25

I have found more information from valgrind and submitted a bug-report in bugzilla, I think it's a linux binutils bug.

https://sourceware.org/bugzilla/show_bug.cgi?id=20317

DanielPovey - 2016-06-29 - 21:40

I have documented in the last comment on https://sourceware.org/bugzilla/show_bug.cgi?id=20317 that the bug can be fixed by taking a certain line from a more recent binutils version 2.26 (this fix works for me; I have to compile binutils from source with the fix). So it looks like it's not a bug in OpenFst but a bug in binutils that it just happens to trigger. Our binutils is version 2.22.

Dan

Log In

ShortestDistance very slow?

PafnoutiT - 2016-06-21 - 11:42

I have an fst (actually a Kaldi lattice, it's an acyclic graph in the tropical semiring) on which I wanted to use fst::ShortestDistance to the final state, however it's very slow. It takes about 1.5 sec to process my lattice, whereas if I loop myself over the states and arcs to compute the same quantity it happens in 0.03 seconds. What could I have done wrong?

KyleGorman - 2016-10-11 - 19:22

It's possible that automatic queue selection is failing because certain property bits are not yet set. You could try turning up logging to see what queue is selected (see http://www.openfst.org/twiki/bin/view/FST/FstAdvancedUsage#State_Queues), or if you're comfortable, could simply assert that the FST is (e.g.,) acyclic (and possibly topologically sorted) using SetProperties() before calling ShortestDistance. It is also possible that the explicit queue is less efficient than manual iteration for this simple case, in which case you should feel free to stick with your implementation.
Log In

Shortest distance very slow?

PafnoutiT - 2016-06-21 - 11:41

I have an fst (actually a Kaldi lattice, it's an acyclic graph in the tropical semiring) on which I wanted to use fst::ShortestDistance, however it's very slow. It takes about 1.5 sec to process my lattice, whereas if I loop myself over the states and arcs to compute the same quantity it happens in 0.03 seconds. What could I have done wrong?

Log In

Python extension compilation error

AlexBogatu - 2016-06-18 - 12:50

Hi, I am trying to use the python extension but I have some problems when I compile the source. I have done the following: - download openfst 1.5.3 - ./configure with --enable-far - make & make install - pip install openfst

This last command fails with the following error:

/usr/local/include/fst/script/fst-class.h:277:3: note: fst::script::FstClass::FstClass()

FstClass() : impl_(nullptr) {}

^

/usr/local/include/fst/script/fst-class.h:277:3: note: candidate expects 0 arguments, 1 provided

At global scope: cc1plus: warning: unrecognized command line option "-Wno-unneeded-internal-declaration" [enabled by default] error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Am I missing something? Thank you!

MarxerR - 2016-08-10 - 10:54

I get a similar error with clang:

building 'pywrapfst' extension

clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c pywrapfst.cc -o build/temp.macosx-10.11-x86_64-2.7/pywrapfst.o -std=c++11 -Wno-unneeded-internal-declaration -Wno-unused-function

pywrapfst.cc:41102:22: error: no matching constructor for initialization of 'fst::script::FstClass'

__pyx_v_tfst = new fst::script::FstClass(__pyx_v_self->_reader->GetFstClass());

^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

/usr/local/include/fst/script/fst-class.h:281:3: note: candidate constructor not viable: no known conversion from 'const fst::script::FstClass *' to 'const fst::script::FstClass' for 1st argument; dereference the argument with *

FstClass(const FstClass &other)

^

/usr/local/include/fst/script/fst-class.h:383:12: note: candidate constructor not viable: no known conversion from 'const fst::script::FstClass *' to 'fst::script::FstClassImplBase *' for 1st argument

explicit FstClass(FstClassImplBase *impl) : impl_(impl) {}

^

/usr/local/include/fst/script/fst-class.h:279:12: note: candidate template ignored: could not match 'Fst<type-parameter-0-0>' against 'const fst::script::FstClass *'

explicit FstClass(const Fst<Arc> &fst) : impl_(new FstClassImpl<Arc>(fst)) {}

^

/usr/local/include/fst/script/fst-class.h:276:3: note: candidate constructor not viable: requires 0 arguments, but 1 was provided

FstClass() : impl_(nullptr) {}

^

1 error generated.

error: command 'clang' failed with exit status 1

KyleGorman - 2016-10-11 - 19:24

There's no reason to call pip. The version up there is not compatible with arbitrary OpenFst versions. If you do --enable-python then make and make install, you have already installed pywrapfst for whatever version of OpenFst you're building.
Log In

Changing Stateid value.

JoseOrtiz - 2016-06-18 - 12:30

I'm trying to read files in GFA format and then parse into an openfst format. In GFA format, each new line starts with a character that identifies what type of line it is. What I've done so far is if it starts with H, it ignore the rest of the line and go the next one. If it starts with S, it means it's a state, so it proceeds to identify each part of the line which consist of five strings including the S. After this, I add a state, but I want my state to hold the value of one of the strings identified in the line, which is a numerical value. As I understand, when you use an AddState, it return a Stateid which begins at zero, then increments by one. If it helps, I'm currently using a StdVectorFst object for the AddState function. I am reading about the SetState function from the vector_fst_base_impl class, would I be able to use this funcion to first get the StateId I want to change and then assign the new value? If so, would this cause any errors?

Thanks.

DoganCan - 2016-06-19 - 16:40

Hi Jose

In OpenFst, state IDs are integers. They are used for uniquely identifying states. They are not tied to the states they identify. Most OpenFst operations do not preserve state IDs.

Cheers, Dogan

Log In

fst/fstlib.h header not found

PaulPoveda - 2016-05-31 - 10:31

Hello everyone,

I tried to install OpenFst to use it with SphinxTrain and it seem to work but I got stuck with the following error :

checking fst/fstlib.h usability... no checking fst/fstlib.h presence... no checking for fst/fstlib.h... no configure: error: fst/fstlib.h header not found

But the OpenFst binaries are well installed in /usr/local/lib/fst and the header too in /usr/local/include/fst..

Can anyone give me a clue about the issue ?

KyleGorman - 2016-10-11 - 19:26

Assuming that /usr/local/include/fst/fstlib.h exists and is readable for your user, you may need to ensure that that directory is in your include path. At the compiler level you will want to use the -I flag to pass the directory, but I'm not sure how to do this with SphinxTrain. There may be something wrong with your environment (this directory should very much be in your include path) or with the SphinxTrain configuration setup, but this doesn't seem to be an issue with OpenFst per se.
Log In

ArcIterator construction is very slow with delayed composition

RyanPrenger - 2016-05-03 - 16:04

Hi. I've written a decoder loosely based on Kaldi's simple-decoder that is reasonably fast on an FST that was composed offline (lex + language model). When I try using the decoder on the FST created with delayed composition it's around 100 times slower. All the time seems to be spent in the ArcIterator construction steps. The language model is around 500 megs, and the lex is around 15 megs. Would one expect such a large slow down? How do people usually get decoders to perform with a large language model (the composition results in an FST about double the size)?

KyleGorman - 2016-10-11 - 19:34

It could be quite slow to work with a delayed composition, particularly if the offline composition is slow. The former is used to implement the latter, and many tricks which speed up offline composition should make working with a delayed composition faster too. These include use of specific matchers (or modifying the inputs, e.g. with arc-sorting or setting the FST's properties) so that ComposeFst chooses the best matcher), state tables, and composition filters (see http://www.openfst.org/twiki/bin/view/FST/FstEfficiency), and selecting the best associativities for cascaded compositions. You might just try to make the offline composition fast and then port that solution to the delayed composition-based decoder.
Log In

Speed of weighted FST loading in c++

EstherJudd - 2016-04-12 - 04:28

I have created a large FST for g2p. Using this command I have been able to get some speed savings when loading. I apply this command in a shell script when building the FST. <br> <verbatim>fstconvert --fst_type=const --fst_align=true in.ofst out.ofst</verbatim> <br> I have also been able to reduce the size of my FST by creating a compressed version using: <br> <verbatim>fstcompress --gzip=true out.fst out_compr.fst</verbatim> <br> Now I read in my final FST in c++ but I use the StdVectorFst. I was reading that it is faster to use StdConstFst but when I use that, I get an error saying that that class is not derived from MutableFst. I use the following command to read in the compressed FST: <br> <verbatim> StdVectorFst *decompressed = new StdVectorFst(); Decompress(remap_fst_name,decompressed,true); </verbatim> <br> So my question is, how can I increase the load speed for this large FST?

MichaelRiley - 2016-05-27 - 10:11

fstcompress is for small file footprint; expands on load and is not fast. constfst is a bit faster to load than vectorfst not memory-mapped and much faster memory-mapped (but then is demand paged as used). you can't mutate a constfst.

EstherJudd - 2016-06-14 - 04:11

Thanks, I guess my question is if I convert a VectorFST to a constFST using the fstconvert command, what is the best way to load it faster in c++? I found this website that talks about this conversion and when I type fst-info it does say it is memory-mapped: http://www.lvcsr.com/openfst-133-mapped-files.html

And if I did want to make the resulting FST smaller without losing accuracy, what other things can I do besides fstcompress?

KyleGorman - 2016-10-11 - 19:38

One generic method for reducing the size (in terms of arcs and states) is to use the Optimize method in optimize.h (a header shipped with Thrax and Pynini). It merges identically-labeled multi-arcs, removes epsilons, and applies determinization and minimization where they are known to terminate. See the documentation there for the specific details. It does about as well as one can hope to do automatically without knowing how the FST will be used.
Log In

Possible bug in ReplaceFstMatcher?

GautamT - 2016-02-25 - 22:59

ReplaceFstMatcher in replace.h instantiates MultiEpsMatcher for each of the component fst with mode kMultiEpsList and non terminal labels as possible epsilon. It does this for both MATCH_INPUT and MATCH_OUTPUT case. Under the condition ofMATCH_INPUT, this seems wrong to me. Let's look at the 2 possible scenarios under MATCH_INPUT.

1. epsilon on replace is false: Under this condition input labels continue to remain and do not become epsilons. Input symbols could represent a domain different from output symbols and therefore input labels with values matching those of the non terminals shouldn't be considered epsilons. Replace operation takes place based on the output label and not input label. Current behaviour would match self loop (kNoLabel) against these. Dropping this behaviour would also result in performance gain since one would not have to do a Find on the underlying machine for each of the non terminal label to match against self loop.

2. epsilon on replace is true: Under this condition, input label accompanying a non terminal output label is always re-written to epsilon irrespective of whether or not its value is also that of a non terminal. Matching self loop against input in this case is tricky because one would need to match against corresponding output label. To begin input labels are sorted to use this matcher, to match against output labels that are not sorted does not seem to be a good idea.

My thoughts are that for MATCH_INPUT if epsilon on replace is false then we don't need to use MultiEpsMatcher for each underlying machine, or not perform AddMutlipleEpsLabel. If epsilon on replace is true then not use this matcher and the user wouldn't be able to take advantage of the option to not always cache. The other possible option would be to mandate that if epsilon on replace is true then input label match that of output label for non terminal arc and check this on detecting a non terminal label in the output arc.

To confirm to myself that this is a bug, I constructed a root fst and a sub fst and created a replace fst that is used for a composition with another fst. The replace fst is the rhs fst or fst2 so that it is matched on the input side. The fsts are created in such a way as to expose this problem. The result was that if one were to not do a lazy replace, i.e. fully expand it and then use it for composition, one would get a different result from using a lazy replace or replace fst. For reproducibility, here are the fsts:

root fst 0 1 1 5 1 sub fst 0 1 5 7 1

fst1 or lhs fst for composition 0 1 1 1 1 expanded fst (fstreplace root 10 sub 5 expanded) using 5 as the non terminal to refer to sub fst and root label being a non existent label (non recursive) 0 1 1 0 1 2 5 7 2 3 0 0 3

As can be seen the composition fst1 o expanded fst would result in an empty fst. However when not expanding and doing the composition it would result in a non empty fst with a final state since ReplaceFstMatcher would treat the input label 5 in the replace fst from the sub fst as epsilon and therefore composition proceeds.

Am I missing something?

GautamT - 2016-02-25 - 23:01

Somehow the fst formatting was off, fixing it.

root fst

0 1 1 5

1 sub fst 0 1 5 7 1

fst1 or lhs fst for composition

0 1 1 1

1

expanded fst (fstreplace root 10 sub 5 expanded) using 5 as the non terminal to refer to sub fst and root label being a non existent label (non recursive)

0 1 1 0

1 2 5 7

2 3 0 0

3

Log In

G.fst in ngram format work fail on arm

JerryLin - 2016-02-20 - 09:13

Hi all, I built G.fst in ngram format on x86 , and run on x86. it works fine I built G.fst in ngram format on arm , and run on arm, it throws bad_malloc exception. if you built G.fst in vector format on arm, and run on arm, it works fine. How to solve it ? Thank you.

MichaelRiley - 2016-05-27 - 10:13

Possibly changed endianity when crossing platforms. Fix - don't do that.

KyleGorman - 2016-10-11 - 19:40

Right, because IO is memory-mapped, don't expect serialized FSTs to work on a platform with a different endianity. It's a small price to pay for faster IO.
Log In

How to compute edit distance of a given text line

SopheaPRUM - 2016-02-18 - 20:50

Hello,

Given a text line containing mistakes (Example "AL WUYS LO W PRICES", the correct text line should be "ALWAYS LOW PRICES"). Considering that i have a lexicon and language model. How can i compute the edit distance of this given text line?

Please help Best regards

KyleGorman - 2016-10-11 - 19:43

The last example here (http://www.openfst.org/twiki/bin/view/FST/FstExamples) involves making an edit transducer and then computing the edit distance.

The paper on Pynini (https://aclweb.org/anthology/W/W16/W16-2409.pdf) has an example of using a noisy channel model and a language model to resolve this kind of ambiguity, as well.

Log In

TypeError: unsupported operand type(s) for -: 'int' and 'RestrictedProperty'

MariamAAMER - 2016-02-04 - 13:01

Hi, I am trying to create a script. The part that doesn't work is the pasted below: class MyGrating(Structure): wg_W = PositiveNumberProperty(required = True) wg_L = PositiveNumberProperty(required = True) taper_L = PositiveNumberProperty(required = True) wg_W_Grating = PositiveNumberProperty(required = True) period_Grating = PositiveNumberProperty(required = True) DC_Grating = PositiveNumberProperty(required = True) n_periods_Grating = PositiveNumberProperty(required = True) width_Grating = PositiveNumberProperty(required = True) Out_wg = PositiveNumberProperty(default=1)# 1 si queremos que dibuje guia, 0 si no la queremos def define_elements(self, elems): inside the define_elements, i have a for loop, and inside it, i have todo make an arithmetic operation between two of the above defined properties. I get an error message that says: TypeError: unsupported operand type(s) for -: 'int' and 'RestrictedProperty' Somebody could help me to resolve it. Thanks to everybody

Log In

Linking problem on 64bit arch

DodoIvanecky - 2016-01-08 - 09:54

I am trying to create a simple application with openFST. While fst.Write is working for my simple graph, functions like RmEpsilon or Determinize are not possible to link. In the first case I am getting: undefined reference to `fst::ReverseProperties(unsigned long, bool)' and it is O.K. since: uint64 ReverseProperties(uint64 inprops, bool has_superinitial); But why/from where is fst::ReverseProperties(unsigned long, bool) called, I have no idea. Any hint is appreciated.

DodoIvanecky - 2016-01-08 - 11:22

Looks like I see the same problem with several Property function. Calling Determinize I got undefined reference to `fst::DeterminizeProperties(unsigned long, bool, bool)'. Which is O.K. because: uint64 DeterminizeProperties(uint64 inprops, bool has_subsequential_label, bool distinct_psubsequential_labels)

DodoIvanecky - 2016-01-08 - 12:33

Solved. Problem is compiler. Needs to be at least 4.8. The reported problem was with 4.6 version.

Log In

Access control:

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng lex-unk.png r1 manage 242.8 K 2014-06-21 - 01:48 UnknownUser Example lexicon with unknown handling
Unknown file formatdiff openfst_auto_20071112.diff r1 manage 8.7 K 2007-11-13 - 05:13 UnknownUser Autoconfiscating patch
JPEGJPG wfsa1.JPG r1 manage 8.8 K 2014-01-16 - 10:22 UnknownUser Tropical WFSA redundant paths example 1
JPEGJPG wfsa2.JPG r1 manage 6.4 K 2014-01-16 - 10:23 UnknownUser Tropical WFSA without redundant paths example
Edit | Attach | Watch | Print version | History: r26 < r25 < r24 < r23 < r22 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r26 - 2018-11-04 - AliiiiRezaaa
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback