---+ [[GRM.NGramLibrary][OpenGrm NGram]] Forum You need to be a registered user to participate in the discussions. <br /> %IF{"context authenticated" then='%MAKETEXT{"You are logged in as [_1]." args="[[%WIKIUSERNAME%][%SPACEOUT{%WIKINAME%}%]]"}%%BR%%LOGOUT%' else='%LOGIN% %MAKETEXT{"or"}% [[%TWIKIWEB%.TWikiRegistration][%MAKETEXT{"Register"}%]]'}% %TOC{depth="2"}% You can start a new discussion here: <blockquote> %ICON{help}% You can use the formatting commands describes in TWiki.TextFormattingRules in your comment. <br /> %ICON{tip}% If you want to post some code, surround it with =<verbatim>= and =</verbatim>= tags. <br /> %ICON{warning}% Auto-linking of TWiki.WikiWords is now disabled in comments, so you can type !VectorFst and it won't result in a broken link.<br /> %ICON{warning}% You now need to use =<br>= to force new lines in your comment (unless inside =verbatim= tags). However, a blank line will automatically create a new paragraph. </blockquote> %COMMENT{type="newtopic"}% ---++ FATAL: NegLogDiff ---+++ Main.LukeCarmichael - 25 Sep 2012 - 17:21 <noautolink> Hello, I run this sequence of commands with the following output. <verbatim> home$ ngramcounts a.far > a.cnts home$ ngrammake --v=4 --method=katz a.cnts > katz.mod INFO: FstImpl::ReadHeader: source: a.cnts, fst_type: vector, arc_type: standard, version: 2, flags: 3 Count bin Katz Discounts (1-grams/2-grams/3-grams) Count = 1 -nan/0.253709/-0.343723 Count = 2 -nan/1.26571/1.19095 Count = 3 -nan/0.467797/-0.532465 Count = 4 -nan/1.29438/1.18879 Count = 5 -nan/0.0740557/-0.489831 Count > 5 1/1/1 FATAL: NegLogDiff: undefined -10.2649 -10.2651 </verbatim> Other methods work fine. How can I diagnose this problem? Thanks, Luke </noautolink> %COMMENT{type="mythreadmode"}% ---++ FATAL: NGramModel: bad ngram model topology ---+++ Main.BenoitFavre - 10 Sep 2012 - 09:18 <noautolink> I generated an ngram model from a .arpa file with the following command: ngramread --ARPA lm.arpa > lm.model ngramread does not complain, but ngraminfo and trying to load the model from C++ code generate the following error: FATAL: NGramModel: bad ngram model topology How can I troubleshoot the problem? </noautolink> ---+++ Main.BenoitFavre - 10 Sep 2012 - 09:27 <noautolink> Adding verbosity results in more mystery... ngraminfo --v=2 lm.model INFO: FstImpl::ReadHeader: source: lm.model, fst_type: vector, arc_type: standard, version: 2, flags: 3 INFO: Incomplete # of ascending n-grams: 1377525 FATAL: NGramModel: bad ngram model topology </noautolink> ---+++ Main.BrianRoark - 11 Sep 2012 - 10:33 <noautolink> Hi, that error is coming from a sanity check that verifies that every state in the language model (other than the start and unigram states) is reached by exactly one 'ascending' arc, that goes from a lower order to a higher order state. ARPA format models can diverge from this, by, for example, having 'holes' (e.g., bigrams pruned but trigrams with that bigram as a suffix retained). But ngramread should plug all of those. maybe duplication? I'll email you about this. </noautolink> ---+++ Main.BrianRoark - 18 Sep 2012 - 10:50 <noautolink> Benoit found a case where certain 'holes' from a pruned ARPA model were not being filled appropriately in the conversion. The sanity check routines on loading the model ensured that this anomaly was caught (causing the errors he mentioned), and we were able to find the cases where this was occurring and update the code. The updated conversion functions will be in the forthcoming version update release of the library, within the next month or two. In the meantime, if anyone encounters this problem, let me know and I can provide a workaround. </noautolink> %COMMENT{type="mythreadmode"}% -- Main.CyrilAllauzen - 09 Aug 2012
This topic: Forum
>
WebHome
>
GrmNGramForum
Topic revision: r5 - 2012-09-25 - LukeCarmichael
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback