Search SRILM-USER Archives

Re: ngram manipulation

There is a hack to do it.
Remove from your LM any ngrams involving the <s> or </s> token (without
changing the other probabilities nad backoff weights).
Then feed your ngrams to "ngram -debug 1 -ppl").  The "sentence"
log probabilities will now correspond to joint ngram probabilities,
since the initial word will back off to a unigram probability, and
the final </s> will count as an OOV and not contrinute to the total
log probability.

It would be easy to add an option somewhere to make this more convenient,
without the need to hack the LM itself.

--Andreas

In message <45F01ED0.2030305 at ADDRESS HIDDEN>you wrote:
> Hello SRILM users,
>
> I have a question on the use of srilm toolkit for LM manipulation.
>
> The language model in the arpa format gives conditional probabilities
> e.g  p(wd3|wd1, wd2)
> Can I compute the joint probability p(wd1, wd2, wd3)  using any utility.
>
> I have a heavy LM with (ngram 1=50002, ngram 2=29077135, ngram 3=40083381).
>
>
> Any help would be greatly appreciated.
> Thanks,
> joel.
>
>
> arpa format:
> p(wd3|wd1,wd2) = if(trigram exists)           p_3(wd1,wd2,wd3)
>                 else if(bigram w1,w2 exists) bo_wt_2(w1,w2)*p(wd3|wd2)
>                 else                         p(wd3|w2)
>
> p(wd2|wd1)= if(bigram exists) p_2(wd1,wd2)
>             else              bo_wt_1(wd1)*p_1(wd2)
>

Click here to go to the SRILM home page.