Divider
  Speech Technology and Research Laboratory
  People
  Current Research Activities
  Past Research Activities
  Publications
  Career Opportunities
  Seminars
  Technologies for License
  In the News
  Contact Us
  STAR Search
  Information and Computing Sciences Division
SpacerAbout UsDividerR and D DivisionsDividerCareersDividerNewsroomDividerContact UsDividerSRI HomeSpacer

Spacer
         
  SRI Logo

Search SRILM-USER Archives

Match: Format: Sort by:
Search:

cross-entropy with OOV

From: "Sergey Protasov" <svp at ADDRESS HIDDEN>
Date: Fri, 2 Nov 2007 13:45:45 +0300

Dear experts,

I need to compute entropy with OOV words...

For example..
If  we have dict_size diffrent words in training corpora
then for test corpora (per word)

entr2  = entr1 +
stats.numOOVs*log2(dict_size_train_corpora)/num_words_test_corpora

entr1 = log2(ppl1)

But in C++ code TextStats.cc I don't know how to get  Dict_size_train_corpora
to compute this.

Dict_size_train_corpora = number_unigrams_train_corpora

Anybody help?

Thanx in advance!

Click here to go to the SRILM home page.

 

About Us  Vertical divider  R&D Divisions  Divider  Careers  Divider  Newsroom  Divider  Contact Us
©2006 SRI International, 333 Ravenswood Avenue, Menlo Park, CA 94025-3493
SRI International is an independent, nonprofit corporation. Privacy policy
Last modified Dec 05, 2008