libreasr.lib.language.TokenizedLanguage

class libreasr.lib.language.TokenizedLanguage(*args, model_file='tmp/tokenizer.yttm-model', **kwargs)

Bases: libreasr.lib.language.Language

Methods

denumericalize
get_idx
get_token
numericalize
randomize

Attributes

EOS
SOS
iEOS
iSOS
replaceable