tokenize_corpus module

Code to tokenize an in-house corpus/corpora using pre-trained tokenizer

tokenize_corpus.get_command_line_args()[source]
tokenize_corpus.main()[source]