SimpleRuTokenizer

class lightautoml.text.tokenizer.SimpleRuTokenizer(n_jobs=4, to_string=True, stopwords=False, is_stemmer=True, **kwargs)[source]

Russian tokenizer.

__init__(n_jobs=4, to_string=True, stopwords=False, is_stemmer=True, **kwargs)[source]

Tokenizer for Russian language.

Include numeric, punctuation and short word filtering. Use stemmer by default and do lowercase.

Parameters

n_jobs (int) – Number of threads for multiprocessing.
to_string (bool) – Return string or list of tokens.
stopwords (Union[bool, Sequence[str], None]) – Use stopwords or not.
is_stemmer (bool) – Use stemmer.

preprocess_sentence(snt)[source]

Preprocess sentence string (lowercase, etc.).

tokenize_sentence(snt)[source]

Convert sentence string to a list of tokens.

filter_tokens(snt)[source]

Clean list of sentence tokens.

postprocess_tokens(snt)[source]

Additional processing steps: lemmatization, pos tagging, etc.

postprocess_sentence(snt)[source]

Postprocess sentence string (merge words).