TokenizerTransformer

class lightautoml.transformers.text.TokenizerTransformer(tokenizer=<lightautoml.text.tokenizer.SimpleEnTokenizer object>)

Bases: LAMLTransformer

Simple tokenizer transformer.

Parameters:

tokenizer (BaseTokenizer) – text tokenizer.

transform(dataset)

Transform text dataset to tokenized text dataset.

Parameters:

dataset (Union[NumpyDataset, PandasDataset]) – Pandas or Numpy dataset of text features.

Return type:

PandasDataset

Returns:

Pandas dataset with tokenized text.