weka.core.tokenizers

## Class Tokenizer

- java.lang.Object
- - weka.core.tokenizers.Tokenizer

- All Implemented Interfaces:
    java.io.Serializable, java.util.Enumeration<java.lang.String>, OptionHandler, RevisionHandler
  Direct Known Subclasses:
    AlphabeticTokenizer, CharacterDelimitedTokenizer, CharacterNGramTokenizer

      public abstract class Tokenizer
      extends java.lang.Object
      implements java.util.Enumeration<java.lang.String>, OptionHandler, java.io.Serializable, RevisionHandler

  A superclass for all tokenizer algorithms.
  Version:
    $Revision: 10203 $
  Author:
    FracPete (fracpete at waikato dot ac dot nz)
  See Also:
    Serialized Form

- - ### Constructor Summary

    Constructors
    Constructor and Description
    `Tokenizer()`

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `java.lang.String[]` | `getOptions()` Gets the current option settings for the OptionHandler.
    `abstract java.lang.String` | `globalInfo()` Returns a string describing the stemmer
    `abstract boolean` | `hasMoreElements()` Tests if this enumeration contains more elements.
    `java.util.Enumeration<Option>` | `listOptions()` Returns an enumeration of all the available options..
    `abstract java.lang.String` | `nextElement()` Returns the next element of this enumeration if this enumeration object has at least one more element to provide.
    `static void` | `runTokenizer(Tokenizer tokenizer, java.lang.String[] options)` initializes the given tokenizer with the given options and runs the tokenizer over all the remaining strings in the options array.
    `void` | `setOptions(java.lang.String[] options)` Sets the OptionHandler's options using the given list.
    `abstract void` | `tokenize(java.lang.String s)` Sets the string to tokenize.
    `static java.lang.String[]` | `tokenize(Tokenizer tokenizer, java.lang.String[] options)` initializes the given tokenizer with the given options and runs the tokenizer over all the remaining strings in the options array.

    - ### Methods inherited from class java.lang.Object

      `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface weka.core.RevisionHandler

      `getRevision`

- - ### Constructor Detail

    - #### Tokenizer

          public Tokenizer()

  - ### Method Detail

    - #### globalInfo

          public abstract java.lang.String globalInfo()

      Returns a string describing the stemmer
      Returns:
        a description suitable for displaying in the explorer/experimenter gui

    - #### listOptions

          public java.util.Enumeration<Option> listOptions()

      Returns an enumeration of all the available options..
      Specified by:
        `listOptions` in interface `OptionHandler`
      Returns:
        an enumeration of all available options.

    - #### getOptions

          public java.lang.String[] getOptions()

      Gets the current option settings for the OptionHandler.
      Specified by:
        `getOptions` in interface `OptionHandler`
      Returns:
        the list of current option settings as an array of strings

    - #### setOptions

          public void setOptions(java.lang.String[] options)
                          throws java.lang.Exception

      Sets the OptionHandler's options using the given list. All options will be set (or reset) during this call (i.e. incremental setting of options is not possible).
      Specified by:
        `setOptions` in interface `OptionHandler`
      Parameters:
        `options` - the list of options as an array of strings
      Throws:
        `java.lang.Exception` - if an option is not supported

    - #### hasMoreElements

          public abstract boolean hasMoreElements()

      Tests if this enumeration contains more elements.
      Specified by:
        `hasMoreElements` in interface `java.util.Enumeration<java.lang.String>`
      Returns:
        true if and only if this enumeration object contains at least one more element to provide; false otherwise.

    - #### nextElement

          public abstract java.lang.String nextElement()

      Returns the next element of this enumeration if this enumeration object has at least one more element to provide.
      Specified by:
        `nextElement` in interface `java.util.Enumeration<java.lang.String>`
      Returns:
        the next element of this enumeration.

    - #### tokenize

          public abstract void tokenize(java.lang.String s)

      Sets the string to tokenize. Tokenization happens immediately.
      Parameters:
        `s` - the string to tokenize

    - #### tokenize

          public static java.lang.String[] tokenize(Tokenizer tokenizer,
                                    java.lang.String[] options)
                                             throws java.lang.Exception

      initializes the given tokenizer with the given options and runs the tokenizer over all the remaining strings in the options array. If no strings remained in the option string then data is read from stdin, line by line.
      Parameters:
        `tokenizer` - the tokenizer to use
        `options` - the options for the tokenizer
      Returns:
        the tokenized strings
      Throws:
        `java.lang.Exception` - if setting of options or tokenization fails

    - #### runTokenizer

          public static void runTokenizer(Tokenizer tokenizer,
                          java.lang.String[] options)

      initializes the given tokenizer with the given options and runs the tokenizer over all the remaining strings in the options array. The generated tokens are then printed to stdout. If no strings remained in the option string then data is read from stdin, line by line.
      Parameters:
        `tokenizer` - the tokenizer to use
        `options` - the options for the tokenizer