Class TokenProducer


public class TokenProducer extends TokenProducer3<RuntimeException>
A simple parser that produces tokens from a String or Reader, and processes them through a user-provided handler.

This parser is intended to deal with handlers that produce only runtime (unchecked) exceptions. If your use case requires dealing with checked exceptions, please use TokenProducer3 instead.

Tokenization Overview

The tokens produced are:

  • Words. Contains letters, digits, format characters (Unicode Cf), connector punctuations, certain symbols and any codepoint in allowInWords or accepted by the supplied CharacterCheck object.
  • Quoted text (within single and double quotes).
  • Grouping characters: {}[]().
  • Separators.
  • Escaped characters: anything after a backslash, unless it is within a quoted text.
  • Control characters.
  • Other characters.
  • Comments. (if a comment-supporting method is used)

A moderate level of control of the parsing can be achieved with a TokenControl object.

The constructors that do not take a characterCountLimit argument process Reader streams of up to one Gigabyte (0x40000000) in size, throwing a SecurityException if they exceed that limit. Use the other constructors if you need to process larger files (or want smaller limits being enforced).

  • Constructor Details

    • TokenProducer

      public TokenProducer(TokenHandler2 handler)
      Instantiate a TokenProducer object with the given handler.
      Parameters:
      handler - the token handler.
    • TokenProducer

      public TokenProducer(TokenHandler2 handler, int characterCountLimit)
      Instantiate a TokenProducer object with the given handler and processing limit.
      Parameters:
      handler - the token handler.
      characterCountLimit - the character count limit.
    • TokenProducer

      public TokenProducer(TokenHandler2 handler, TokenProducer3.CharacterCheck charCheck)
      Instantiate a TokenProducer object with the given handler and CharacterCheck.
      Parameters:
      handler - the token handler.
      charCheck - the character checker object.
    • TokenProducer

      public TokenProducer(TokenHandler2 handler, TokenProducer3.CharacterCheck charCheck, int characterCountLimit)
      Instantiate a TokenProducer object with the given handler and CharacterCheck.
      Parameters:
      handler - the token handler.
      charCheck - the character checker object.
      characterCountLimit - the character count limit.
    • TokenProducer

      public TokenProducer(TokenHandler2 handler, int[] allowInWords)
      Construct a TokenProducer object with the given handler and an array of codepoints allowed in words.
      Parameters:
      handler - the token handler.
      allowInWords - the array of codepoints allowed in words.
    • TokenProducer

      public TokenProducer(TokenHandler2 handler, int[] allowInWords, int characterCountLimit)
      Construct a TokenProducer object with the given handler and an array of codepoints allowed in words.
      Parameters:
      handler - the token handler.
      allowInWords - the array of codepoints allowed in words.
      characterCountLimit - the character count limit.