Package org.gvsig.utils.console.jedit

Class TokenMarker

java.lang.Object
org.gvsig.utils.console.jedit.TokenMarker
Direct Known Subclasses:
BatchFileTokenMarker, CTokenMarker, EiffelTokenMarker, HTMLTokenMarker, MakefileTokenMarker, PatchTokenMarker, PerlTokenMarker, PHPTokenMarker, PropsTokenMarker, PythonTokenMarker, ShellScriptTokenMarker, SQLTokenMarker, TeXTokenMarker
public abstract class TokenMarker extends Object
A token marker that splits lines of text into tokens. Each token carries a length field and an indentification tag that can be mapped to a color for painting that token.

For performance reasons, the linked list of tokens is reused after each line is tokenized. Therefore, the return value of markTokens should only be used for immediate painting. Notably, it cannot be cached.

Version:
$Id$
Author:
Slava Pestov
See Also:
  • invalid reference 
    org.gjt.sp.jedit.syntax.Token

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    class 
    TokenMarker.LineInfo
    Inner class for storing information about tokenized lines.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected Token
    firstToken
    The first token in the list.
    protected int
    lastLine
    The last tokenized line.
    protected Token
    lastToken
    The last token in the list.
    protected int
    length
    The number of lines in the model being tokenized.
    protected TokenMarker.LineInfo[]
    lineInfo
    An array for storing information about lines.
    protected boolean
    nextLineRequested
    True if the next line should be painted.

  • Constructor Summary

    Constructors
    Modifier
    Constructor
    Description
    protected
    TokenMarker()
    Creates a new TokenMarker.

  • Method Summary

    Modifier and Type
    Method
    Description
    protected void
    addToken(int length, byte id)
    Adds a token to the token list.
    void
    deleteLines(int index, int lines)
    Informs the token marker that line have been deleted from the document.
    protected void
    ensureCapacity(int index)
    Ensures that the lineInfo array can contain the specified index.
    int
    getLineCount()
    Returns the number of lines in this token marker.
    void
    insertLines(int index, int lines)
    Informs the token marker that lines have been inserted into the document.
    boolean
    isNextLineRequested()
    Returns true if the next line should be repainted.
    Token
    markTokens(Segment line, int lineIndex)
    A wrapper for the lower-level markTokensImpl method that is called to split a line up into tokens.
    protected abstract byte
    markTokensImpl(byte token, Segment line, int lineIndex)
    An abstract method that splits a line up into tokens.
    boolean
    supportsMultilineTokens()
    Returns if the token marker supports tokens that span multiple lines.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • firstToken

      protected Token firstToken
      The first token in the list. This should be used as the return value from markTokens().

    • lastToken

      protected Token lastToken
      The last token in the list. New tokens are added here. This should be set to null before a new line is to be tokenized.

    • lineInfo

      protected TokenMarker.LineInfo[] lineInfo
      An array for storing information about lines. It is enlarged and shrunk automatically by the insertLines() and deleteLines() methods.

    • length

      protected int length
      The number of lines in the model being tokenized. This can be less than the length of the lineInfo array.

    • lastLine

      protected int lastLine
      The last tokenized line.

    • nextLineRequested

      protected boolean nextLineRequested
      True if the next line should be painted.

  • Constructor Details

    • TokenMarker

      protected TokenMarker()
      Creates a new TokenMarker. This DOES NOT create a lineInfo array; an initial call to insertLines() does that.

  • Method Details

    • markTokens

      public Token markTokens(Segment line, int lineIndex)
      A wrapper for the lower-level markTokensImpl method that is called to split a line up into tokens.
      Parameters:
      line - The line
      lineIndex - The line number

    • markTokensImpl

      protected abstract byte markTokensImpl(byte token, Segment line, int lineIndex)
      An abstract method that splits a line up into tokens. It should parse the line, and call addToken() to add syntax tokens to the token list. Then, it should return the initial token type for the next line.

      For example if the current line contains the start of a multiline comment that doesn't end on that line, this method should return the comment token type so that it continues on the next line.

      Parameters:
      token - The initial token type for this line
      line - The line to be tokenized
      lineIndex - The index of the line in the document, starting at 0
      Returns:
      The initial token type for the next line

    • supportsMultilineTokens

      public boolean supportsMultilineTokens()
      Returns if the token marker supports tokens that span multiple lines. If this is true, the object using this token marker is required to pass all lines in the document to the markTokens() method (in turn).

      The default implementation returns true; it should be overridden to return false on simpler token markers for increased speed.

    • insertLines

      public void insertLines(int index, int lines)
      Informs the token marker that lines have been inserted into the document. This inserts a gap in the lineInfo array.
      Parameters:
      index - The first line number
      lines - The number of lines

    • deleteLines

      public void deleteLines(int index, int lines)
      Informs the token marker that line have been deleted from the document. This removes the lines in question from the lineInfo array.
      Parameters:
      index - The first line number
      lines - The number of lines

    • getLineCount

      public int getLineCount()
      Returns the number of lines in this token marker.

    • isNextLineRequested

      public boolean isNextLineRequested()
      Returns true if the next line should be repainted. This will return true after a line has been tokenized that starts a multiline token that continues onto the next line.

    • ensureCapacity

      protected void ensureCapacity(int index)
      Ensures that the lineInfo array can contain the specified index. This enlarges it if necessary. No action is taken if the array is large enough already.

      It should be unnecessary to call this under normal circumstances; insertLine() should take care of enlarging the line info array automatically.

      Parameters:
      index - The array index

    • addToken

      protected void addToken(int length, byte id)
      Adds a token to the token list.
      Parameters:
      length - The length of the token
      id - The id of the token