Home > Articles

  • Print
  • + Share This
This chapter is from the book

A String Tokenizer Class

A number of languages provide a simple method for dividing strings into tokens separated by a specified character. While VB does not provide a class for this feature, we can write one quite easily using the little-known Split function. The goal of the Tokenizer class will be to pass in a string and obtain the successive string tokens back one at a time. For example, if we had the simple string

Now is the time

our tokenizer should return four tokens.

Now
is
the
time

The critical part of this class is that it holds the initial string and remembers which token is to be returned next.

We could write this class using the Instr function, or we could use the Split function, which approximates the Tokenizer but returns an array of substrings instead of having a class interface. The class we want to write will have a nextToken method that returns string tokens or a zero length string when we reach the end of the series of tokens.

The whole class is shown here.

'String tokenizer class
Private s As String, i As Integer
Private sep As String   'token separator
Private stokens() As String 'array of tokens
Public Sub init(ByVal st As String)
 s = st
 setSeparator " "
End Sub
Private Sub Class_Initialize()
 sep = " "   'default is a space separator
End Sub
Public Sub setSeparator(ByVal sp As String)
  sep = sp
  stokens = Split(s, sp)
  i = -1
End Sub
Public Function nextToken() As String
 Dim tok As String
 If i < UBound(stokens) Then
   i = i + 1
   tok = stokens(i)
 Else
   tok = ""
 End If
 nextToken = tok    'return token
End Function

The class is illustrated in use in Figure 3-5.

03fig05.jpgFigure 3-5. The tokenizer in use




This is the code that uses the Tokenizer class.

Private Sub Tokenize_Click()
 Dim tok As New Tokenizer
 Dim s As String

 tok.init txString.Text  'set the string from the input
 lsTokens.Clear          'clear the list box
 s = tok.nextToken       'get a token
 While Len(s) > 0        'as long as not of zero length
   lsTokens.AddItem s    'add into the list
   s = tok.nextToken     'and look for next token
 Wend
End Sub
  • + Share This
  • 🔖 Save To Your Account