Class | BufferedTokenizer |
In: |
lib/em/buftok.rb
|
Parent: | Object |
BufferedTokenizer takes a delimiter upon instantiation, or acts line-based by default. It allows input to be spoon-fed from some outside source which receives arbitrary length datagrams which may-or-may-not contain the token by which entities are delimited.
Commonly used to parse lines out of incoming data:
module LineBufferedConnection def receive_data(data) (@buffer ||= BufferedTokenizer.new).extract(data).each do |line| receive_line(line) end end end
New BufferedTokenizers will operate on lines delimited by "\n" by default or allow you to specify any delimiter token you so choose, which will then be used by String#split to tokenize the input data
Extract takes an arbitrary string of input data and returns an array of tokenized entities, provided there were any available to extract. This makes for easy processing of datagrams using a pattern like:
tokenizer.extract(data).map { |entity| Decode(entity) }.each do ...