perl-String-Tokenizer - A simple string tokenizer

Distribution: ROSA Enterprise Desktop
Repository: ROSA Contrib Updates x86_64
Package name: perl-String-Tokenizer
Package version: 0.50.0
Package release: 1-rosa.lts2012.0
Package architecture: noarch
Package type: rpm
Installed size: 24.31 KB
Download size: 16.24 KB
Official Mirror:
A simple string tokenizer which takes a string and splits it on whitespace. It also optionally takes a string of characters to use as delimiters, and returns them with the token set as well. This allows for splitting the string in many different ways. This is a very basic tokenizer, so more complex needs should be either addressed with a custom written tokenizer or post-processing of the output generated by this module. Basically, this will not fill everyones needs, but it spans a gap between simple 'split / /, $string' and the other options that involve much larger and complex modules. Also note that this is not a lexical analyser. Many people confuse tokenization with lexical analysis. A tokenizer mearly splits its input into specific chunks, a lexical analyzer classifies those chunks. Sometimes these two steps are combined, but not here.



  • perl(String::Tokenizer) == 0.50.0
  • perl(String::Tokenizer::Iterator)
  • perl-String-Tokenizer == 0.50.0-1:2012.0

    Install Howto

    1. Enable ROSA Contrib Updates repository on Install and Remove Software"
    2. Update packages list:
      # urpmi.update -a
    3. Install perl-String-Tokenizer rpm package:
      # urpmi perl-String-Tokenizer


    • /usr/lib/perl5/vendor_perl/5.12.3/String/
    • /usr/share/doc/perl-String-Tokenizer/Changes
    • /usr/share/doc/perl-String-Tokenizer/README
    • /usr/share/man/man3/String::Tokenizer.3pm.xz


    2009-07-28 - J?r?me Quelin <> 0.50.0-1mdv2010.0 + Revision: 401608 - rebuild using %%perl_convert_version - fixed license field