Maxixe is a project mainly written in Ruby, it's free.
A small statistical segmenter for any language.
Maxixe is an implementation of the Tango algorithm describe in the paper "Mostly-unsupervised statistical segmentation of Japanese kanji sequences" by Ando and Lee. While the paper deals with Japanese characters, it should work on any unsegmented text given enough corpus data and a tuning of the algorithm paramenters.
Get a corpus of text you will can use to train the segmenter. Pre-segment a small number of sentences by hand. Create a large index for optimizing. Use optimized paramters for segmenting.
pre_segmented = [["MYDOGISINTHEHOUSE", "MY DOG IS IN THE HOUSE"],
["FOURNICEDOGS", "FOUR NICE DOGS"],
["MYCATLIKESMYDOG", "MY CAT LIKES MY DOG"]]
index = Maxixe::Trainer.generate_corpus_from_io([2,3,4,5], "ILIKEMYDOG
THISHOUSEISMYHOUSE
MYDOGISSONICE
WHOLIKESDOGSANYWAY
CATSANDDOGSUSUALLYFIGHT
INMYHOUSETHEREAREFOURDOGS
IWANTAHOUSEFORMYDOG")
optimal = Maxixe::Trainer.optimize(index, pre_segmented)
optimal is now {:n => ["2","4"], :t => 0.5, :score => 0}. Use this to make a segmenter.
Also, see http://rogerbraun.net/maxixe-a-simple-statistical-segmenter-for-any.
gem install maxixe should do the trick.