SearchInMemory as a reseach about full text/fault-tolerant searches

SearchInMemory is my first impression about how works and how should work full text search engines or fault-tolerant searches (FTS). It begins as a research about how stuff works.

It was also a test for: is it hard to write own full text search engine?

You may consider it as a experiment to uncover how full text search engines works.

I have also tried a BinaryTree implementation of index, but it wasn't so good for me as HashIndex.

But even when I finished a really huge part of code there is still place for improvements(you can use it as a roadmap for your own FTS), like:

• n-gram indexing and searching with levenstein sorting
• wildcard searching
• excluding some phrases from results
• caching generated indexed on some memory based structure on disk
• steeming words to others
• improved and more complex way to do faceting
• a socket connector for searcher from a unix level
• index updating and deleting particular records
• whole phrases in " " signs, like: "billy bob" to match exactly this phrase (need to improve inverted indexes in HashIndex)
• possibilities of import/export data from indexes using formats: json/xml/csv
• tweaking results based on special criteria or queries

Cheers, Rafal "RaVbaker" Piekarski

Contact: web: http://about.me/ravbaker twitter: ravbaker github: https://github.com/RaVbaker

Great start for your own research:

• http://en.wikipedia.org/wiki/Levenshtein_distance - a minimal knowlegde about comparing similar words

• http://en.wikipedia.org/wiki/Inverted_index - goot start for building indexes - specially full inverted indexes

• http://en.wikipedia.org/wiki/N-gram - N-grams, what it is and why?

• http://googleresearch.blogspot.com/2006/08/all-our-n-gram-are-belong-to-you.html - quite old Google Research department post about n-grams in practise - with a large available dataset.

• http://ngrams.googlelabs.com/ - a practise usage of ngrams with Books Ngram Viewer from Google.

• http://today.java.net/pub/a/today/2005/08/09/didyoumean.html - great article about how Did you mean works with Lucene. Very inspiring post - but mainly about Java.

• http://framework.zend.com/code/filedetails.php?repname=Zend+Framework&path=%2Ftrunk%2Flibrary%2FZend%2FSearch%2FLucene.php - sourcecode from Zend Framework with their PHP implementation of Lucene. Nice source of thougts.

• http://www.ir.uwaterloo.ca/book/ - A book when you think BIG. It's about building your own service for full text search engine scalable almost like Google/Bing. Lots of theory and C/C++ code and algorithms. For very begining I suggest reading an excerpt from chapter 4 - Static inverted indicies - http://www.ir.uwaterloo.ca/book/04-static-inverted-indices.pdf

• Helpful php functions: http://www.php.net/manual/en/function.levenshtein.php, http://www.php.net/manual/en/function.metaphone.php, http://php.net/manual/en/function.soundex.php, http://docs.php.net/manual/en/language.types.array.php :)