Arachne is a project mainly written in Python, it's free.

Arachne is a simple in python written document crawler. At the moment Arachne only supports text files (*.txt), but this will be extended. I plan to integrate a pluginsystem for different crawling methods (web, irc, pdf, ..). Take a look at the searc