Webman is a project mainly written in Ruby, it's free.
Web crawling helper using EM and Hpricot
404: Not Found