Site-Structure-Analyzer is a project mainly written in RUBY and JAVASCRIPT, it's free.

Spider and analyzis for a given domain, Rails 3 with anemone