Site-spider is a project mainly written in Ruby, it's free.

Threaded site spider, useful for finding 404s and 500s. Probably won't destroy all your data.