Obey_robots_dot_txt is a project mainly written in Ruby, it's free.

Easy to use extension of Net::HTTP to let you obey robots.txt while crawling/scrapping/mining