Simplespider-techbrowser is a project mainly written in C#, it's free.

simple spider implemetation for techbrowser