The class pulls the HTML from a specified web page and separates the body text, title, and URLs. This project displays the results in a text box, but you could quite easily adapt it to store the information in a database for searching capabilities. You could also make it parse the links it finds for multi-level indexing. Designed to be an example only; fully commented. I am a beginner, so if you find problems or a better way to do something, let me know.

Download files: csharp_html_spider.zip