Comments on: Classification and clustering [part one] http://teofilachirei.wordpress.com/2009/05/17/classification-clustering-1/ Teofil Achirei's official blog Thu, 03 Sep 2009 12:30:43 +0000 http://wordpress.com/ hourly 1 By: teofilachirei http://teofilachirei.wordpress.com/2009/05/17/classification-clustering-1/#comment-43 teofilachirei Mon, 18 May 2009 19:23:19 +0000 http://teofilachirei.wordpress.com/?p=294#comment-43 Right now, using the Thesaurus and Link classes the crawler can only determine if a document is interesting (it's on topic) and if *some* links also look interesting and should be considering for crawling. The next step (and I'll post about it pretty soon) is to determine as accurate as possible the links that are worth to follow. That's the reason the thesaurus is so simple right now. Also, the number of links extracted from a certain page will probably be limited, because I don't want to crawl only one website. After this step is complete, I'll modify the Thesaurus class so it can be used for classifying documents (pages). Simply put: the thesaurus will have "categories" and each category will have a certain list of keywords. An hash map should be simply enough for this. To resume: - cluster or classify documents, not links - use the thesaurus just for the Best-First algorithm - the thesaurus will be soon improved and I'll come with some simple document classification Right now, using the Thesaurus and Link classes the crawler can only determine if a document is interesting (it’s on topic) and if *some* links also look interesting and should be considering for crawling.
The next step (and I’ll post about it pretty soon) is to determine as accurate as possible the links that are worth to follow. That’s the reason the thesaurus is so simple right now.
Also, the number of links extracted from a certain page will probably be limited, because I don’t want to crawl only one website.
After this step is complete, I’ll modify the Thesaurus class so it can be used for classifying documents (pages). Simply put: the thesaurus will have “categories” and each category will have a certain list of keywords. An hash map should be simply enough for this.

To resume:
- cluster or classify documents, not links
- use the thesaurus just for the Best-First algorithm
- the thesaurus will be soon improved and I’ll come with some simple document classification

]]> By: Ionut Popa http://teofilachirei.wordpress.com/2009/05/17/classification-clustering-1/#comment-42 Ionut Popa Mon, 18 May 2009 13:38:13 +0000 http://teofilachirei.wordpress.com/?p=294#comment-42 So are you going to use the classifier class to classify both the page and the links? Or if the document matches the thesaurus you'll extract all the links from it? So are you going to use the classifier class to classify both the page and the links? Or if the document matches the thesaurus you’ll extract all the links from it?

]]>