For me, this is why semantic search is always going to be limited. Until websites adhere to standards of some sort (ideally XML or meta tags) then there will always be this problem.
Occasionally we need to get out of the mud, close our eyes and spin ourselves around in circles, head up to the clouds until we fall down on the freshly cut grass.
The Kentucky Derby website is of little help in explaining the tradition of wearing hats. But it is a tradition that dates back at least 100 years and shows no sign of dying off anytime soon. The hats tend to be on ladies, although as you can see in the photo to the right, some gentlemen engage in the tradition as well. The hats also tend to have a large brim, which in some cases is so large that it obscures the wearer's vision. LeAnn Rimes wore a hat to the 2009 Kentucky Derby that was so large she had to tilt her head a certain way, just to see. And the hats need not match your outfit. In fact, it seems that anything goes. Whether they are enormous and oddly decorated, or classic and tastefully arranged, any sort of hat is acceptable.
The problem then is (bear with me here) that every website is built differently. So let's go back to the derby hats example - the crawler needs to be able to identify products, pictures and prices and produce a nicely structured result. This is almost impossible with a generic web crawler. To approach each and every retailer to get a feed would be quite some task so you're left with creating a specific crawler for each site.
The same applies to websites as well. If your website has something important to offer, other websites will refer to your website. When it comes to the online world, the references are made by linking to the referred website. If many sites are linking to you then it means tladies big floppy hats your site has something important to offer the online community. If this is the case, Google too will consider your site to be an important site and will give a high Pagerank. Pagerank, which is also referred to as PR, is a numeric value assigned by Google to websites. The numeric value runs from 0 to 10.
