Commenting on my last post, Jack objected that the problem is too hard, because everything is important, to various degrees. To take this in to account, a “hand-crafted approach” is needed. On the topic of everything being important: this is true. However, the vast majority of content on the Internet is simply not the kind of information that the semantic web is concerned with. You mighe change, slightly, the way you link to a product or item to make it more trackable. You might standardise the way you treat tags. You might more have greater standardisation in terms of flagging, for example, a review as a review, and a rating as a rating. However, the changes you would make are relatively small compared with the amount of content out there.  The time investment from individuals, minimal, the benefits large- once the tipping point has been reached.

Karan objected on the basis that the problem is too computationally difficult. However, as I suggest above, once the data is assembled, the problem is not at all difficult. The question then, is how to make that data available to Google. I borrow heavily from Berners-Lee in the following logic.

How to assemble data about transaction accounts in Australia? First, I suppose, we need to get a list of banks.  APRA has such a list. Now if APRA wanted to provide a bank search function, they could either try to lift the information off the website using textual cues or- far more easily- take advantage of behind-the-scenes tagging. This page describes such-and-such a product, which is of a particular kind (link to definition on APRA website), and each fee would be tagged as being a fee of a type (again, linking to an APRA definition.) The only difference from the current situation is the links. The content is all in place.

Once this happens, the APRA search can easily find the bank account I want. But how to get these results directly in Google? If all the tagging and linking is done in a standards compliant way- say, as described by W3C- Google can pick up on those hints and make use of the data too. Better yet, Google can easily use it’s page rank mechanism to identify that the APRA search function is a good match for my query. If the APRA search function is itself appropriately designed and tagged, Google can pass my query directly to it, and return the results without any further action on my part.

Identifying the search is easy, but Google automatically using it is somewhat complicated. This is by no means insurmountable, however. We’re talking about Google, after all. All of this is easily within the capabilities of technology we have right now. All that’s missing is the co-operation.

Is if it’s even in-theory possible to have a general-purpose search engine that you can just drop in to a knowledge-rich environment and, hey-presto, get the answers you’re looking for?

After thinking about this for a week, and reading Tim Berners-Lee’s 2001 article from the SciAm, I’m inclined to think that: no, it’s not possible. And it’s pretty obvious that it’s not possible. As Tim writes, “To date, the Web has developed most rapidly as a medium of documents for people rather than for data and information that can be processed automatically.” This goes double for the pile of information I need to digest at work. The trouble is that I have a bunch of unstructured data, and to work out what it all means and how it all relates, I need to do a lot of work. Of course it’s not possible for a search engine to do this currently!

They don’t even come close. Tim has a few examples of questions that, given a sufficiently structured data source, are very easily computable. Here’s my example: what bank in Australia has the lowest fee structure, with a preference for low overseas withdrawal fees. Plug that in to Google, and you’ll not get an answer. Oh, you might get linked to a half a dozen comparison sites, but they never seem to quite be interested in what you’re interested in. You might get linked to a Choice Magazine article that seems to have the answer: if you’re willing to pay $15 to access the article for 3 days. But you won’t get tabulated results saying: this are the bank offerings that most closely match your needs.

The semantic web could change that. Wouldn’t it be lovely. And yet it hasn’t happened. Are the technologies out there? Well, that’s my next question. Will it be as obvious as my last?

Say, if I visit a blog, I’m only interested in whatever has most recently been posted. I’ll read that, and if I like it, I’ll come back whenever I remember. If I forget to visit, no matter. I can read as much or as little as I want.

It shouldn’t have done, but it took me by surprise that using a knowledge database is rather different. All of a sudden I’ll need to know- now!- the correct interpretation of paragraph 3 in the subsection 4a of the guiding text. And how that impacts when the moon is in the fourth house. Woops. I think that might have been a post I forgot to read. Now, there are a bunch of ways that I could find the answer. A search engine, perhaps. But it had better be a smart search engine. I’m pressed for time. Now, I don’t know about you, but clicking “I’m feeling lucky,” normally doesn’t work for me on Google.

What I’m wondering is if it’s even in-theory possible to have a general-purpose search engine that you can just drop in to a knowledge-rich environment and, hey-presto, get the answers you’re looking for? Or is it rather the case that, even with an excellent search engine, you still need to do a lot of work teaching it about the knowledge it’s searching through, and the type of questions it’s going to get?

Now, I said that this is to help me know all this stuff: that’s not really true. It’s more a knowledge repository. The idea is to get all the knowledge out of people’s heads before they leave the company. New staff have to learn it the old fashioned way. I’m finding out the hard way that this is not so easy. Which has got me thinking: what could be done to make it easier? This blog is a place for my thoughts and discoveries on this.

I’ve got a bunch of ideas already, which need developing. I’ve clicked through to Wikipedia and ooh’d and aah’d at terms like “Information Theory” and “Expert Systems.” What I propose is a two-pronged approach. On the one hand, I’ll expand and work out some of my thinking. On the other, I’ll explore at some reactions to the literature. I start tomorrow!