Should the Net Forget? When Should We?

A very thought-provoking post in Carr's Roughtype: Should the Net Forget?. This article describes and comments on a recent NYTimes change in their search engine optimization (SEO) policies to increase the rate at which their articles come up on the first page of a search.

The result of this was that many older articles in their archives started showing up in searches. This also created a flood of complaints because the articles were incomplete, sometimes snapshots in time, out of the context that preceded or followed the article. This has caused some individuals embarrassment and difficulties. The NYTimes would have to add sometimes extensive footnotes and rewrites to set the record straight, which they do not want to do, claiming it would be 'rewriting history'.

I see an interesting corporate analog. In the late 1980s, as part of an artificial intelligence team, we interviewed executives seeking to gather the expertise they used to run the company. The object was to be able to leverage this knowledge in new ways and make it more available for people at all levels. One interviewee made the point that there are some things we do not want to remember, or at least not make them easy to remember and re-apply.

Even very clever knowledge can be misapplied in incorrect contexts. The mantra of AI at the time was that this was true, but all you had to do was to add the context of application to the system as well. That turned out to be much more difficult than we thought, and very high level knowledge applications of this type did not succeed. What worked were relatively narrow and focused applications of knowledge to problems.

Looking at the similarity to the NYT case. Its not just a matter of finding a solution via search, it requires adapting the solution to the current situation. Just because an item has many inbound links does not mean it's the best solution at this time, or under these situations. Its not to say we should forget, the historian will eventually be tasked to reassemble the bits of information. It's that we should not be so sure that an optimal search is best for both the provider and the user of that knowledge.