Hi-Tech Social Tech versus Delphi Method

As I wrote yesterday, I began my work in social technology with an interest in matching problems, using hi-tech methods to find each person a suitable job and compatible spouse. But social technology as a discipline has its roots in estimation and decision theory.

Just as it has been decades since I began pushing the concept of solving social problems with hi-tech social technology, so it has been decades since I began emphasizing the relationship between those older branches of the discipline and the matching problems which most concerned me.

For now let me just state without justification my belief that the social relationships between people who can work well together solving estimation problems will be extraordinary in other ways. I’ll work through that another day.

Though the term is now most often used for things related to social networking, it was first used as the title of the book “Social Technology” by Olaf Helmer, Bernice Brown and Theodore Gordon, individuals associated with the RAND Corporation

The key problem addressed in that book and others from the same intellectual circle was getting better answers, predictions and estimates from a team of people than from any one person.

An example of this social technology was the Delphi Method developed by researchers from the RAND Corporation. I first encountered it in Alvin Toffler’s book “Future Shock”, then most notably in the wonderful book “Shockwave Rider”, by John Brunner.

Rarely mentioned today, I think Brunner is my favourite science fiction author. Though published in 1975, this particular book is especially relevant today.

Brunner’s description of huge Delphi Pools is related to the RAND Corporations rather absurd policy analysis market idea. This was almost exactly wrong: what matters for estimation is not the size of the pool but its error-covariance. For matching, however, the size of the pool is an essential issue.

The developers of the Delphi Method did not understand that unless a team of people is carefully selected and matched to their tasks, they may accomplish less than a single person, no matter how large the team is. What matters is the teams internal error-covariance. People who make the same kinds of mistakes are likely to reinforce each other’s tendency to make those mistakes. This problem is not something which plagues decision makers, it is also at the root of much social conflict, from marital discord all the way up to international warfare.

Suppose that a very large number of yes/no questions on some topic. Including questions on related topics and some of general knowledge would be preferable. The following scenario assumes that the questions are chosen so that each person who takes the test gets approximately half the questions right, but the distribution of correct answers is quite uncorrelated. This is unlikely in a real life situation, though a careful selection of test questions may approximate it.

Instead of looking at the correlations between test questions, look at the correlations between the wrong answers given by individuals. Consider teams of three people and get them to answer similar questions by majority vote.

A team with a high error covariance amonst its members may get almost no anwers correct, if for each question two out of the three choose the wrong answer. To repeat: such an ill-chose team may get almost all questions wrong.

On the other hand, a team with a low error covariance may get almost all questions right, since in a majority vote of the three their errors may cancel out, at least two of the three usually being right. A well chosen team may get almost all questions right.

The basic method then:

1. Choose questions which are hard enough that approximately have the test-takes with get each question correct, while minimizing answer correlations between questions.

2. Divide individuals into teams of three according to their error covariance on the test.

3. Use these teams to answer real-life questions on the chosen topics.

The same basic methods can be used to match pairs of individuals to get numerical answers. A pair of people may make very good estimates, a trio, very good decisions.
With much more difficulty these pairs and trios can be combined into larger units, but simply creating big Delphi pools is absurd.

I leave for another day an explanation and some illustrations of the need for large pools of candidates for interpersonal matching purposes, an entirely different matter.

  dpw

This entry was posted in Uncategorized and tagged , . Bookmark the permalink.

Leave a Reply