Society’s Urgent Needs

Change Can and Must Happen Quickly

The world has terrible problems.   As these web pages demonstrate, I may have a solution — I’m sure I do.   But I may not be around to apply that solution for very much longer, so I feel a sense of urgency, a need to communicate what I know quickly.   But I’m not the only one feeling an urgent need to communicate.   This world has many people who are starving to death, right now, as you read these words.   It has many people dying of diseases we know how to cure.   There are also many who will perish from violent crimes and wars that could be prevented.   And for each person who actually dies from these causes there are many more who will suffer intolerable agony as a result of them.   These people also feel an urgent need to communicate, to call out for help.   Most of them won’t get any help. This is an awful situation, but I think I know how to resolve it .

The solution is not to send donations to some charitable organization.   For all the good such charities undoubtedly do, they are only bandaid-solutions which at best treat the symptoms and often just cover them up — they almost never treat the underlying causes.  Genuine solutions only, no bandaids allowed. For this is the old  SocialTechnology  logo:genuine solutions only

It might be better to express this positively, but there doesn’t seem to be a suitable word that means ‘non-bandaid’.  (‘Surgical’ is too threatening, ‘systemic’ too obscure).  Words aside, the idea is to find genuine solutions that treat the underlying cause.

As I write these words in the summer of 2001, I can look back on a decade of relatively few social changes in the rather contented North American society around me.   This calm fools some people into thinking major changes don’t happen or don’t happen quickly.   People in eastern Europe might have a very different view, looking back on thedecade of dramatic social change that followed the collapse of communism.   But people in African countries that saw more of the political coups, wars, and famines that have plagued that continent regularly for a very long time would certainly see the pace of change much differently — especially if their own family was involved.

Let not our present situation delude us into thinking change is rare or slow.   It can happen very fast.   Nor should we believe only bad changes happen fast.    A young child born in western Europe about 1939 would probably have experienced life in an ugly and dangerous war zone almost from birth, and by 1945 might well think that the norm.   The rapid recovery and dramatic changes of the next decade would surely have been a marvel.   Positive change can also happen quickly.

The very positive changes I am predicting and trying to bring about could happen quite quickly.   By way of illustration, I’d like to consider the reactions a young person of   2010  or   2020  might have to the television and movies of today.   I’m sorry if some of this sounds implausible to you, but to help you see the whole picture I’m inserting links to more sober explanations for each point.   So here are possible reactions or questions from the children of another generation:

  • “Why was there crime?   Why would anybody do that?   Nobody does now .”
  • “Did they actually let people go hungry?   And why couldn’t those people get food for themselves, like today ?”
  • “Why were so many people lonely?”  “Nobody thinks it’s hard to find true love now , do they?”
  • “War is so stone-age!   There haven’t been any wars for ages , right?  That’s what I thought.”
  • “School?  Classrooms full of kids all getting the same treatment?   No wonder education was so bad then !”
  • “What was a job?   Why couldn’t people just work together like now ?   Did they really have bosses?”
  • “You mean there were places where kids didn’t get any education at all?”   ” Why not?
  • “How could people be unhappy  —  didn’t they have lots of friends to cheer them up?   We all do .”
  •   …   and so on.

I hope you followed the links.    This is not just fantasy — I have genuine solutions ready to implement — and when they are realized the children of the future, your own children or grandchildren, may very well ask such questions.

Copyright 2001 Douglas P. Wilson

Posted in Old Pages | Leave a comment

The Relationship between Science and Technology

What is Science?   What is Technology?

And what is the relationship between them?

This page is about the meaning of ‘science’ and ‘technology’, with special emphasis on the often misunderstood relationship between science and technology, and it also serves as an introduction to the idea of social technology.

Living here on a small island, off of Canada’s west coast, I am surrounded by the beauty of nature and the peace that comes with an attractive environment. Perhaps because of the beauty of the local environment, we have on this island many people who think of themselves as environmentalists.

I’d like to introduce my views on science and technology by discussing first the one very misunderstood word which always comes up around here when anyone talks about environmentalism, and that is the word `technology.’ Many environmentalist take that as a dirty word, and say they are opposed to technology, which they blame for many of the current threats to our environment. I think that view is based on a mistake, one which comes from a fundamental misunderstanding of the relationship between science technology.

As long as I can remember, I have been interested in technology. But for most of that time I had a very limited and distorted view of what ‘technology’ means. Like many other people, I confused technology with hardware, and believed it was something invented by scientists. As a child I had a passion for collecting odd bits of machinery and electrical things, and dreamed of being a scientist. Scientists not only had enormous prestige, but they obviously had the best toys, and their careers seemed to consist of making more of them.

I recall that as a junior high-school student I went to an Open House at the University of British Columbia, where they invited the public to visit all the labs and facilities. Most interesting to me, of course, were the buildings of the Science Faculty, particularly Physics and Chemistry. We also visted the various engineering departments in the Faculty of Applied Science.

The very name “Applied Science” seemed to convey a lesser status, and my overwhelming impression is of large grey machines and something like a machine shop. Though I personally liked working with machinery and even doing metal work, I had hated the Grade 9 Industrial Arts course I had to take, and imagined that engineering was just the university equivalent of Industrial Arts.

All this was totally unfair to engineering and technology, words that I didn’t really understand until long after I went through university. As a child, what I liked best was to take apart some salvaged piece of an old car, or other bit of machinery, to see how it worked. I loved ingenious things, cleverly designed parts that worked together in interesting ways. What I hated about junior high-school Industrial Arts was the total absence of anything ingenious or clever. We seemed to spend all our time filing pieces of metal into uninteresting shapes.

But in fact the words “ingenious” and “engineering” both come from the same Latin root, `ingenium’, meaning cleverness or natural capability, and the essence of engineering has always been the application of human ingenuity, the clever contrivance of devices.

Closely related to engineering is technology. In high school we learned that the ending “-ology” meant “study of”, so that technology might be described as the study of techniques. I think it is more in keeping with the original Greek root to gloss “-ology” as “discussion of”, which is what the ancient Greeks did to study anything: they sat around and talked about it. What seemed most conspicuously absent in Industrial Arts 9 was any discussion of techniques; the teacher would tell us one way of doing something, like filing that piece of metal, and we would spend the rest of the period doing it.

I am fascinated by ingenious devices, and love to talk about techniques for making such devices, whether they be made of metal or exist entirely in software. But there was no engineering or technology listed in my high-school curriculum, only the boring Industrial Arts courses and what was called Science.

All through high-school I was an exceptional science student, i.e. nerd, but I never really understood what science was about. What I liked best about science was actually the technology involved in it.

Galileo’s discovery of the phases of the moons of Jupiter was actually of much less interest to me than the telescope he used in the process. If I dreamed of space travel, it was not because of any excitement over what we could learn about the moons of Jupiter from close up, but because of the rocket ships that would get us there.

Science class was almost universally hated by my fellow students, who found it boring. I can’t blame them for this, since the school managed to make the most interesting aspects of science tedious.

I remember one junior high-school science class in which the teacher explained how atoms had little hard cores around which electrons went in circular orbits. Now first of all, that just isn’t true, and real scientists had known it isn’t true for at least 30 years. But more important, it meant that one of the most interesting stories in the history of science, something that might have gotten a few students to pay attention, was being glossed over, replaced by a dry account of one particular, very obsolete, model.

At the time, I had just recently read about the early days of atomic physics, which I found fascinating because of the ingenious techniques and equipment used in the experiments. I remember annoying the teacher and the other students by trying to describe a more recent model of the atom, but that didn’t accomplish much. There was no time to go into the story I had read about the early years of quantum mechanics, and without that story, there wasn’t much I could say that would interest anyone.

The word `Science’ comes from the Latin word for knowledge, and means a specific kind of knowledge, the kind that answers questions about “what”, or “why”. Some people ask such questions a lot, and are interested in knowledge for its own sake, but most people are not. Rather more of us are interested in the other kind of knowledge, “know-how”, the knowledge of how to do something. This is a knowledge of techniques for doing things, and is the subject of technology.

I was fascinated by the technology or “know-how” of vacuum pumps and particle accellerators, and incidently learned some of the factual knowledge that was acquired by using these devices. The story of how these devices were invented and used was of great interest to me, and I wished I could have shared it with some of my fellow students, who might then have understood some of my enthusiasm for this apparently dry subject.

In other areas the distinction between scientific and technological knowledge is also important, and again the technological “know-how” is often neglected. I was rather a poor history student in high-school, partly because the techniques of archeology and document research by which historical knowledge is obtained were never discussed. We learned the name and dates of various kings and queens, but never had any introduction to the methods and techniques of the the historian.

This is particularly unfortunate, because technological knowledge is very much more durable than scientific knowledge. Even if a new technique is discovered with makes an old one obsolete, the old one is still usable. In school we learned few techniques, and a lot of supposed facts, many of which we now recognize as false. Reading old high-school textbooks is a rather shocking experience: much of what we were taught as kids just wasn’t right.

My current view is that technology is the most important of human endeavors, and includes the study of all techniques used in any aspects of human life. To me, law and politics is technology, as is medicine, agriculture, forestry, and any other area in which human ingenuity is applied. Many of these disciplines have associated sciences, but in many cases the technology is more highly developed than the science.

It seems obvious to me that engineering is not just “Applied Science”, but the application of technological knowledge, or “know-how”, which is logically independent of science, and may even be logically prior to science, as it is historically prior. People usually learned how to do something before they learned what was actually happening or why it worked. People learned to brew beer before the enzymatic actions of yeast were understood, they learned to smelt metals before the oxidation-reduction reactions were understood, and they learned to make concrete before the chemistry of tobermorphite gels was understood.

Some people use the word `Science’ to mean the body of scientific knowledge itself; I have no quarrel with that. But those who use it to mean the process and methods of discovery are referring to what I now think of as a branch of technology, a form of engineering. There are various techniques used to discover, validate, and gather scientific knowledge. The study of these techniques is one branch of technology. Their application is engineering. And their results are scientific knowledge.

All this discussion of science and engineering is a preamble to what I really want to talk about, which is social science and social technology, including social engineering — a term with many negative connotations. As a person who has spent a lot of time reading historical and political novels as well as non-fiction books on history and politics, the term `social engineering’ has a dark, unpleasant resonance, conjuring up images of totalitarian states and horrible technocratic “utopias” in which the people are oppressed by the engineers of the state. So I minimize my use of ‘social engineering’, and talk mostly about ‘social technology’ instead.

In this Atomic Age, when the threats of nuclear terrorism and environmental disaster are growing, it is not surprising to meet people who describe themselves as “Luddites” or “technophobes.” To such a person any talk of social engineering must have a particularly unpleasant ring.

But to me, social engineering is all around us, always has been, and always will be, since it means just the application of techniques for organizing or managing society. There is a very legitimate fear, however, since engineers are prone to a particular form of sin, which in the field of social engineering has very unpleasant consequences indeed.

As I have said, the essence of engineering is the ingenious design, and to many engineers the design and implementation of their contrivances is all of engineering. But designs have a way of capturing the minds of their creators, and so we have the particular sin of the engineer: the implementation of something undesirable and unwanted simply because it is an exciting design, a good solution to a “technically sweet” problem, something fun to work on.

An instance of the solution to Robert Oppenheimer’s “technically sweet” problem, as he called it, was dropped on Hiroshima — something that should give us all second thoughts about interesting problems.

The engineer’s sin is made possible by lax requirements analysis, and indeed, the analysis phase of engineering is often badly done. Engineering is filled with examples of good solutions to the wrong problem, brought about by bad analysis.

Most often bad engineering shows up in unwanted side effects. People want refrigeration, for example, but a side effect of the production of millions of refrigerators might be a hole in the ozone layer. Oops! Was this bad engineering? Well, yes. Good engineering includes consideration for side effects, even unknown and unanticipated one. The trouble is that the responsibility for the failure to anticipate the effects of releasing nasty gases into the atmosphere is almost as well diffused as the gases themselves.

But good social engineering, as I understand the term — good social technology, at least, would include the invention of tools and techniques for making society work, so that the concerns of the most farsighted and environmentally sensitive people get integrated into the process by which society creates and manages its physical technology.

For more information on social technology, please see my social technology page . For more information on requirements analysis and the engineer’s sin, please look at my page on the role of requirements analysis  in social technology.

Copyright © 1992 and1998 Douglas P. Wilson

Posted in Old Pages | Leave a comment

Traveling Salesman Approximations

Approximation algorithms for huge instances of the

Traveling Salesman Problem

This page is now obsolete, but may be of some interest. What it says is not wrong, it is just out of context, missing the key information about why this problem is part of the greater project being undertaken (see links below).

(No-Backtracking, Minimal Look-Ahead, Fast Enough for Huge Problems, Not Too Bad an Approximation)

Motivation:  far from being an academic exercise, the travelling salesman problem has several real-world applications and shows up in research work in many areas.    The algorithms discussed below grew out of work on the Acronymic Language project.   For that project a need arose to (approximately) solve a TSP for a graph with millions of vertices.

The key notion for the first algorithm is maintaining a priority queue of edges, where the highest priority is given to edges whose unavailability would cause the most grief later on.

Repeatedly take the edge with the highest priority and add it to the tour.   Each time this done, the two vertices the edge joins will have one fewer possible edge, from a maximum of two (one incoming and one outgoing edge per vertex).   When the number of edges joining a vertex to the tour reaches two, no more can be added, and therefore all edges incident on that vertex become unavailable (and are therefore removed from the queue).   An unpleasant side-effect of declaring an edge unavailable is that it may force the vertex at the other end of it to have longer connections to the tour.  So we priorize edges to minimize that possibility.    A better algorithm might be quite clever at guessing the future side effects of using up edges, looking many steps ahead like a good computer chess program, but the one presented here just looks one step ahead.   Good enough for now.

The edges incident on a given vertex can be sorted in terms of length.   There may be many of them — one graph studied recently had 638 edges meeting at one busy vertex.   If these are sorted, then the two shortest edges represent the best possible way of connecting this vertex to the tour — ignoring the entire rest of the graph.   If the shortest edge becomes unavailable because the vertex at the other end of it acquires both of the edges it is allowed in any tour, then the best way of connecting the current vertex will be through the second and third shortest edges.   That will not improve anything — at best the sum of the lengths of the second and third shortest edges will be no more than the sum of the first and second was, but in a great many cases, most cases, the sum of second and third will be longer and so the best possible tour (from the point of view of this vertex) will have gotten longer.   That increase in length is the penalty paid for losing the use of the shortest edge.   From the point of view of the vertex at the other end of this edge there is also (probably) an increase in overall tour length if the edge becomes unavailable.   The sum of these two numbers represents a first crude estimate of the damage done by prematurely losing the use of that edge, so it is also a good priority figure to apply to the edge, associating therefore a higher priority with a larger penalty (larger estimated increase in tour length if edge not used).

If all vertices have lots of edges of nearly the same lengths, then there are many almost equivalent solutions.   This algorithm deals with the other kind of vertex, one with a few edges of dramatically different lengths.   In that case it is vital to avoid the long edges.   So the shorter edges of those vertices get a higher priority.   Priorities are adjusted as we go along.

Note:  almost the same algorithm is also used for huge instances of a bipartite matching problem .  In both of these the problem is to select a special subgraph of the orginal graph.   Like the TSP, the Minimal Spanning Tree problem involves creating a connected subgraph with all the original vertices.   The Bipartite Matching problem involves creating a graph with half as many connected (2-vertex) components as the original graph has vertices.   But all three involve a minimization over the length of edges in the result.   The MST problem is easy because there are no side-effects and so the algorithm can be greedy.

This algorithm does no backtracking — once an edge is added to the tour it is there to stay, we never exchange it for some other edge.   There is some moderate look ahead :  when an edge is added to the tour (because it had the highest priority) we must re-examine the two vertices it connected, and if either or both of them still have one edge to be added, the other edges incident on them must be examined and possibly re-prioritized.   But we don’t go any further than that.   And so the result might not be very good.   It will be (roughly) as good as it can be without backtracking and deeper lookahead, but that’s pretty good in a lot of cases.

A data structure:

List the vertices in (some) order.

For each vertex list the incident edges, sorted in ascending order by length.

Pack this into an array, vertex after vertex and within each vertex the edges.  (array has as many entries as edges * 2)
Index into this array the start and end of each vertex’s edgelist in another array (array has as many entries as vertices)

(use tool to create problem instance then turn into table formatted like this:)

Number Edge
Number 1 1 1 2 2 3 3 3 3 3

(this page is an example of Work In HTML Directly)

Some kind of Code-And-Go where I could write Java (or some other language) right here and test it as I go would be nice.   —  Is that possible with what I have?   I suppose I could set up everything so this page points to class files with a prespecified name, click into ms-dos prompt to write code and compile it, then click back here to run the class files.    Try it!     For this problem what would be nice would be having simultaneously the visible graph and the tabular-format arrays, both changing in real time as the program runs  —  algorithm animation on graph and array at once.

Settle for less!    (but plan for more later  — graph and array could be kept logically separate, windowless mirrors, etc.)    Note: you are using html for memo and idea stuff.    Could this be made better and integrated with memoware?

(My priorities, though, must be text first, then diagrams (static ones), then java stuff that moves).

(in part this and other graph theory pages are being included to teach the non-mathematical the stuff they need to learn to understand the rest of the pages, hence all the english)

Copyright (c) 1998  Douglas P. Wilson

Posted in Old Pages | Leave a comment

Can It Work?

Some problems are easy, some far to difficult.  Can we actually create software which will do what is necessary?

The most important distinction goes under the name “Inverse Problems”, which will be the title of a web page soon. This is shorthand for a well understood mathematical phenomenon that has been at the root of modern mathematics since Newton, at least. It has been very difficult to explain this to people until very recently when technology based on inverse problems has become commonplace.

When it’s ready I’ll refer you immediately to that Inverse Problems page for a better explanation but for now you should simply consider the nature of the public key cryptography now used all over the net. Public key cryptography is used in Pretty Good Privacy, PGP, and similar cryptographic systems based on the well-known mathematical phenomenon wherein two problems which are inverses of one another are known, but one of them is easy and one hard.

Usually the problems involved are multiplying two large numbers to get their even larger product and the inverse problem of factoring a very large number to demonstrate that it is the product of two smaller ones. When the numbers are integers, (sometimes called whole numbers), and the multiplication must be exact, the second problem, factoring, can be very very hard, while the first one remains easy.

Mathematicians and computer scientists often use the shorthand phrase “inverse problems” to describe this situation. It is a misleading term, but a common one. Strictly speaking multiplication and division are inverses, as are addition and subtraction, but the term “inverse problem” is usually reserved for the case when the extreme difficult of one of these inverses causes a big problem.

Again, strictly speaking, encryption using a public key and decryption using a private key are inverses, like the multiplication and division they are based on, but the shorthand term “inverse problem” is used here to refer to the much harder problem of cracking the code so as to decrypt the encrypted text without knowing the private key beforehand. Another way to describe this is to say that PGP uses a “trapdoor algorithm” — if you push on the trapdoor from one side it opens easily, but to make it open from pushing on the other side is hard, you have to actually break it down.

I’ll hope all the publicity given to cryptography in recent years means you understand the basic concept, so I can just move on to other examples and to the bigger picture they suggest.

Another inverse problem comes up in data compression, especially in “lossy” data compression. Compressing files with zip or gzip is carefully done to lose no information at all so that the expanded (or inflated, or exploded) text recovered by decompressing the compressed file later reproduces the original uncompressed file exactly, with nothing lost in the translation.

If you have zipped up large directories full of big files as I do so often you may have noticed that it takes longer to compress the files than it does later on to unzip them. Compression and decompression are obviously inverse operations, that goes without saying, but they are not equally easy to do, and so we start to use the term “inverse problems” to say in shorthand that the inverse of the easy operation is a hard problem.

But in lossy compression such as is used in making jpeg and mp3 files, some data is lost — the file that results from decompression is not exactly the same as the file that was compressed. The trick is to make sure the data that was lost wasn’t important anyway, mostly just noise, while preserving the important data that we can think of as the signal. Another, even more powerful form of lossy data compression is fractal compression where very high compression ratios are available through mathematical magic that almost nobody understands.

In all these lossy compression systems it is much easier to decompress the compressed file (see the jpeg, listen to the mp3 file) than it is to create those files by compressing the originals. In fractal compress the situation is even more extreme and doing the original compression operation is very difficult and time consuming, much much harder than decompressing later.

The mathematics behind CASA and its companion projects is almost entirely based on inverse problems, actually on methods very similar to those used in jpeg and mp3 compression. The results are then used in combinatorial optimization operations that are also mathematically challenging, but none of this would be practical without the lossy compression step.

I’ll leave the discussion of technical details to the “How It Works” and “Does It Work?” pages. This page here is for a different purpose, the explanation of a very important concept.

On these pages I am trying to find a middle ground where there is none, in the middle of the vast crevass between C.P. Snow’s “Two Cultures”. I’ve spent a lot of time over the years in various kinds of mathematical activity that would interest the scientists and engineers on one side of the chasm but are mostly incomprehensible to the arts and humanities on the other side. But I’ve also spend a lot of time writing text in plain English (more or less plain English) which scientists and engineers often find to vague or imprecise — not technical enough. I am (somewhat) comfortable on both side of the gap, so I do both things.

As a result I (think I can) produce pages and programs so that others from either camp will find something of value in what I’ve done. Computer programs, especially the rather mathematical ones I usually write, are quite technial and formal objects — if written in a real language like ML they can even be proven correct. Pages of text like this one are very much informal objects that have no precise definition and aren’t meant to. They are meant to be roughly understood, not proven correct or used to calculate anything.

But one thing I don’t do very much is write formal mathematical equations or formal proofs. I’ve obtained a lot of symbolic mathematics software which makes manipulating equations quite easy and a lot of theorem proving software which makes proving theorems quite easy, but I still don’t do it very much. It has taken me many long years to figure out why that is.

For a long time I have found my own disinterest in equations puzzling. It’s not that I lack interest in equations found in books. I have often used a math text to find an important equation so I could translate it into a computer program. Especially in numerical analysis software it is often a quite mechanical task to translate the equation into a program, and that’s something I’ve often done. But I’ve never spent much time manipulating equations on my own — and have often felt at fault for not doing so, as if I am too stupid to do it creativly.

But in the past few years I’ve come to see this as a situation involving an inverse problem. Turning an equation from a book into a piece of software is quite mechanical, and in some programming languages like APL the resulting program looks a lot like the original equation. But the inverse operation is not so easy. It is not so easy to make a computer program (not derived from equations) that works, then turn it into a set of equations or a formal proof.

Similarly it is not very difficult to take an existing computer program that works and write an explanation of the underlying concepts and tell people how it works. But it is not so easy to go from concepts and understanding in the other (inverse) direction and make a piece of software out of them.

So, as you might guess, I have come to suspect an inverse problem at the heart of this. In recent years I’ve tried to describe this problem many times to many people without much success but maybe it will be easier now. My own code name for this is “SCIVTECH”, which I pronounce in a way that better reflects its linguistic roots as if it was spelled “skivtech”, and which abbreviates the distinction of “science versus technology”, “sci vs. tech”.

What Isaac Newton did was science, and his equations describing planetary motion have been held up as an example that has inspired many generations of scientists all over the world. But actually applying Newton’s equations and formulae is simply a matter of technology. One can download from the web some nice little computer programs that accurately predict the positions of the planets for hundreds of years in the future. Such programs are not at all intelligent — none of them does what Newton did when he figured out his scientific theory — they are just tools, applied science, techniques, or piece of technology.

It can be very hard to come up with mathematical equations modelling the real world and the history of modern physics illustrates just how hard it can be. But applying these equations is just a matter of technology. Not always easy, but just a matter of following where the equations lead.

So I see the inverse problem here as one of Science vs. Technology. Science is hard — often too hard. I think that’s why the so-called Social Sciences have been such a disappointment. They are trying to do the science, to come up with generalizations of human being and human society that might play a role analogous to Newton’s physical laws. That’s too hard. That’s the hard side of the inverse problem. The easier side is “Social Technology” and I intend to demonstrate just how easy that is by doing it.

But to talk of capital-S Science and captital-T Technology is to open philosophical beds of worms and breed arguments where what I really want to do is use tools and techniques to accomplish something useful, to make society work better.

It is much easier to talk of data compression and expansion, and to use them in situations involving other inverse problems that can be managed successfully only if we understand this vital distinction and avoid tilting at windmills. It is much easier to turn around and do the inverse, tilting away from windmills.

This exact distinction can be clearly seen in the work of psychologists who study the human personality, especially that of R.B. Cattell and his students. They asked people large numbers of questions and then used factor analysis to study the answers. The results were expressed as vectors (sequences of numbers) representing positions in a 20 dimensional space — more commonly reduced to 16 dimensions.

Apparently Cattell thought he was doing Science, but factor analysis is a mathematical technique based on the same methods of generalized fourier analysis as the cosine transforms in the software that makes jpeg compressed images. Just as we don’t think of these programs as little scientists that develop theories of pictures, just technology — tools, we shouldn’t really consider Cattell’s work as science. He was just prototyping various techniques and tools. He was doing technology.

CASA and the other projects described on these pages are also based on this technology. People will be asked to fill out questionaires and the results will be compressed into short descriptive vectors which compress or summarize the answers to the questions and thus the personalities of the people who supplied those answers. But the results are not going to be published as scientific discoveries. They will just be used to do something useful. In the CASA technology they will be used as inputs to other programs that will solve combinatorial optimization problems then turn the results into suggested matchings between people and people, people and jobs, people and other social activities, and so on.

No real science there, just technology. Social technology. The easy side of an inverse problem. Human beings are such complicated creatures and human society so much more complicated that only the easy side may ever be done. A general theory of people and human society has been proposed for centuries but we still seem a long way from it. I say that it is, like other kinds of science, the hard side of the inverse problem, and perhaps too hard to ever happen. But I’m sure we can manage the easy side and use what little we do know for something useful.

I’ve been talking and writing about this for many years but have somehow failed to get through to people. So instead of writing more long messages like this one and posting it to the SocialTechnology mailing list, I have been writing software to demonstrate and further develop the technology. It’s not all ready to go, but I do have a prototype — held together with duct tape, bailing wire, and epoxy cement, but more or less workable.

I’m now in the process of tidying up this prototype and installing it on the web for anyone to use. This is not going to be easy, so I hope I can get some help with it. As one way of getting help I am posting a request for it on every page I put up. Another way of making progress by myself is to put up as much as I can of what exists, with only minor changes, then invite people to play with it. This will take a while because there is a lot of junk to sort through.

I’ve made a firm committment to incremental progress — putting up a little bit each day and doing a little bit of testing of cleanup, testing, and proofreading as I go along. I’ve also make a firm committment to online progress reports, updated daily, at (or one of two other pages, and the included, or framed file at which is better for those on slow modems because it doesn’t need a frameset).

After years of trying I have managed to interest a very small number of people who might be willing to help with these projects. So far they fall into two groups: those only willing to work on this if it is a non-commercial open-source project and those only willing to work on this if it is a commercial venture to make money. There are so few people interested at all that I don’t want to risk losing half of them by opting for the other. So there will be pages about both.

The non-commerial open-source stuff is written up in various places, especially on the old index page, now at while the commercial venture is still described at (and various other places not linked from it). An overview of the obscenely large overall project of which these two are but components can be found at — it is not for the faint of heart.

I am posting this on the SocialTechnology mailing list and will soon turn it into a webpage. I don’t know if this should count as progress and be recorded on my progress page (listed above), but trying to put all this in words on the web has been an ongoing of mine for many years now so I’ll mention it there anyway.

Your comments and question will always be welcome, and soon there will be a guestbook on the website to make that easier.

Copyright (c) 1998 and 2000  Douglas P. Wilson

Posted in Old Pages | Leave a comment

The Real Plan

So far, the only plan is this:

  • Put everything that has been done so far on the web.
  • Beg for people’s help on every page
  • Appeal to the greed of the greedy who might see the commercial potential
  • Appeal to the idealism of the idealistic who might see the future of social technology for peace, prosperity, and an end to crime, poverty, exploitation, loneliness, ignorance, wasted lives, sexual predation and other social ills.
  • Hope somebody else with more common sense gets involved and makes the rest of us behave.
  • Meanwhile, work away at it, doing whatever we can do ourselves.

But how about a real project plan?   Some work has been done towards on, but not much yet.  A vast amount still needs to be done here, obviously.   A lot of text that might be incorporated has been written, but it badly needs revision.

Copyright © 2000 Douglas P. Wilson

Posted in Old Pages | Leave a comment

Term Extraction from Parallel Texts

One way to specify precise meanings is to use two words instead of one.   ‘Hide’ might mean a skin or ‘to conceal’.  ‘Skin’ might mean a piece of leather or ‘to flay’, to remove the skin of an animal.   But ‘hide/skin’ clearly means the outermost part of an animal or the leather coming from it, not ‘to conceal’ or ‘to flay’.   Word pairs (and sometimes triples) are fundamental in some of the research reported elsewhere in these pages.

Rather than using word pairs from one language, it is also effective to use pairs from two or more languages.   To make that easier for a person without vast linguistic abilities it is possible to extract pairs of words in two languages by comparison of parallel texts.  For this to succeed, non-bipartite weighted matching algorithms are needed.

In this work Biblical texts have been used because they are readily available in machine-readable form in many different languages.

Copyright © 2009   Douglas Pardoe Wilson

Posted in Old Pages | Leave a comment


This is a specific reference to the page Practical Immortality , as the phrase is used by the author of both these pages, DouglasPardoeWilson .

For most of my life I have read on average about one book a day, but I must confess that a large number of these have been Science Fiction books.  A surprisingly large number of these books have concerned not just aliens from other worlds, but humans or aliens who have transcended any form of physical bodies.

In many of such books the transcendent beings are like what we call spirits, capable of interacting with ordinary humans in terrifying or enlightening ways.  In more realistic books these transcendent beings are supported in some kind of computer system or systems.  A dimly recall one in which the entire surface of a planet, down to a depth of one mile, was a computer filled with human beings.

My own novels, which are much too large and elaborate to have any published form other than as files on the Internet, have a science fiction element as well, and also includes human beings living in transcendent form in a computer.  But I describe quite explicitly how this is possible, and try to demonstrate how realworld humans can have their entire personality and identity tranfered into a computer, so that the being in the computer is not just a copy of the person, but the actual person.  For this to happen the realworld body must either die or be placed in a coma, so that the being in the computer is most definitely the one and only person, not any kind of copy or simulation.

I view this as an entirely practical solution to the problem of immortality, and believe we should work for it.

A related issue is that of resurrection.  As I describe in my books, there can be a time delay between the person’s personality and identity being transcribed, after which the physical body dies or is placed in a coma, and the actual reappearance of that person in the computer’s virtual world.  There is no clear limit on that delay, so this can be form of resurrection of the dead.  Indeed, where sufficient information is elsewhere available, a dead person whose personality and identity where never intentially transcribed may be resurrected.

Though written in fictional form, I believe all of this is possible and practical.  It is something we should work towards.

Please read the brief excerpt from The Green Family Chronicles, on the page People Recovery .  This may outrage or make no sense to people who have not read Practical Immortality .

Copyright © 2009, Douglas Pardoe Wilson

Posted in Old Pages | Leave a comment

A Non-Arbitrary Language Composed Entirely of Acronyms

Since my imagination was first captured by the idea of a non-arbitrary ideal language, twenty-five years ago, this idea has gone through many variations, some with quite interesting properties.

Oddly enough I think the original  idea of  an acronymic language with its summarization and expansion properties has held up reasonably well. I think.   The summarization property was simply the ability to summarize any phrase, clause, or sentence in a single word by simply taking the initial letters of each word and concatenating them.   The idea that such an acronym would in fact  be an accurate summary of the phrase or sentence is really the defining property of a whole class of languages, of which I imagined there would be a single best language, The Acronymic Language, which might best be referred to as TAL, though for complicated reasons I usually used the unpronounceable acronym TGW instead.   The expansion property is obviously the inverse operation, though the ability of context to control expansion makes it at once less fundamental and more interesting.

So of the other variants of the idea have not fared so well, mostly because they run afoul of big numbers, the combinatorial explosion. When I first started actively working on the acronymic language– trying to actually create it, all by myself , I did not realize how many combinatorial difficulties would stand in my way.

Of all variants of the acronymic language, the one that is easiest to explain is one of the hardest ones to actually do — and is the site of one of my more egregious errors. I found myself try to explain the acronymic language to my niece Marika a few weeks back — without mentioning the project’s appellation. Most of what I what I said made no impact, but I caught a wee glimmer of understanding with the old “dictionary-order = thesaurus-order” idea.

I have somewhere on my shelves a spelling dictionary — just an alphabetical list of words, no definitions — a lot more words than there could be in a small paperback if definitions were included, and so quite useful if you know what the word means but have forgotten how to spell it.

I also have on my shelves several synonym-books or thesauri, none entirely satisfactory.

One of these is the huge Rodales Synonym-Finder which is like many of the pocket saurians school kids seem to prefer: no separate index, no rational scheme like Peter Mark Roget invented for his thesaurus, just plain old alphabetical order. The trouble with that scheme is obvious: instead of putting ‘cut’, ‘slice’, and ‘trim’ in one category where all the other synonyms can be found as well, there has to be a lot of repetition, in which almost identical lists of synonyms are provided for each word. And so, for all its size the huge Rodales book contains fewer synonyms per word than the Roget dinosaur it sits next to.

For years I dragged out the phrase ‘thesaurus order’ to describe something much like the large main list of Roget, ignoring the index.

Remembering that this is part of a description for an ideal language, not a scheme for use with English, it seems possible to claim that every word in the language has just 1 basic meaning, putting aside metaphorical uses, and that words that look similar, containing many of the same letters in the same order, would be approximate synonyms.

And so, finally, it seems possible to imagine a useful thesaurus in which each word is listed just once, next to various synonyms — between it’s two closest synonyms, in fact. No index would be necessary since this list of words would be in alphabetical order (for whatever alphabetical order is used in the language). Since the words would be in alphabetical order, the list would be identical with a spelling dictionary as described above: just a list of words in alphabetical order.

Well now. That’s quite an idea, really, but is it entirely coherent? Does it make sense? Is it logically consistent? And if it is a coherent, consistent idea, is it actually possible to create such a list? Surely I must answer yes to at least one of these questions or there’s no point in all this.

I think there is some point in all this, because each of those questions can “almost” be answered yes. All the questions about consistency and coherence must technically be answered “no”, because I have conflated two consistent ideas, but I can salvage something from them.

The question about the possibility of creating such a list can technically be answered “yes” for either of the two consistent ideas, which I’ll describe below, but only technically — some omnipotent deity might indeed create either list, but no human or computer is likely to because here’s where we run smack into one of those large numbers that I spend so much time trying to evade.

Well, my first task is to take cap in hand and apologize to you and the rest of humanity for conflating two ideas, each of which has some merit alone, but cannot logically be combined.

Briefly, it seems that if we are designing an artificial language, we can consistently envision a useful thesaurus-sort-of-list in which each word is listed just once, and that can be sorted in some alphabetical order, like a spelling dictionary, so that spelling will also indicate the meaning of each word. But it does not follow that each word in that list will be sandwiched between its two nearest synonyms.

On the other hand, we can consistently envision a somewhat similar list of words, in which again each word occurs exactly once, which does have the interesting property that each word does occur adjacent to its two closest synonyms.

The easier task to explain is the latter one: can we create such a list of words for our hypothetical language, in which each word is listed exactly once, between its two closest synonyms? As far as I can see this is an entirely consistent and coherent idea, and in principle it could be done.

But nobody is going to make such a list any time soon, because it is an NP-complete problem like the travelling salesman problem, which require a drastically increasing amount of processing as the number of nodes or points increases.

Indeed, if you imagine each word as a town on the salesman’s itinerary, with neighbouring towns as close synonyms, you should be able to see that it really is the travelling salesman problem in disguise.

In fact all NP-complete problems seem to be the travelling salesman problem in disguise, since there is a profound similarity amongst all problems of that type, such that the discovery of a method for solving any one of them in a reasonable amount of time would make it possible to solve all of them in a similar amount of time. As far as I know nobody has actually proven such a discovery impossible, but most mathematicians and computer science people seem to believe it is impossible.

I mention above that I had (to my shame) conflated two formulations of the problem. What is the other one? Suppose you have a alphabetical list of words like this sequence:

dib dic did dif … dix diz dob doc dod dof

Rather than thinking of ‘diz’ being a word intermediate in meaning between ‘dix’ and ‘dob’, it would be entirely consistent to suggest that all the words beginning ‘di’ are parallel in meaning to words beginning ‘do’ so that ‘dob’ is not similar meaning to the nearby word ‘diz’ but to the more distant word ‘dib’. Indeed, the sequence of words from ‘dib’ to ‘dif’ would then be seen as parallel to the sequence of words from ‘dob’ to ‘dof’.

It turns out that this problem is no easier to implement than the previous one. The relationship between them is the same as the relationship between two equivalent was of encoding numbers with a binary code, the ordinary binary code and the Gray code in which only one bit changes between adjacent numerical values. Had we solved either problem we could immediately use this relationship to solve the other.

Indeed, as I have worked with various versions of this problem over the years I have become quite attached to binary representations, instead of real or complex numerical representations. More about this at another time.

For now, all I want to say is that the NP-completeness of these problems is not a death sentence for the project. There are approximation schemes which could approximately solve either problem, and I’ve had some limited success in doing just that.

I want to replace the artificial example (the words from ‘dib’ to ‘dof’) with a real one for a small data set. How long with that take? Well, probably forever, if you are going to wait for an example that meets my impossibly high standards. But if you want a woefully inadequate example that just barely gets the point across, perhaps a few days.

Copyright © 1998 Douglas P. Wilson

Posted in Old Pages | Leave a comment

A Few Technical Notes

Since the following was written a new page with more technical information has been written.

Technical information will be found on many different pages here, but this will be a central location for finding and accessing all of it.  For now the term “technical” will refer to things involving the inner workings of computer and computer software, and to anything mathematical beyond the elementary level.  This is to unify to disparate communities, since many people including computer programmers and analysts are uncomfortable with the more mathematical sides of their own discipline, let alone others, and many professional mathematicians know little about computers and have never tried to program one.   But math and computers are related in many ways and experts in either are justifyably regarded by the rest of humanity as too wrapped up in technical details.   This page is for people who are indeed enraptured, so to speak, with all the technical details and like it that way just fine.

Eventually this page will give a quick summary and point the technical reader to other pages for those details, but for now it will just give a very quick summary.   This very large project covers a lot of subject matter with a fairly small number of techniques from two branches of computer science and their mathematical relatives.   Survey data from sociological studies and as much as can be collected here on the web from forms on this site will be analyzed using numerical methods of the usual kind with vectors and matrices, doing well known operations involving pseudoinverses, eigenvectors, and so on,  and then the results will be used as the input to a lot of combinatorial optimization techniques involving bipartite matching, minimum spanning trees, shortest path and travelling salesman problems, and so on.   The results will be used in a heuristic way, given to users as suggestions for social, political, or economic activity which they will be free to test, follow, or ignore as they please.   We hope to persuade them to provide feedback for us, so we can apply machine learning techniques including neural network modelling to produce suggestions that are better in two ways:

  1. The suggestions should catch their interest more, and so be more likely to be tested or followed
  2. The suggestions should be better overall, producing more successful social contacts and relationships.

This may sound something like some nerd or technogeek’s mad plan to get laid though devious high-tech means, which may not have been so very far from the truth, once  —  the originator of these ideas and perpetrator of this web site may even occasionally admit that such a motive had crossed his mind in earlier days.   But as he has aged and the fires have dampened down a bit, the overall idea has started to seem more valuable for it’s overall effect on society as a whole, and for everyone in it, and less like a path to personal pleasure.

The idea has grown considerably over the years, until it now looks like a method of performing a lot of different services for people.   The first form that has been posted at this site include a list of such services, which includes finding people jobs, friends, investments, places to live, people to share this with in various intimate or merely affectionate way, investors to fund projects and so on.   Another take on the possible services to society has been written up as the proposal for an Internet-based company , a commercial venture that looks like it might be very profitable for somebody.

Actually the project just described, for social matching and social search of various kinds is only one of several ambitious projects based on the same technology.   An overview of all (well, most) of them should illustrate the extent to which an overactive imagination can turn to a life of technological fantasy exceeding even the more animal fantasies of those years when the internal fires burned brightest.

For more information on this, see the home page , the progress report page , or various amusing pages at the personal site of the author of these lines, such as the discussion of Net Net Baud Rate , or the Bicycle and the Matching Transfomer .

Copyright © 2000 Douglas P. Wilson

Posted in Old Pages | Leave a comment

Technological Fantasies

This post was taken from the home page of a site about future technology.   I am less concerned with how practical the ideas are than I am with the ideas themselves.  I call these Technological Fantasies.

I used to have a very different web site of this name, expensively hosted with its own domain name.  Should have kept it alive regardless of cost, but didn’t. Anyway, I have have many technological fantasies and hope you have more. Here are some which evolved during the writing of a giant novel, called, of course, Technological Fantasies.

1. A system for actually finding compatible people for various reasons, one which would actually work, unlike such things as dating services.

See the FindCompatibles web page and blog listed here for information.

2. Practical immortality in a virtual world, but an actual transfer of personality to that medium, not just creation of a copy. See the Practical Immortality website and blog listed here for more information.

3. Small computers which are entirely wireless, getting power from sunlight, communicating by digital radio. One example from the book is a computer the size of a grain of sand. In that chapter one of the characters displays a whole handfull of such sand, which can not only communicate by digital radio but display text across the combined surfaces of the grains, which have microscopic LEDs on all surfaces. Indeed, using piezoelectric effects, this handfull of sand can receive and process spoken sound, then reply, again with piezoelectric vibrations.

4. A device to be worn, which has enough computer power and powerful enough digital radio links to other computers so it can draw on an enormous computing network. As well as the simple task of acting as a portable phone, this device would support a variety of simple but powerful commands. In the book I mention many, but in particular SysTalk which does not take contact information for any person as a parameter, but simply scans the globe for the best possible match, the person who is the best for you to talk to at that moment, (and the most likely to be free to take your call). Another command is SysTeach, which also takes no parameters but just begins or continues a lesson that is the most appropriate for you at that moment.

Oh, many many more things, my technological fantasies. I would like to hear yours. The complete novel, Technological Fantasies is now available, to be read online or downloaded. The links to these individual volumes may be broken, as they were put on a free site.  Please go to my books page if so. > Volume One   Volume Two   Volume Three   Volume Four   Clicking on these links will usually bring up the pdf file to read online.  To download the files you must save them.  For example in MS Windows you right click on the links then select Save Target As.

Copyright © 2008, Douglas Pardoe Wilson

Posted in Old Pages | Leave a comment