Science and Technology
This page is a companion piece to one with the promising title of ” How It Works I should probably have one called ” Does It Work? and perhaps I will, soon enough, (those two links go only to placeholder pages at the moment) but that’s a big distinction that should not be glossed over. I’ve been trying to explain this distinction to people for at least 20 years now, without much success, but this time there is some hope of success.
Several other distinctions need to be made first.
Inverse or Trapdoor Problems
The most important distinction goes under the name “Inverse Problems”, which will be the title of a web page soon. This is shorthand for a well understood mathematical phenomenon that has been at the root of modern mathematics since Newton, at least. It has been very difficult to explain this to people until very recently when technology based on inverse problems has become commonplace.
When it’s ready I’ll refer you immediately to that Inverse Problems page for a better explanation but for now you should simply consider the nature of the public key cryptography now used all over the net. Public key cryptography is used in Pretty Good Privacy, PGP, and similar cryptographic systems based on the well-known mathematical phenomenon wherein two problems which are inverses of one another are known, but one of them is easy and one hard.
Usually the problems involved are multiplying two large numbers to get their even larger product and the inverse problem of factoring a very large number to demonstrate that it is the product of two smaller ones. When the numbers are integers, (sometimes called whole numbers), and the multiplication must be exact, the second problem, factoring, can be very very hard, while the first one remains easy.
Mathematicians and computer scientists often use the shorthand phrase “inverse problems” to describe this situation. It is a misleading term, but a common one. Strictly speaking multiplication and division are inverses, as are addition and subtraction, but the term “inverse problem” is usually reserved for the case when the extreme difficult of one of these inverses causes a big problem.
Again, strictly speaking, encryption using a public key and decryption using a private key are inverses, like the multiplication and division they are based on, but the shorthand term “inverse problem” is used here to refer to the much harder problem of cracking the code so as to decrypt the encrypted text without knowing the private key beforehand. Another way to describe this is to say that PGP uses a “trapdoor algorithm” — if you push on the trapdoor from one side it opens easily, but to make it open from pushing on the other side is hard, you have to actually break it down.
I’ll hope all the publicity given to cryptography in recent years means you understand the basic concept, so I can just move on to other examples and to the bigger picture they suggest.
Another inverse problem comes up in data compression, especially in “lossy” data compression. Compressing files with zip or gzip is carefully done to lose no information at all so that the expanded (or inflated, or exploded) text recovered by decompressing the compressed file later reproduces the original uncompressed file exactly, with nothing lost in the translation.
If you have zipped up large directories full of big files as I do so often you may have noticed that it takes longer to compress the files than it does later on to unzip them. Compression and decompression are obviously inverse operations, that goes without saying, but they are not equally easy to do, and so we start to use the term “inverse problems” to say in shorthand that the inverse of the easy operation is a hard problem.
But in lossy compression such as is used in making jpeg and mp3 files, some data is lost — the file that results from decompression is not exactly the same as the file that was compressed. The trick is to make sure the data that was lost wasn’t important anyway, mostly just noise, while preserving the important data that we can think of as the signal. Another, even more powerful form of lossy data compression is fractal compression where very high compression ratios are available through mathematical magic that almost nobody understands.
In all these lossy compression systems it is much easier to decompress the compressed file (see the jpeg, listen to the mp3 file) than it is to create those files by compressing the originals. In fractal compress the situation is even more extreme and doing the original compression operation is very difficult and time consuming, much much harder than decompressing later.
The mathematics behind CASA and its companion projects is almost entirely based on inverse problems, actually on methods very similar to those used in jpeg and mp3 compression. The results are then used in combinatorial optimization operations that are also mathematically challenging, but none of this would be practical without the lossy compression step.
On these pages I am trying to find a middle ground where there is none, in the middle of the vast crevass between C.P. Snow’s “Two Cultures”. I’ve spent a lot of time over the years in various kinds of mathematical activity that would interest the scientists and engineers on one side of the chasm but are mostly incomprehensible to the arts and humanities on the other side. But I’ve also spend a lot of time writing text in plain English (more or less plain English) which scientists and engineers often find to vague or imprecise — not technical enough. I am (somewhat) comfortable on both side of the gap, so I do both things.
As a result I (think I can) produce pages and programs so that others from either camp will find something of value in what I’ve done. Computer programs, especially the rather mathematical ones I usually write, are quite technial and formal objects — if written in a real language like ML they can even be proven correct. Pages of text like this one are very much informal objects that have no precise definition and aren’t meant to. They are meant to be roughly understood, not proven correct or used to calculate anything.
But one thing I don’t do very much is write formal mathematical equations or formal proofs. I’ve obtained a lot of symbolic mathematics software which makes manipulating equations quite easy and a lot of theorem proving software which makes proving theorems quite easy, but I still don’t do it very much. It has taken me many long years to figure out why that is.
For a long time I have found my own disinterest in equations puzzling. It’s not that I lack interest in equations found in books. I have often used a math text to find an important equation so I could translate it into a computer program. Especially in numerical analysis software it is often a quite mechanical task to translate the equation into a program, and that’s something I’ve often done. But I’ve never spent much time manipulating equations on my own — and have often felt at fault for not doing so, as if I am too stupid to do it creativly.
But in the past few years I’ve come to see this as a situation involving an inverse problem. Turning an equation from a book into a piece of software is quite mechanical, and in some programming languages like APL the resulting program looks a lot like the original equation. But the inverse operation is not so easy. It is not so easy to make a computer program (not derived from equations) that works, then turn it into a set of equations or a formal proof.
Similarly it is not very difficult to take an existing computer program that works and write an explanation of the underlying concepts and tell people how it works. But it is not so easy to go from concepts and understanding in the other (inverse) direction and make a piece of software out of them.
So, as you might guess, I have come to suspect an inverse problem at the heart of this. In recent years I’ve tried to describe this problem many times to many people without much success but maybe it will be easier now. My own code name for this is “SCIVTECH”, which I pronounce in a way that better reflects its linguistic roots as if it was spelled “skivtech”, and which abbreviates the distinction of “science versus technology”, “sci vs. tech”.
What Isaac Newton did was science, and his equations describing planetary motion have been held up as an example that has inspired many generations of scientists all over the world. But actually applying Newton’s equations and formulae is simply a matter of technology. One can download from the web some nice little computer programs that accurately predict the positions of the planets for hundreds of years in the future. Such programs are not at all intelligent — none of them does what Newton did when he figured out his scientific theory — they are just tools, applied science, techniques, or piece of technology.
It can be very hard to come up with mathematical equations modelling the real world and the history of modern physics illustrates just how hard it can be. But applying these equations is just a matter of technology. Not always easy, but just a matter of following where the equations lead.
So I see the inverse problem here as one of Science vs. Technology. Science is hard — often too hard. I think that’s why the so-called Social Sciences have been such a disappointment. They are trying to do the science, to come up with generalizations of human being and human society that might play a role analogous to Newton’s physical laws. That’s too hard. That’s the hard side of the inverse problem. The easier side is “Social Technology” and I intend to demonstrate just how easy that is by doing it.
But to talk of capital-S Science and captital-T Technology is to open philosophical beds of worms and breed arguments where what I really want to do is use tools and techniques to accomplish something useful, to make society work better.
It is much easier to talk of data compression and expansion, and to use them in situations involving other inverse problems that can be managed successfully only if we understand this vital distinction and avoid tilting at windmills. It is much easier to turn around and do the inverse, tilting away from windmills.
This exact distinction can be clearly seen in the work of psychologists who study the human personality, especially that of R.B. Cattell and his students. They asked people large numbers of questions and then used factor analysis to study the answers. The results were expressed as vectors (sequences of numbers) representing positions in a 20 dimensional space — more commonly reduced to 16 dimensions.
Apparently Cattell thought he was doing Science, but factor analysis is a mathematical technique based on the same methods of generalized fourier analysis as the cosine transforms in the software that makes jpeg compressed images. Just as we don’t think of these programs as little scientists that develop theories of pictures, just technology — tools, we shouldn’t really consider Cattell’s work as science. He was just prototyping various techniques and tools. He was doing technology.
CASA and the other projects described on these pages are also based on this technology. People will be asked to fill out questionaires and the results will be compressed into short descriptive vectors which compress or summarize the answers to the questions and thus the personalities of the people who supplied those answers. But the results are not going to be published as scientific discoveries. They will just be used to do something useful. In the CASA technology they will be used as inputs to other programs that will solve combinatorial optimization problems then turn the results into suggested matchings between people and people, people and jobs, people and other social activities, and so on.
No real science there, just technology. Social technology. The easy side of an inverse problem. Human beings are such complicated creatures and human society so much more complicated that only the easy side may ever be done. A general theory of people and human society has been proposed for centuries but we still seem a long way from it. I say that it is, like other kinds of science, the hard side of the inverse problem, and perhaps too hard to ever happen. But I’m sure we can manage the easy side and use what little we do know for something useful.
I’ve been talking and writing about this for many years but have somehow failed to get through to people. So instead of writing more long messages like this one and posting it to the SocialTechnology mailing list, I have been writing software to demonstrate and further develop the technology. It’s not all ready to go, but I do have a prototype — held together with duct tape, bailing wire, and epoxy cement, but more or less workable.
I’m now in the process of tidying up this prototype and installing it on the web for anyone to use. This is not going to be easy, so I hope I can get some help with it. As one way of getting help I am posting a request for it on every page I put up. Another way of making progress by myself is to put up as much as I can of what exists, with only minor changes, then invite people to play with it. This will take a while because there is a lot of junk to sort through.
I’ve made a firm committment to incremental progress — putting up a little bit each day and doing a little bit of testing of cleanup, testing, and proofreading as I go along. I’ve also make a firm committment to online progress reports, updated daily, at www.SocialTechnology.ca/progress.html (or one of two other pages, www.SocialTechnology.ca/status.html and the included, or framed file at www.SocialTechnology.ca/progplan.html which is better for those on slow modems because it doesn’t need a frameset).
Copyright © 2000 Douglas P. Wilson
Copyright © 2009 Douglas Pardoe Wilson
Other relevant content:
Please see these web pages:
The main Social Technology page.
Find Compatibles , the key page, with the real solution to all other problems explained
Technological Fantasies , a page about future technology
Social Tech a page about Social Technology, technology for social purposes. I think I was the first person to use this phrase on the Internet, quite a long time ago.
Roughly corresponding to these web pages are the following blogs :
Social Technology the main blog, hosted on this site, with posts imported from the following blogger.com blogs, which still exist and are useable.
Find Compatibles devoted to matching people with friends, lovers, jobs, places to live and so on, but doing so in ways that will actually work, using good math, good algorithms, good analysis.
Technological Fantasies devoted to future stuff, new ideas, things that might be invented or might happen, such as what is listed above and below.
Sex-Politics-Religion is a blog about these important topics, which I have been told should never be mentioned in polite conversation. Alright that advice does seem a bit dated, but many people are still told not to bring up these subjects around the dinner table.
I believe I was the first person on the Internet to use the phrase Social Technology — years before the Web existed.
Those were the good old days, when the number of people using the net exceeed the amount of content on it, so that it was easy to start a discussion about such an upopular topic. Now things are different. There are so many web pages that the chances of anyone finding this page are low, even with good search engines like Google. Oh, well.
By Social Technology I mean the technology for organizing and maintaining human society. The example I had most firmly in mind is the subject of Find Compatibles , what I consider to be the key page, the one with the real solution to all other problems explained.
As I explained on my early mailing lists and later webpages, I find that social technology has hardly improved at all over the years. We still use representative democracy, exactly the same as it was used in the 18th century. By contrast, horse and buggy transporation has been replaced by automobiles and airplanes, enormous changes.
In the picture below you will see some 18th century technology, such as the ox-plow in the middle of the picture. How things have changed since then in agricultural technology. But we still use chance encounters, engagements and marriages to organize our home life and the raising of children.
I claim that great advances in social technology are not only possible but inevitable. I have written three novels about this, one preposterously long, 5000 pages, another merely very very long, 1500 pages. The third is short enough at 340 pages to be published some day. Maybe. The topic is still not interesting to most people. I will excerpt small parts of these novels on the web sometime, maybe even post the raw text for the larger two.
This site includes many pages dating from 1997 to 2008 which are quite out of date. They are included here partly to show the development of these ideas and partly to cover things the newer pages do not. There will be broken links where these pages referenced external sites. I’ve tried to fix up or maiintain all internal links, but some will probably have been missed. One may wish to look at an earlier version of this page , rather longer, and at an overview of most parts of what can be called a bigger project.
Type in this address to e-mail me. The image is interesting. See Status of Social Technology
Copyright © 2007, 2008, 2009, Douglas Pardoe Wilson
I have used a series of e-mail address over the years, each of which eventually became out of date because of a change of Internet services or became almost useless because of spam. Eventually I stuck with a Yahoo address, but my inbox still fills up with spam and their spam filter still removes messages I wanted to see. So I have switched to a new e-mail service. Web spiders should not be able to find it, since it is hidden in a jpeg picture. I have also made it difficult to reach me. The picture is not a clickable link. To send me e-mail you must want to do so badly enough to type this address in. That is a nuisance, for which I do apologize, but I just don’t want a lot of mail from people who do not care about what I have to say.
Copyright © 2009 Douglas Pardoe Wilson