The success of information theory in 1984 is not accidental. At this time, mathematical probability theory, mathematical statistics, mathematical logic, operational research, engineering communication technology, electronic technology, automatic control technology and so on are gradually maturing. Computers have appeared, and statistical mechanics, quantum mechanics and biology have provided important scientific methods. It was on the shoulders of our predecessors that Xiannong saw the dawn.
Shannon first described and quantitatively measured a series of basic concepts with rigorous mathematical methods, such as source, information, information quantity, channel, coding, decoding, transmission, reception, filtering and so on. , so that information research from the rough qualitative analysis stage into the precise quantitative stage, thus developing into a real scientific discipline.
The study of Morse telegraph coding will lead to the study of information with the concept of probability. For example, in English telegrams, "THe probability of the letter e is much greater than that of q, and the probability of the sequence th is much greater than that of XP." Therefore, Shannon further noted: "The basic problem of communication is to accurately or approximately reproduce the information selected by the sender at the receiving end of the information. The usual information is meaningful, ... and the semantic aspect of communication has nothing to do with engineering problems. " "What is important is that the actual message is always selected from the possible message set." In other words, Shannon realized two main points: (1) Communication engineering has nothing to do with semantics; (2) The information processed by the communication system is random in nature. So he thought that "information is something that can be used to eliminate uncertainty" and tried to define the amount of information accurately by probability method. Let the information source have n different symbols. X 1, x2…xn, and their occurrence probabilities are p 1(x 1), p2(x2)…pn(xn) respectively. Xiannong introduced the concept of information entropy;
Where k is a constant, which represents a certain degree of uncertainty of information sources and becomes a measure of information quantity. Information entropy borrows the idea of thermal entropy from the second law of thermodynamics in19th century. Here, Shannon does not regard information as meaningful news, but the degree of freedom when combining various element symbols in the information source into a message, that is, the uncertainty of the successive appearance of each element symbol. Obviously, the greater the degree of freedom, the greater the information. For example, if only B can appear after A and there is no chance of free choice, then the information content of messages A and ab is the same. So it is natural to describe the amount of information with uncertainty (or degree of freedom, or degree of confusion). It is this revolutionary idea that has become the cornerstone of classical information theory. If the logarithm in the definition of information quantity is based on 2, its unit is called bit by Shannon, and when it is based on 10 or e, the corresponding units are called decit and nat.
It is indeed a great breakthrough to investigate the communication theory with the concept of agricultural randomness first. The vigorous development of probability theory and mathematical statistics in the early 20th century provided suitable mathematical tools for information theory. He pointed out: "Discrete sources generate information symbol by symbol. The selection of consecutive symbols is based on some probabilities, which usually depend on the selection of previous symbols and the symbols to be selected. Any physical system or mathematical model of a physical system that can generate a sequence of symbols controlled by a set of probabilities can be called a stochastic process. " In this way, mathematical achievements such as stationary random process and statistical correlation theory are quickly transferred to the research field of communication process. The communication system described by Shannon is:
Source: represented by probability distribution p(x) on message symbol tables X and X.
Coding: the operation of converting source information into signals that can be input into channels. Mathematically, it refers to the function fn: xn → un.
Channel: it consists of input (beat) signal set u, output (received) signal set v and transition probability distribution matrix p(u v). Where p(u v) is the probability of receiving v in case of hitting u.
Decoding: the operation of restoring the channel received signal to the source message, that is, GN: VN → XN.
Shannon's other contribution to information theory is the description and study of the maximum capacity of the channel. Channel capacity refers to C units that can pass through the channel every second. Today, there are sources that output H units per second. One of his conclusions is that there is an upper limit C/H for the average rate of symbol transmission without noise interference, which can be close to but not exceeded. In order to extend this theorem to the actual situation with noise interference, Xiannong distinguishes two kinds of uncertainties. One is the "unnecessary" uncertainty caused by error transmission and noise interference, and the other is the uncertainty of the "need" of the source itself. So the useful information should be the uncertainty of the source minus the ambiguity caused by noise. Shannon pointed out: when C>h, the transmission error can be very small; When c is less than h, the error is difficult to control. Therefore, the "unnecessary" uncertainty must be greater than or equal to H-C. Finally, at least one coding can reduce the ambiguity caused by noise and make it very close to H-C. Shannon gave a strict expression of this important theorem:
The basic theorem of information theory. It is known that a system with a capacity of c >: 0, for any ε >, any signal (data) is taken from the traversing source; 0 and a positive number r (0
Shannon's information theory research first deals with discrete random variables, and then it can be easily extended to continuous situations, with little need for major revision.
Shannon's original paper published in 1948 did not expect to be widely used in communication systems. However, people soon realized that the tools provided by Shannon were very effective for practical communication. For example, there are often some redundant information in communication. When we send electricity, we always try to compress the number of words and reduce redundancy. However, Shannon pointed out that information redundancy is often useful, but eliminating it may not be beneficial. Redundancy in English is about 50%. For example, a message that "people are not sages, to err is human" is transmitted and received as "people are not sages, to err is human", which is generally understandable by some knowledgeable people. However, if we send and receive "Yankee hits Red Man" as "Yankee hits Rebs", although only one letter (D becomes B) is wrong, the meaning can't be guessed. This is because the redundancy of the two messages is different.
Shannon's original paper published in 1948 "shocked the scientific community like a blockbuster". Brand-new ideas and original methods have become a model for people to follow for a time. It is not only exquisite and beautiful in theory, but also provides a new tool for solving many engineering and technical problems, brings new hope and has high practical value. However, there are also some abnormal situations. In 1956, Shannon once said: "In recent years, information theory has simply become the most fashionable subject. It was originally just a technical means adopted by communication engineers, but now it will occupy an important position in general magazines and scientific publications. ..... it's exaggerated. Many colleagues in different disciplines, either admiring its name or seeking new methods of scientific analysis, have introduced emerging theories into their respective fields. In a word, information theory is famous now. This reputation certainly makes people in our line feel happy and excited, but it also breeds a danger ... Information theory is by no means a panacea for communication work, especially for others. You know, it is very rare to uncover all the mysteries of nature at once. Otherwise, people will be disappointed once they know that only the touching words of information, entropy and redundancy can't solve all the problems. That artificial prosperity will suddenly collapse overnight. "
Shannon's suggestion was taken seriously in the second half of 1950s, which put the development of information theory on a healthier track. In the past decades, modern information technology has developed rapidly on the basis of Shannon, such as signal mathematicization, microwave technology, satellite communication and so on, which has become a huge industry. Due to the rapid development and popularization of electronic computers, information science is infiltrating into almost all disciplines of social science and natural science, such as medicine, biology, genetic engineering, linguistics, psychology, management, economics, and even into people's daily lives. This situation is very rare in the history of mathematics and even the whole history of science.
Xiannong reached the peak of scientific career in the 1940s and 1950s, and continued to publish scientific achievements and do some important work. For example, the paper 1959 put forward the discrete source coding theorem under the fidelity criterion, which later developed into the "information rate distortion theory" and became the theoretical basis of band compression and data compression. At 1967, I also saw him co-publishing articles with others. Then gradually withdrew from the historical stage. 1980 retired and spent his old age in Boston.