January 3, 2022: Several small commercial firms are producing software systems that enables non-military users to quickly, often instantly, analyze large quantities of digital data to track trends. Developers of this software are building on decades of work in this area plus improved communications and computer technology.

For example, one of these firms, Orbital Insight offers what they describe as AI (Artificial Intelligence) powered software that quickly performs geospatial analysis on huge collections of commercially available data from satellite, UAV, balloon and other images, along with phone geolocation data. This rapid analysis provides accurate predictions about vehicle sales, crop yields and insurance risks.

These techniques were first developed by the military after 2001, to detect and track threats. This included terrorist attacks or the actions of irregular fighters as well as the effectiveness of friendly forces. Intelligence agencies also used it to track the strategic (wide scale, long term) capabilities of nations and their military forces. The military and intel agencies often did this by adapting commercial security software. Now there is so much commercial data available that potential users find the services offered by firms like Orbital Insight affordable and effective for many business applications.

What made this all possible was the appearance, since the 1990s, of huge quantities of data and inexpensive supercomputers to analyze it. Supercomputers were first developed, as were the first computers, for military and scientific applications. These ultra-powerful computers are used for code breaking and to help design weapons (including nukes) and equipment (especially electronics). The military also needed a large amount of computing power for data mining. This involved pulling useful information about the enemy from ever larger masses of text or visual information. Because there's never enough money to buy all the supercomputers, which were super expensive, needed, military researchers came up with ways to do it cheaper. Back in the 1990s it was military researchers who figured out how to use GPUs (Graphic Processing Units, from high end graphic cards) for non-graphic computing. GPUs do something similar to what supercomputers do, lots of math calculations of a fairly simple type, and eventually the manufacturers of GPUs realized that there was a commercial (not just military) demand for GPUs serving as supercomputers. Currently most GPUs are used for non-video tasks, including rapid analysis of huge quantities of digital video and other data.

This is what the military needed to make video analysis software practical. Israel and the United States pioneered the development of software systems that take over the tedious job of watching video feeds from UAVs or security cameras. Some software apps specialize in video stakeouts of small areas as in a building or single doorway or window. The other major use was the analysis of recorded activity over a wider area and uses statistical analysis, and databases of known activity, to look for useful patterns. This sort of analyses worked but until GPU data computing came along, was often too slow to be effective.

This was the continuation of a long-term trend. Since the 1990s the U.S. has been using software to help scour satellite and aerial recon pictures for useful information. There were simply not enough trained photo analysts to examine the growing number of photos generated in the course of intelligence work. A related problem was the boredom of watching videos for hours. This problem was gradually alleviated by the use of pattern matching software that could detect movement that was in need of human attention. Research has shown that people staring at live video feeds start losing their ability to concentrate on the images after about twenty minutes. This problem has been known for some time, and the military, not to mention civilian security firms, have long sought a technological solution. It's actually not as bad with UAVs because the picture constantly changes but cameras that are staring at the same spot can wear operators out very quickly.

The basic tech solution is pattern analysis. By the 1990s there was a shift from analog to digital video and pictures. With digital it was easier to translate the video into numbers, and then analyze those numbers. Government security organizations have been doing this for some time but after the fact. It's one thing to have a bunch of computers analyze satellite photos for a week, to see if there was anything useful there. It's quite another matter to do it in real time. But computers have gotten faster, cheaper, and smaller in the last few years, and programmers have kept coming up with more efficient algorithms for analyzing the digital images. Commercial firms soon had software on the market that could analyze, in real time, video and alert a human operator if someone, or something you are looking for, appears to be there. Firms like Orbital Insight have taken this rapid analysis further.

Real time analysis software is rapidly evolving. You don't hear much about it because if the enemy knows the details of how it works, they can develop moves that will deceive it or, to be more accurate, make the pattern analysis less accurate. Russia used these deceptions during the Cold War to deceive the growing number of American digital photo satellites. The Russians called their deception technique maskirovka, which worked by ordering changes in normal activity by military or factory workers or equipment when the American photo-satellites were overhead. These large satellites could be spotted and tracked from the ground and their activities were usually predictable. The effectiveness of this maskirovka was not fully understood until the Soviet Union collapsed and Western intel agencies obtained data on what was really happening in Russia compared to the distorted maskirovka data photo satellites recorded. Maskirovka was known to exist during the Cold War, but the extent of its effectiveness was not. Current geospatial analysis can detect and adjust for the use of maskirovka but that’s a dynamic process. Those being observed are always coming with new deception to make their activities less predictable.

Early applications of image and activity analysis software were used as an adjunct to human observers and gradually took over. There will always be a human in the loop, to confirm what the software believes it has found. The new systems are possible because GPU data processing allows older video to be scanned faster than real time, allowing a lot of valuable information to be extracted from video taken years ago.

The proliferation of video cameras on the battlefield, in UAVs, ground robots, and for base security, has provided a huge library of images that show enemy forces doing what they do again and again. This can range from moving around carrying weapons, to using those weapons, to the particular driving patterns of people up to no good. This is a unique resource, and the U.S. Department of Defense has put together a library of these images. This is similar to older still pictures libraries, which were eventually used by pattern recognition software to let software examine the millions of images digital photo satellites began producing decades ago. The basic problem was that there were soon too many pictures for human analysts to examine. Computers had to do much of the work or else most of the images would go unexamined. This technology was quickly adapted to the kind of combat encountered in Iraq and Afghanistan and terrorist operations in general.

Now commercial firms like Orbital Insight are making natural disaster prediction more accurate and enabling more effective tracking of ship movements, including warships and smugglers. Commercial firms can analyze older data to see what they could have done better and adjust current methods to make their firms more efficient and profitable. It’s another example of the ancient effort to turn swords into plowshares.