April 7, 2010: The U.S. Department of Defense, based on experience in Iraq, is developing an automated intelligence analysis and data distribution (to troops who need it) system. NEOS (Next Generation Exploitation Optimization System) uses off the shelf hardware and software, to create a system that will use past experience to collect and analyze all the electronic data being collected by sensors over and around the battlefield, and then determine which data is useful for which troops on the ground or in the air. NEOS then alerts those troops and sends the data. Actually, it sends an interpretation of the data. The electronic sensors, usually onboard aircraft or UAVs, senses radio and cell phone signals, as well as the electronic noise broadcast by generators, vehicles and other equipment. Troops tuned in to NEOS get a text report of what is squawking (electronically) in their vicinity, with advice on what it might mean. Troops can connect to NEOS via laptops in their vehicles or smart phone type devices. All this came from the way troops learned how to cope with the growing flood of data they found themselves gathering in Iraq.

The Iraq experience changed the way military intelligence operations are conducted. This came about because smaller combat units, and a growing number of sensors, were now able to pass more information to the intelligence specialists, and these troops now have more powerful hardware and software to do something with the flood of new data. In the past, the brigade, division and higher level intelligence shops conducted a lot of their own intelligence gathering (and had their own aircraft and reconnaissance units for that purpose). In theory, they were supposed to be getting a lot of information from subordinate units. But that was a paperwork nightmare. The smaller units had lots of good intel, but they were out there getting shot at, and had a hard time filling out reports. Intelligence specialists were supposed to debrief patrols and troops involved in other combat operations. But, as always, there were shortages of manpower to collect the information and put it to use.

That began to change in Iraq, where a lot of the debriefing got automated, and even the smallest infantry patrol, could easily report their findings electronically. By capturing the data electronically at the lowest level, and building database tools to handle it, information did not expire (as battlefield information tends to quickly do).

With lots more fresh battlefield data available, it was now possible to do things that, only a few years earlier, were believed the province of much larger, and better funded, organizations. For example, there is data fusion. This is collecting intelligence from many sources, and sorting through it for useful patterns and items that, in seen in the right context, are very valuable. The first of these fusion centers were set up at the national level two decades ago. But during the course of the Iraq war, the concept moved down the food chain. Cheaper, and more powerful, computer hardware was able to use analysis software to speed the fusion process, even in the hands of a relatively inexperienced operator.

Some of the most effective intelligence tools used in combat today, data mining and predictive analysis, were invented a century ago to support the development of junk mail. Who knew? For decades, the statistical tools used to determine who to send junk mail to (so the sender would make a profit) were not much use to the military. Then came cheaper, and more powerful computers, and the development of data mining and analysis tools. This made a big difference, because the more data you have to work with, the easier it is to predict things. This has been known for centuries.

Now, with thousand dollar laptop computers equipped with several hundred gigabytes of storage, you can put large amounts of data in one place, do the calculations, and make accurate predictions. This wasn't possible thirty years ago, when a 75 megabyte hard drive cost $45,000, and the computer doing the calculations cost even more than that. You also didn't have digital photography (more data you can store for analysis), or a lot of data, in general, stored electronically. It's all different today. That thousand gigabyte hard drive (holding several thousand times more data than the $45,000 one of yore) costs less than a hundred bucks.

 In the last few years, intel analysts have realized how powerful their tools are, and what the potential could be. For those who studied math, statistics or business in college, they know the power of data mining, because it has become a very popular business tool. In places like Iraq and Afghanistan, lots of data is being collected all the time. It was data mining that led to the capture of Saddam, and the death of Zarqawi. Actually, over a hundred senior (team leader and up) al Qaeda terrorists have been killed or captured in Iraq using these techniques.

Data mining is basically simple in concept. In any large body of data, you will find patterns. Even if the bad guys are trying to avoid establishing a pattern to their actions, they will anyway. It's human nature, and only the most attentive pros can avoid this trap. Some trends are more reliable than others, but any trend at all can be useful in combat. The predictive analysis carried out with data mining and other analytic tools has saved the lives of hundreds of U.S. troops, by giving them warning of where roadside bombs and ambushes are likely to be, or where the bad guys are hiding out. Similarly, when data was taken off the site of the Zarqawi bombing, it often consisted only of names, addresses and other tidbits. But with the vast databases of names, addresses and such already available, typing in each item began to generate additional information, within minutes. That's why, within hours, the trove of data generated dozens of raids, and even more leads.

Speed has always been an advantage in combat, but, until recently, rarely something intelligence analysis was noted for. No longer. Predictive analysis is something the troops depend on, not only for tips on what to avoid, but for names and places to go after. NEOS extends this to all electronic transmissions being collected, and instantly lets troops in the area know who might be planning to do what to them.