August 14,2008:
The war in Iraq has changed the
way military intelligence operations are conducted. This came about because
smaller combat units were now able to pass more information to the intelligence
specialists, and these troops now have more powerful hardware and software to
do something with the flood of new data. In the past, the brigade, division and higher level intelligence shops
conducted a lot of their own intelligence gathering (and had their own aircraft
and reconnaissance units for that purpose). In theory, they were supposed to be
getting a lot of information from subordinate units. But that was a paperwork
nightmare. The smaller units had lots of good intel, but they were out there
getting shot at, and had a hard time filling out reports. Intelligence
specialists were supposed to debrief patrols and troops involved in other
combat operations. But, as always, there were shortages of manpower to collect
the information and put it to use.
That began
to change in Iraq, where a lot of the debriefing got automated, and even the
smallest infantry patrol, could easily report their findings electronically. By
capturing the data electronically at the lowest level, and building database
tools to handle it, information did not expire (as battlefield information
tends to quickly do).
With lots
more fresh battlefield data available, it was now possible to do things that,
only a few years earlier, were believed the province of much larger, and better
funded, organizations. For example, there is data fusion. This is collecting
intelligence from many sources, and sorting through it for useful patterns and
items that, in seen in the right context, are very valuable. The first of these
fusion centers were set up at the national level two decades ago. But during
the course of the Iraq war, the concept moved down the food chain. Cheaper, and
more powerful, computer hardware was able to use analysis software to speed the
fusion process, even in the hands of a relatively inexperienced operator.
Some of the
most effective intelligence tools used in combat today, data mining and
predictive analysis, were invented a century ago along with the development of
junk mail. Who knew? For decades, the statistical tools used to determine who
to send junk mail to (so the sender would make a profit) were not much use to
the military. Then came cheaper, and more powerful computers, and the
development of data mining and analysis tools. This made a big difference,
because the more data you have to work with, the easier it is to predict
things. This has been known for centuries.
Now, with thousand dollar laptop computers equipped with hundred gigabyte (or
more) hard drives, you can put large amounts of data in one place, do the
calculations, and make accurate predictions. This wasn't possible thirty years
ago, when a 75 megabyte hard drive cost $45,000, and the computer doing the
calculations cost even more than that. You also didn't have digital photography
(more data you can store for analysis), or a lot of data, in general, stored
electronically. It's all different today. That hundred gigabyte hard drive
(holding over a thousand times more data than the $45,000 one of yore) costs
less than a hundred bucks.
In the last few years, intel analysts have
realized how powerful their tools are. And for those who studied math,
statistics or business in college, they know the power of data mining, because
it has become a very popular business tool. In places like Iraq and Afghanistan,
lots of data is being collected all the time. It was data mining that led to
the capture of Saddam, and the death of Zarqawi. Actually, over a hundred
senior (team leader and up) al Qaeda terrorists have been killed or captured in
Iraq using these techniques.
Data mining is basically simple in concept. In
any large body of data, you will find patterns. Even if the bad guys are trying
to avoid establishing a pattern to their actions, they will anyway. It's human
nature, and only the most attentive pros can avoid this trap. Some trends are
more reliable than others, but any trend at all can be useful in combat. The
predictive analysis carried out with data mining and other analytic tools has
saved the lives of hundreds of U.S. troops, by giving them warning of where
roadside bombs and ambushes are likely to be, or where the bad guys are hiding
out. Similarly, when data was taken off the site of the Zarqawi bombing, it
often consisted only of names, addresses and other tidbits. But with the vast
databases of names, addresses and such already available, typing in each item
began to generate additional information, within minutes. That's why, within
hours, the trove of data generated dozens of raids, and even more leads.
Speed has always been an advantage in combat,
but, until recently, rarely something intelligence analysis was noted for. No
longer. Predictive analysis is something the troops depend on, not only to tips
on what to avoid, but for names and places to go after.