December 5, 2013: Information from the past is becoming a more important factor in present and future conflicts. In Iraq, American troops collected an enormous amount of data that is expected to be useful in the future. This included data and material collected after each (well, most) incidents where American troops were in combat. Typically troops are assigned to conduct their own CSI (Crime, or, in this case, "Combat" Scene Investigation) of each incident. In addition to several hundred thousand of these reports, there are about 100,000 just covering IEDs (improvised explosive devices, mainly roadside bombs) that include recovered fragments of the bombs. These databases are computerized, something which makes them a lot more useful for military researchers and planners. In fact, before the 1980s, when computerization became inexpensive and easy to implement, a lot of such military data was largely unknown to those who could use it and difficult to analyze in any event.   

For example, during the Vietnam War the U.S. Army began studying each combat incident (that produced American casualties) with the same through investigative techniques it had earlier adopted for accidents. The accident investigation was meant to find common patterns that could be changed to avoid future accidents. It was believed the same useful lessons could be learned from analyzing combat incidents. Thus, participants were interviewed, maps of the incident created, and even the fragments taken from dead or wounded troops was retained, along with some of their medical records. Those records survived the war but were "lost" until an enterprising Navy doctor (who began his military career as an enlisted SEAL in Vietnam and after medical school ended up teaching at the DoD Medical School) hunted them down in the 1980s and put the data on a computer for analysis. The results of that analysis was amazing (showing patterns of successful and unsuccessful attacks). So by the 1990s, with everyone in the military using a PC, and powerful (easy to operate) database software available, the military CSI drill became standard and increasingly useful. By the time the Iraq war came along, everyone was into thorough reconstruction of combat events. The reason was simple, to find out where the mistakes were and correct them. In the past the analysis had to be done using data stored on punched cards and subjected to analysis using card sorting equipment. The mainframe computers didn’t become affordable until the 1970s, and even then good statistical software and random data access was not cheap enough to make intense analysis possible. That all changed with the personal computer and microchip revolution that got started in the 1970s and matured by the 1990s.

This data revolution first made itself felt in Iraq. After the defeat of Saddam's armed forces in 2003, fighting continued with hit and run attacks by Saddam loyalists and Islamic radicals. These attacks hit a peak of about 30 a day in early November 2003. Each such incident was treated to the full CSI drill. Photos were taken, maps drawn, troops and witnesses interviewed, and damage (if any) inspected, along with enemy weapons or munitions (if available). Officers and NCOs then examined each incident and looked for things U.S. troops might have done to avoid getting hit, or to strike back at the ambushers. Since all of these incidents went into one database, it was also possible to look for patterns. Oddly enough, because of this CSI database, American investigators know more about enemy tactics than most enemy fighters. This is because the Iraqi opposition consists of several pro-Saddam or anti-Western factions and these groups cannot easily communicate with each other. While the attackers kept trying different types of attacks, coalition forces just as quickly developed new defensive measures. There's nothing really new in these small combats, for the basic ideas have been around for decades and most are clearly described in paper and CD-ROM versions of al Qaedas "how to be an Islamic terrorist" manual. But the CSI work lets commanders know how well defensive measures are working and which units are better at it than others. While the CSI operations take time and effort, the payback in information is literally a life saver.

As the fighting in Iraq and Afghanistan continued the U.S. added new technologies and hardware as they became available. The “computer revolution” of the 1970s was really the beginning of a tech revolution that saw a sharp increase in the availability of new technology. The military is still scrambling to make the most of these new and, for most, unexpected technologies. One of the more useful bits of tech was the new software that made it possible to quickly find patterns that were otherwise undetectable and indecipherable. That sort of thing had been developing for over a century (since the invention of the punch card and mechanical tabulating devices, a technology that lasted into the 1980s). Suddenly you not only had unimaginable amounts of data but you had it anytime, anywhere as long as you had an Internet connection. In theory this should have been a tremendous advantage for those who had it. But it did not work out that way immediately. It took a few years for the troops to get everything working together. Now it’s taken for granted that data, captured anywhere, can be quickly processed (scanned, translated, and analyzed for useful patterns) and return (sometimes in hours or less) useful leads.

It was largely luck that much of this database and networking magic showed up about the same time as September 11, 2001. As expected, in a new war there quickly developed unexpected new ways of fighting using the unexpected new technologies. Under the pressure of combat (which tends to dispense with all the usual peacetime delays and “we can’t do that” attitudes) solutions were quickly found that made the most of the new tools.

The best solutions were often the result of borrowing from commercial or government operations that were already using the new tech successfully. Thus the military was soon in touch with those that had already developed solutions to using the database/network/analytic software opportunities successfully. This connection was frequently made because of reservists (who were already users of these new tools) called up to serve in Iraq and Afghanistan. One of the most useful of these experienced users were the detectives, policemen, and other law enforcement specialists who found themselves under fire in Iraq or Afghanistan and wishing they had some of their new database tools from their civilian job. It wasn’t long before the army and marines had the new law enforcement tech and it changed warfare, and especially irregular warfare, forever.

A major innovation was the tremendous increase in the use of biometric (fingerprints, iris, facial recognition) identification. Before long the U.S. had developed tools that enabled combat troops to use biometrics on the battlefield. The main tool was called SEEK (Secure Electronic Enrolment Kit). This is a portable electronic toolkit that collects biometrics from people anywhere and at any time. This included fingerprint scans, eye (iris) scans, and digital photos of suspects. All this eventually ends up in a master database, which eventually contained data on millions of terrorists, suspected terrorists, their supporters, and other "persons of interest." Troops in the field can carry part of that database with them in their SEEK kits, so that wanted people can quickly be identified and captured. This is what the American commandos did on the 2011 Osama bin Laden raid. While DNA tests (which take hours to perform, on not-so-portable equipment) are the best form of ID, if you have fingerprints, iris scans, and a photo you are nearly as certain. Even just fingerprints and the face scan/photo is pretty convincing.

In Afghanistan the government used SEEK kits to collect data on nearly two million Afghans, so these people could be issued very secure (hard to fake) ID cards. For the government, this makes it more difficult for criminals, Taliban, and Islamic radicals in general to infiltrate the government or just operate freely. The U.S. has long been collecting biometrics from those they arrest or otherwise encounter and want to positively identify. This data makes it easier to figure out who is naughty and who is not.

All this began during the war in Iraq. At the same time the Department of Defense adopted many practices that major police departments had long employed to track down criminals. Troops in Iraq, especially reservists who were police, noted that the war in Iraq was mostly police work (seeking individual terrorists among a large population of innocent civilians). One of the more useful techniques for this is biometrics. That is, every time the troops encounter a "person of interest," they don't just take their name and address, they also use SEEK to collect the biometric data. The fingerprints are particularly useful because when they are stored electronically you can search and find out immediately if the print you have just lifted from somewhere else, like off the fragment of a car bomb, is in there or not. The digital photos, from several angles, are also useful because these pictures are run through software that creates a numeric "ID" that can be used by security cameras to look for someone specific, or for finding someone from a witness description. Other nations are digitizing their mug shots, and this enables these people to be quickly checked against those in the American database.

Often bomb makers are found because of fingerprints lifted from bomb fragments. Later raids frequently encounter suspicious characters but no evidence that justifies an arrest, until the fingerprints are checked against the bomb maker and SEEK database. The database of IED fragments were also checked for design techniques, which can indicate which individual or team built a particular bomb. This use of fingerprints led to the identification of over a thousand people involved in making bombs. At least a hundred were put on terrorist watch lists and many were eventually arrested or killed. This included several Iraqis who made it to the U.S. as refugees (along with 70,000 other Iraqis). These men were arrested and prosecuted.

It only takes about two minutes per subject to use SEEK to take the biometric data, so any suspicious characters are quickly added to the master database. Now, after several years of this, raiding parties know to grab any guy who seems to panic at the sight of the biometrics equipment coming out. The terrorists know that biometrics is bad news for them and they fear it. Combat troops now get training on how to use the biometrics gear and everyone now accepts that this stuff is a powerful weapon in the war against terrorists. Adapting this expertise to creating very difficult-to-fake IDs is not a large leap but it's not one that will result in many press releases.

There’s always a downside and for espionage organizations the use of biometric information for identification documents, like passports and those used to access heavily guarded facilities, has become a major problem. The use of biometrics does its job very well keeping out spies, terrorists, and saboteurs. The downside is that it also limits the activities of your own spies. This has led to efforts by espionage agencies to get around this "problem." The espionage organizations will not comment on what, if any, solutions they have come up with. That is to be expected.

While biometrics is one success story for the sudden appearance of the data advantage, there were many areas where getting the most out of data is still a work in progress. For example, military intelligence hoped that the networks and better data tools would make it easier to get new information from troops (just back or still in the middle of raids and patrols) to those who collect and analyze it and get the results of that analysis right back to the troops. Delays in doing this has always been a major problem because the analysis usually got back to the troops too late. Now, in theory, that should no longer be the case. But, all too often, it still is a problem. The chief obstacle is the troops and a lack of time, plus the distraction of being in a firefight or otherwise preoccupied with life-threatening situations. By trial and error solutions are being worked out.

Meanwhile, the intel analysts at all levels (battalion up to the very top) have learned a lot from the BI (Business Intelligence) industry, which has developed a lot of powerful research and marketing tools that have direct military application. This is all very geeky but the simple description is software that can quickly find patterns to huge quantities of data or activity. Thus the urgency with which troops grab enemy laptops or even large piles of paper records (even al Qaeda keeps lots of records). The troops know that quickly putting this stuff through a scanner, translation software, and analytics software will usually produce some new suspects to go after and often a current address as well.

The military also has lots of valuable historical data to mine. For example, there are detailed records of soldier and marine casualties. The army restricts access to the data, as it can provide the enemy with useful information on how effective they are. Some basic data is made public but the details will be locked up for a decade or more. Studying this data is a full time job for many people in the military, and there is a constant stream of suggestions resulting from this analysis, and those suggestions often turn into yet another small decline in combat deaths.