March 10, 2016: National intelligence services (like the CIA and MI6) continue to find themselves relying more and more on civilian sources for the best data and analysis. A recent example was revealed because of all the anxiety over the huge numbers of illegal migrants trying to get into Europe and other Western countries, many of them by boat. Turns out that the best tool for reducing the use of ships for smuggling was an Israeli firm that built a business on creating a database of normal, and abnormal (and usually illegal) behavior by ships at sea for shipping and maritime insurance companies.

This data was easier to collect since the 1990s when all larger ships were required to use the AIS (Automated Identification System) which is essentially an automatic radio beacon (transponder) that, when it receives a signal from a nearby AIS equipped ship, responds with the ship's identity, course, and speed. This is meant to enable AIS ships to avoid collisions with each other. An AIS activity database makes it possible to identify patterns of normal and abnormal behavior. The abnormal behavior, like arriving outside a port and waiting for several days to enter, is what smugglers are often forced to do to avoid arrest. Same thing with travelling outside the most efficient (in terms of fuel used and weather encountered) routes. With enough of this data and a thorough analysis it is very difficult for seagoing criminals to escape detection. Now that navies and coast guards are increasing using this “maritime BI (Business Intelligence)” tool to more quickly shut down the criminal gangs making over a billion dollars a year from all this people smuggling.

AIS is also used to send ships important traffic and weather information. AIS is one of two ship tracking systems required, by law, for most ocean going ships. INMARSAT (International Maritime Satellite) is a more elaborate and longer range system. It enables shipping companies to keep track of their vessels no matter where they are on the planet. INMARSAT uses a system of satellites, which transmit AIS-like signals to anywhere on the oceans. It only costs a few cents to send an INMARSAT signal to one of your ships and a few cents more to receive a reply. Shipping companies have found the INMARSAT a useful business tool as well as a safety feature.

These two systems are now required by law (international agreements) for all sea going vessels greater than 300-tons. The technology has worked, and the U.S. Navy has found them particularly useful in counter-terror operations. Coast Guards the world over have also found the systems a big help. But apparently pirates in some areas have gained access to the systems (via bribes or theft) and a large number of pirate attacks appear to have been helped by technology meant just to safeguard ships at sea. Iran, and other nations involved in smuggling, learned how to have INMARSAT send a false signal, concealing where the ship actually is. This can work for a while but a nation with lots of recon satellites, warships, and cooperation from most of the world’s shipping can get around this.

The use of AIS data is part of a trend in dual-use intelligence tools that depend on OSINT (Open Source Intelligence). While the U.S. intelligence community long resisted recognizing the importance of OSINT, especially after the Cold War ended in 1991, the enthusiastic acceptance of Internet-based OSINT by so many individual military personnel and commercial information gatherers has led to enthusiastic official government acceptance of what many intelligence professionals now consider a crucial tool and one that can only grow in usefulness.

The Internet has made OSINT a really, really huge source of useful intelligence. It's not just the millions of gigabytes of information that is placed on the net but the even more voluminous masses of message board postings, blogs, emails, and IMs (instant messaging) that reveal what the culture is currently thinking. It was corporate intelligence practitioners who alerted the government intel people to the growing usefulness of Internet based data. Even before the Internet became a major factor in the late 1990s corporations had developed a keen interest in gathering intel on competitors, new markets, and all manner of things that might affect them. The Internet has made this a much more useful and affordable exercise, especially since corporations are less likely to break the law when gathering intel, or have access to the powerful legal tools available to government investigators and analysts.

For years corporate intel specialists were concerned that government agencies, especially the CIA, were not taking sufficient advantage of OSINT. Part of the problem was cultural. The intelligence agencies have always been proud of their special intel tools, like spy satellites, electronic listening stations, and spy networks. Most of these things are unique to government intelligence operations. People who use this stuff tend to look down on a bunch of geeks who simply troll the web. Even when the geeks keep coming up with valuable stuff, they don't get any respect. That began to change after September 11, 2001, when many intelligence specialists, who were reservists, were called to active duty. Many of these men and women worked in BI (Business Intelligence, sometimes called corporate spying) and brought with them a respectful attitude towards OSINT and spectacular (to the government intel people) ability to use it.

Before long many junior members of the intel agencies were using OSINT more frequently. Then it was pointed out that there was growing evidence that some foreign countries were exploiting OSINT (especially the Internet) more effectively than the United States. No foreign intel agency will admit to this, but there are indications that some nations are mining the Internet quite intensively and effectively. Data mining is a heavily used commercial tool that the U.S. intel agencies have used, but now they have adopted the corporate techniques of plowing through vast quantities of unclassified data and often finding gold.

An example of this official acceptance occurred in 2012, when the U.S. Army issued a manual, Army Techniques Publication 2-22.9, which detailed how to use open source (mainly searching the Internet) intelligence most effectively. This was the kind of OSINT troops had been using for over a decade. The publication of ATP 22.9 was a way for the senior army leadership to say, "message received and understood." ATP 22.9, despite all the useful tips it contains, won't go far in helping the many soldiers already using the Internet, but it will be useful in convincing their bosses that a lot of useful stuff can be obtained from the Internet.

The government and military intel community has the money and software chops to screen and analyze huge quantities of data on the Internet, both text and pictures. Despite all these resources the intel behemoths continue to get overtaken by civilian amateurs. A large factor in this was the appearance of Google Earth and other commercial satellite photo sources. This revolutionized military intelligence and the way news on military affairs is developed and spread. Case in point was details on the transformation of the Chinese armed forces and the activities of the North Korean military. Both China and North Korea have long been very secretive about military affairs. But the appearance of Google Earth (originally as Earth View) a decade ago changed everything. By putting so much satellite photography at the disposal of so many people, in such an easy- to- use fashion, unexpected discoveries were made.

People soon discovered that if they had a high-speed Internet connection, they could use Google Earth to find satellite photos of all sorts of interesting stuff. This was especially true of the "Forbidden Kingdoms" (China, Russia, North Korea, and a few others). While the CIA and the military has had access to satellite photos of these countries since the 1960s, little of it was shown to the public. Now that so many people can examine these lower resolution civilian satellite images amazing new discoveries are being made. Many of these commercial satellite photos cover vast stretches of the Forbidden Kingdoms that previously were only scrutinized by a few intel agencies. But the greater number of civilians found things that were newsworthy and never reported before. Things like new military bases, test sites for new weapons, and the new weapons themselves. The open discussion of these findings, most of them already known to the large national intel agencies, brought forth insights and analysis that was often superior to what the much smaller number of professional analysts were capable of. Another example of “the wisdom of the crowd.”

Technically, the countries in question can request that Google not show these classified military facilities. But in making that request, they point out where the classified operation is. So far, a lot of this stuff is just there to find. And users find it. This is called "crowdsourcing" (where large numbers of people accomplish impressive feats of research or analysis because they can quickly mobilize and get to the task via the Internet). The U.S. military will not say that they appreciate the work done via crowdsourcing, but individual analysts and intelligence officials have made it known, unofficially, that crowdsourcing is another useful tool that unexpectedly came their way via the Internet.