A research arm of the intelligence community wants to sweep up public data on everything from Twitter to public webcams in the hopes of predicting the future.

The project is the brainchild of the Intelligence Advanced Research Projects Activity, or Iarpa, a relatively new part of the spy community that’s supposed to help investigate breakthrough technologies. While other projects exist for predicting political events, the Open Source Indicators program would be perhaps the first that mines data from social media websites.

The idea is to use automated analysis to sift through the deluge of publicly available data to help predict significant societal events, like a popular revolution. The nascent project, called “Open Source Indicators,” is just the latest move by the national security community to come to grips with the flood of information now available on social media. As Danger Room’s Lena Groeger has reported, it’s also intended to predict natural disasters or economic disruptions.

The science underlying the project is the notion that early indicators of major social upheavals might be hidden in plain, socially-networked sight. “Some of these changes may be indirectly observable from publicly available data, such as web search queries, blogs, micro-blogs, internet traffic, financial markets, traffic webcams, Wikipedia edits, and many others,” the announcement, published August 25, says. “Published research has found that some of these data sources are individually useful in the early detection of events such as disease outbreaks, political crises, and macroeconomic trends.”

Indeed, social media sites, such as Twitter and Facebook, garnered major attention during recent events like the Arab Spring, and have been credited with helping to organize protesters and even foment revolution. Authoritarian governments trying to hold on to power noted the trend, and attempted at times to shut down access to those sites — and occasionally the Internet as a whole — in the hopes of stymieing efforts to organize protests.

The idea of the U.S. intelligence community culling data from social media is still a new one, and is likely to raise a number of questions. For example: what constitutes public data?

Iarpa, for its part, defines public data as “lawfully obtained data available to any member of the general public, to include by purchase, subscription or registration.” That raises its own host of questions, like whether the intelligence community could register a fake profile on Facebook, in order to “friend” people and obtain more information.

For those who fear the all-seeing surveillance state, Iarpa says there are some things the program won’t do. It won’t be used to predict events in the United States, for instance. Nor will it be used to track specific individuals.

– crossposted at the Medill National Security Zone

Photo: Lena Groeger