Earlier this month at an undisclosed location in the Middle East, computers using special algorithms helped intelligence analysts identify objects in a video feed from a small ScanEagle drone over the battlefield.
A few days into the trials, the computer identified objects — people, cars, types of building — correctly about 60 percent of the time. Just over a week on the job — and a handful of on-the-fly software updates later — the machine’s accuracy improved to around 80 percent. Next month, when its creators send the technology back to war with more software and hardware updates, they believe it will become even more accurate.
It’s an early win for a small team of just 12 people who started working on the project in April. Over the next year, they plan to expand the project to help automate the analysis of video feeds coming from large drones — and that’s just the beginning.
“What we’re setting the stage for is a future of human-machine teaming,” said Air Force Lt. Gen. John N.T.“Jack” Shanahan, director for defense intelligence for warfighter support, the Pentagon general who is overseeing the effort. Shanahan believes the concept will revolutionize the way the military fights.
“This is not machines taking over,” he said. “This is not a technological solution to a technological problem. It’s an operational solution to an operational problem.”
Called Project Maven, the effort right now is focusing on helping U.S. Special Operations Command intelligence analysts identify objects in video from small ScanEagle drones.
In coming months, the team plans to put the algorithms in the hands of more units with smaller tactical drones, before expanding the project to larger, medium-altitude Predator and Reaper drones by next summer.
Shanahan characterized the initial deployment this month as “prototype warfare” — meaning that officials had tempered expectations. Over the course of about eight days, the team refined the algorithm, six times.
“This is maybe one of our most impressive achievements is the idea of refinement to the algorithm,” Shanahan said.
Think of it as getting a new update to a smartphone application every day, each time improving its performance.
Before it deployed the technology, the team trained the algorithms using thousands of hours of archived battlefield video captured by drones in the Middle East. As it turned out, the data was different from the region where the Project Maven team deployed.
“Once you deploy it to a real location, it’s flying against a different environment than it was trained on,” Shanahan said. “Still works of course … but it’s just different enough in this location, say that there’s more scrub brush or there’s fewer buildings or there’s animals running around that we hadn’t seen in certain videos. That is why it’s so important in the first five days of a real-world deployment to optimize or refine the algorithm.”
While the algorithm is trained to identify people, vehicles and installations, it occasionally mischaracterizes an object. It’s then up to the intel analyst to correct the machine, thus helping it learning.
The team has paired the Maven algorithm with a system called Minotaur, a Navy and Marine Corps “correlation and georegistration application.” As Shanahan describes it, Maven has the algorithm, which puts boxes on the video screen, classifying an object and then tracking it. Then using Minotaur, it gets a georegistration of the coordinates, essentially displaying the location of the object on a map.
That’s new, it’s different and it’s much needed for an analyst because this was all being done manually in the past,” the general said.
“Having those things together is really increasing situational awareness and starts the process of giving analysts a little bit of time back — which we hope will become a lot of time back over time — rather than just having to stay glued to the video screen,” Shanahan said.
After the Predator and Reaper video feeds get the algorithms, the plan is to put them to work on Gorgon Stare, a sophisticated, high-tech series of cameras carried by a Reaper drone that can view entire towns.
“When you look at the data labeling that has to go on, the algorithms that have to be trained and refined, that’s really what I would call the PhD-level problem that we have up next,” Shanahan said of pairing the algorithms with Gorgon Stare.
Right now, the algorithms reside in the computers that receive the video from the drones. At some point down the road, the goal is to put the technology “at the edge” on the drones themselves as well.
“The combination of those two is very powerful,” Shanahan said. “We see redundancy as important in a future world in which you may lose the ability to communicate back to big enterprises in the United States.”
The algorithms use commercial technology, which has allowed the project to move quickly — lightning fast by government standards.
“We are not trying to do something over in the department that is already being done incredibly successfully in the commercial world,” Shanahan said.
Former Deputy Defense Secretary Bob Work stood up the project in April. Two months later, they received funding from Congress and six months later the first algorithms were used on the battlefield, delivering on a promise to reach combat by year’s end.
“We are learning lessons every day for the first time about how do you actually integrate AI into Department of Defense operationally fielded programs, not research and development, not test beds, but capabilities that are being used by warfighters day in and day out,” Shanahan said.
