Chinese researchers have developed rudimentary software that automatically turns video clips into comic books, building on the comic book industry’s increasing use of tools to assist with the artistic process.
Researchers at the Hefei University of Technology published a paper on the project, which generates stylized frames and places speech bubbles above the characters who are talking.
“As a type of artwork, comics is prevalent and popular around the world,” reads the abstract. “However, despite the availability of assistive software and tools, the creation of comics is still a labor-intensive and time consuming process.”
The software, which is called Movie2Comics, maps the faces of actors to match them with dialog, picks appropriate frames to extract, and applies a filter to achieve the comic book effect – which looks like a rather crude posterization effect in available samples. Their primary goals, according to the team, were to preserve narrative information from the clip and to output content that matches artistic convention in comics and graphic novels.
Figuring out which character was talking posed a challenge. Researchers ultimately achieved about 85 percent accuracy in assigning speech bubbles, which they hope to improve, according to PhysOrg. Narrative pacing is also challenging for artificial intelligence algorithms, which still struggle to make sense of human speech.
The team may also develop an interface so that a human can tweak the generated content.
Researchers tested the software by generating comic book scenes from Hollywood films including “Titanic” and “Sherlock Holmes,” and evaluating how well users understood the comics compared to the original film clip. There was a minor decrease in comprehension from the clip to the comic book, which was partially attributed to poorly-assigned speech bubbles.
“We have two future plans,” researcher Meng Wang told PhysOrg. “One, improve the performance of each component, such as script-face mapping, and hope we can generate perfect clips without user interaction. Two, integrate speech recognition technology to generalize the software, such that we can generate comics without movie scripts (currently we need the script file of the movie clip, but by using speech recognition we can automatically transform speech to texts).”
As impressive as the samples look, some comic book aficionados may worry the software will be used to generate a new generation of critically panned comic book adaptations of feature films.
The Hefei University of Technology, which was founded in the mid-20th century and is run by the Chinese Ministry of Education, is located in Anhui Province.