Music by the Numbers
Pictured above: Frederick Bianchi, professor of music, and the technology behind the Virtual Orchestra.
You settle into your seat to enjoy a performance of Les Misérables. The lights dim, the overture starts, and the brass, woodwinds, and percussion begin working their magic. You peer into the orchestra pit where instead of the dozens of musicians you expect to see creating those sounds, there’s only a handful. Sharing the pit with the performers is a computer that provides the rest of the instrumentation as someone gently taps on a keyboard to sync the performance of the virtual instruments to the tempo set by the conductor.
by Ami Albernaz
Over the past two decades, the Virtual Orchestra, or “Sinfonia,” has been part of more than 200,000 performances — from Broadway, to London, to Las Vegas, to scores of regional theatre companies around the country. Developed by Frederick Bianchi, professor and director of computer music research at WPI, the Virtual Orchestra makes musical theatre productions, operas, and similar events realistic and more cost-effective to stage and to take on the road. Given the cuts that have decimated arts budgets, the technology has been a lifeline, making it possible for many small theatre companies to continue to bring their patrons top-notch musical entertainment.
The Virtual Orchestra has come a long way since Bianchi began working on it in the mid-1980s. A musician and composer long interested in discovering new ways to control music electronically — whether through tape recorders or synthesizers — he first imagined the system when he was a professor at the College-Conservatory of Music, University of Cincinnati.
The idea was to simulate an orchestra in real time — not only sounding like an orchestra, but behaving like one, following the tempo of the conductor, getting louder, getting softer.— Frederick Bianchi
A rudimentary version was used in 1989 to perform the score from the opera Iphigenia in Tauris. Since then, years of refinement — along with continuous advances in computer processing speeds and memory — have produced a sophisticated, integrated, cutting-edge interactive system. Its flexibility makes the virtual orchestra far more versatile than prerecorded music, which is still commonly used in Las Vegas shows and on Broadway.
“The idea was to simulate an orchestra in real time — not only sounding like an orchestra, but behaving like one, following the tempo of the conductor, getting louder, getting softer,” says Bianchi, whose New York–based company, Realtime Music Solutions, continues to develop and market the technology worldwide.
Bianchi’s creation has not been without controversy, as musicians’ unions in New York and London have vehemently protested it. Yet Bianchi emphasizes that his invention was intended to integrate with live musicians, not replace them.
“The whole idea was to enable theatre and opera companies to overcome size, space, acoustic, aesthetic, and economic logistics indigenous to the late 20th century,” he says. “If you look at regional theaters in this country, many are now doing shows they might not otherwise have done because they didn’t have the budget, space, or enough musicians.”
Also, as technology has come to play an ever greater role in music production, the virtual orchestra has been accepted as common practice in modern theatre performances. It has been used in the National Broadway tours of Jekyll & Hyde, Phantom of the Opera, Titanic, Miss Saigon, Oklahoma, and Les Misérables, as well as several Cirque du Soleil productions.
While the Virtual Orchestra is a proven technology, Bianchi is always looking to improve it and expand its capabilities. For example, over the years he has experimented with innovative ways for conductors and musicians to interface and interact with the system. In his current research he is looking to move beyond control by physical gesture as he explores ways to untether musicians from physical interfaces, which he says will never be able to keep up with the human bandwidth. “The real-time capture and interpretation of brain activity using functional MRI and other analysis methods may well be the final frontier in understanding and optimizing our creative abilities,” he says.
Technology Meets the Choir
Using a computer to simulate the sound of musical instruments is one thing. But can the same technology create a believable replica of the sound of human voices? That is the challenge Bianchi and John Delorey, director of choral music at WPI, are tackling through their “virtual choir” project, which makes use of the Virtual Orchestra platform to create a vivid and flexible simulation of the complex intertwining of bases, tenors, altos, and sopranos.
Delorey and a group of 11 WPI students demonstrated the technology at St. Paul’s Covent Garden in London during the summer of 2011. Human and virtual singers performed the 16thcentury masterpiece Spem in Alium by English composer Thomas Tallis. The composition, which includes 40 vocal parts, is notoriously difficult for today’s choirs, which are used to performing pieces with just four parts. All 40 vocal parts were carefully recorded by WPI students in a studio in Alden Memorial and then programmed into the Virtual Orchestra system. As with the orchestra, tempo and musical nuance are controlled in real time.
“Renaissance music is so much more complicated for singers than any other music,” says Delorey, who leads eight choral groups at WPI. “It was before instruments were really brought into the fold, so singers were taught at another level that we don’t understand today. Everybody had to be a virtuoso.”
Though the technology is in its early stages, Delorey has begun imagining its future possibilities, both to take complex performances on the road and to use as a learning tool. “You could imagine a high school teacher using it,” he says. “It can be used to approach music you couldn’t approach otherwise.”
The virtual choir is just one of Delorey’s technological pursuits. For the past few years he has been working with students on a digital choral folder that will allow music to be electronically stored and displayed. The reader will have two screens to display sheet music the way singers are used to seeing it. Several teams of undergraduates and graduate students have developed a series of prototypes that use the same type of electronic paper technology found in the Kindle and Nook e-readers. They envision a system in which multiple folders will be stored in racks that will connect them to a digital library of sheet music so they can quickly update all of the folders simultaneously.
“Sheet music can be hard to keep track of, because you can have so much of it, and using all that paper can be so wasteful,” Delorey says. “We wanted to come up with a device that was simple, so that it would be affordable.”
Though the digital reader is still being fine-tuned, Delorey and his students signed on to reduce paper at the 2012 American Choral Directors Association conference. Rather than printing out sheet music for the conference’s many performances, Delorey’s team projected the scores on screens and allowed attendees to save them as PDFs on USB keys. This allowed attendees to walk away with as many as 16,000 compositions, instead of the usual 80 to 100 handed out at the conference.
Preserving a Vanishing Heritage
Turning masses of paper into digital records is also part of what motivates Rich Falco, director of jazz studies at WPI, in his foray into technology. That paper — newspaper clippings, photographs, concert posters and programs, and manuscripts — are the documents and memories that represent the history of jazz in New England. For the past 10 years, Falco and teams of dedicated students have been working to preserve that historical record — which also includes audio recordings and radio and television programs — by digitizing it and uploading into the online Jazz History Database (jazzhistorydatabase.com).
A virtual museum, the database includes material dating back to the 1920s. Falco’s goals are to preserve the legacies of individual musicians — including many from the Worcester area — and to show younger jazz musicians and fans how jazz evolved as styles were passed on from musician to musician. He also aims to pay tribute to the unsung heroes of jazz.
“We’re not looking at first-tier musicians; that material has already been preserved,” he says. “I’m looking for the second-tier musicians — they’re deserving of recognition, too. They’ve contributed so much to this city, this region.”
Falco began collecting material for the database 10 years ago. At the time, he says, “the technology wasn’t there to do what I was imagining.” Now he leads two seminars a year dedicated to the database. Students gather and digitize old audio recordings, television programs, newspaper articles, and photos. The work can be tedious; processing a half-hour television show, for instance, can take up to 17 hours of work.
Students are rewarded for the hours of tedium by having the chance to interview musicians and their families to capture their stories firsthand, thus preserving their legacies forever. Some of the musicians have died since their histories were captured; others were convinced that they had been forgotten.
“[The seminar] is a marriage of music and technology in the best possible sense,” Falco says.
Falco’s seminars are consistently filled, and some students continue working on the database as volunteers after the term is over. The database itself is evolving, and now includes sections for jazz poetry and paintings and photography by and about jazz musicians. Falco hopes soon to make it easier for visitors to contribute to the site by uploading material.
Longer term, he says he’d like to establish “satellites” in other cities in the United States and abroad with strong jazz traditions. The processing of materials would be done at WPI. “We’re uniquely positioned, with our technology education and our students’ strong interest in music, to play an important role in this preservation process,” he says.
“No one else I know is documenting those traditions now, and much of that material is deteriorating so quickly,” he adds. “And when it’s gone, it’s gone forever.”
This is Your Brain on Jazz
Musical improvisation is a uniquely creative process. A jazz musician riffing on a melody seems lost in the music as if transported to another state of consciousness. But what really happens in the brain during a jazz performance and what can that knowledge teach us about the creative process? Richard Falco and Frederick Bianchi have set out to answer those questions through an unusual collaboration with a brain researcher, Karl Helmer at Harvard Medical School. Using advanced brain imaging research techniques, they hope to take a peek into the brain in action to see if there is a characteristic signature associated with improvisation. The researchers will work with jazz musicians who will be scanned by a functional MRI unit while playing and improvising music. Falco and Bianchi say they are excited about the prospect of unlocking some of the mysteries associated with human creativity, and plan to put what they learn to use in their own work as musicians, researchers, and educators.