How Much Latency Can Live Musicians Tolerate?

The quick answer: about 20 milliseconds.

Given that we are lucky enough to have two co-founders — Ryan (and Fred — that have also trained as professional musicians, we wanted to test for ourselves the amount of delay that could be tolerated by two performers that were trying to ‘groove’ together when separated by Internet-induced delays. This is an important number, because it is easy to imagine how amazing it could be if you could create a virtual world which enabled multiple band members to perform live together when each was not physically on the same stage or in the same studio.

Testing 1, 2, 3 

Attempts and experiments at multi-person live performances have already happened in Second Life and other virtual worlds, but the large and variable delays have generally forced the performers to adopt a serialized strategy where, for example, a bass player might lay down a basic groove which is then streamed to the next musician who adds her own line and then streams that to the next one, etc.

Although this does result in a live performance, the earlier players in the stream are ‘blind’ to the later ones, so they are not truly playing together and responding with subtlety to each other. But the demanding example of musicians playing together and continuously adjusting millisecond-to-millisecond as they watch/hear each other is what we wanted to explore… how much latency could we add before we ‘broke’ their ability to play together, or at least made it discernibly less fun?

To test this, we set up our two experts as performers, separated visually and by enough of a distance that they could not ‘feel’ the floor moving with the music, and wearing high quality headphones connected by a device which allowed us to impose an exact amount of latency (single millisecond resolution) between them. An experimenter would then dial up various amounts of delay, allow them to play for a while, and then record their verbal assessments of how good or bad the experience was for them. By changing roles and trying various instruments, we were able to learn that much below about 10 milliseconds of one-way delay, they typically could not discern the impairment, but that by about 20 milliseconds they would uniformly report the experience as highly undesirable.

Looking Ahead 

This outcome is a challenging one to the dream of having multi-performer live Internet jam sessions: to get one way latency down to 10 milliseconds would require very high packet rates — say 250 packets-per-second or more, along with the performers being separated by only modest real-world distances. At the speed of light, 5 milliseconds of delay corresponds to a physical distance of about 1,500 kilometers. Also, Internet routers today still impose a delay that can easily be greater than this, for example the one-way delay from San Francisco to our EC2 machines in San Jose is about 7 milliseconds. Ultimately we will get there, but for now we can expect that our virtual live performers may still need to be in the same room together. But, of course, this will still make for some pretty amazing potential for live performers to interact with their audience while performing in a virtual world — more on this to come.