Remote Music Without the Latency Race

Almost every discussion of playing music over the Internet starts from the same premise: the problem is latency, therefore the job is to reduce latency.

It is a reasonable premise. It has produced good engineering. It also quietly assumes something worth examining, which is that every performer must hear every other performer essentially immediately.

Suppose that assumption is negotiable. What if synchronization does not require immediate mutual audibility at all?

That question leads to a second architecture, with a different set of tradeoffs. It is the one Lyrekos is built on. This article explains how it works, gives the delay numbers we measured across the Pacific, and is specific about where the approach stops.

Why distance creates a floor that software cannot break

Start with the part nobody can engineer around.

Light in optical fiber travels at roughly 200,000 kilometers per second, about two-thirds of its speed in vacuum. Los Angeles to Sydney is about 12,000 kilometers as the crow flies, which puts the theoretical one-way propagation delay near 60 milliseconds and the round trip near 120.

But packets do not fly like crows. They follow cable routes, they hop through switches and routers, and each hop costs something. Real paths run well above the theoretical floor.

Set that against what ensemble musicians can actually tolerate, which is roughly 20 to 30 milliseconds of mutual delay, and tighter when instruments are involved. Sound moves about a foot per millisecond, so that budget is roughly the width of a choir loft. It is the reason musicians standing on risers never think about physics and musicians on separate continents cannot avoid it.

The gap between those two numbers is the whole subject.

What the delays actually are

Abstract discussion of latency is less useful than measurements, so here are ours, taken during our Australia to Los Angeles test. Our cloud server was in Oregon.

It is worth breaking the delay into three parts, because they behave differently:

• The network. Every possible solution must deal with this one. It is the part that is closest to physics.
• The browser. This is partly our own choice. We decided to do everything in a browser rather than ask people to download and install an application, and that decision costs many milliseconds. A native app with direct driver access has a shorter audio path.
• The jitter buffers. These absorb variation in delay, and there are three of them per round trip. They are necessary, and they are a quality-versus-latency tradeoff. Remove them and audio breaks up.

The table below reports the first two. The jitter buffers can more than double these numbers, so treat everything here as a lower bound. For reference, when Apple AirPlay works inside your home the jitter buffer is 15 seconds deep — seconds, not milliseconds — and in my house even that fails when the microwave oven is running.

Two things in that table are worth sitting with.

First, the transpacific number is what you would expect. Half a second, against a budget of twenty to thirty milliseconds, is not a near miss. It is off by more than an order of magnitude.

Second, and more interesting: the short hop fails too. Sherman Oaks to Venice is a drive across Los Angeles. Routed through a server in Oregon, in a browser, it still comes to about a quarter of a second. Distance was never the only problem.

There are well-understood ways to attack these numbers. If everyone were in Los Angeles, you would abandon the browser for a specially configured downloadable application. You would move the server into the city or eliminate the server entirely and connect people directly. You would ban Wi-Fi and require Ethernet. You would ask everyone to buy an audio interface. Each of those buys back real milliseconds.

We wanted Lyrekos to work worldwide, and we wanted it to work for groups with no IT support. Those two goals are incompatible with winning the latency race, so we stopped running it.

Approach one: make the latency small

The established answer is to attack every term in that budget.

Systems such as JackTrip, Jamulus and FarPlay do this seriously and well. Minimal buffering. Light compression or none at all, trading bandwidth for delay. Wired connections. Dedicated audio interfaces with short driver paths. Careful attention to the operating system’s audio stack, which is often the largest single term nobody notices.

Under the right conditions the result is genuinely remarkable: musicians in different buildings playing together with mutual reaction intact, improvising, following each other. That is a real capability and it is worth having.

The conditions are the point, and stating them is not a criticism. This approach optimizes for interactive mutual reaction, and it purchases that with distance limits, network quality requirements, equipment requirements and per-participant configuration. Those are not defects. They are the price of the thing it is buying.

If you want four jazz musicians in the same metropolitan area to improvise together, this is the correct architecture and nothing else is close.

One way is cheap. Round trips are expensive.

Here is the observation the second approach is built on, and it is more general than it first appears.

Latency only becomes a musical problem when information must go out and come back inside a beat. That is a round trip, and the twenty to thirty millisecond budget applies to it.

Information that only travels one way has no such budget. It can be delayed by a second, or fifteen seconds, and nothing musical breaks, as long as the receiving end knows how to place it in time.

That distinction sorts the problem cleanly:

• Round trips: two singers listening to each other and adjusting. A player reacting to an unannounced key change. Anything where A affects B and B affects A within the same passage.
• One way: a backing track reaching a singer. A previous take reaching a singer. A conductor’s gestures reaching the ensemble. A finished performance reaching an audience.

Everything in the second list can be arbitrarily delayed and then, with careful engineering, aligned. Everything in the first list cannot.

The architectural question is not “how do we make the network fast enough for round trips?” It is “how much of a rehearsal actually needs a round trip?”

The answer, for most ensemble music, turns out to be surprisingly little.

Approach two: stop requiring the round trip

The conventional model is a loop:

Performer A → Internet → Performer B → reaction → Internet → Performer A

Everything in that loop must complete inside the ensemble timing budget, and it crosses the network twice. That is what makes distance so expensive: the cost is incurred twice per interaction, continuously, for every pair of musicians.

The alternative breaks the loop into one-way deliveries:

Shared reference → A
Shared reference → B
Shared reference → C
then: A + B + C → aligned to the reference → synchronized ensemble

Each performer’s timing is defined by their relationship to a common reference rather than by their relationship to each other. Delay from the reference to each performer still exists, but it no longer must be small, and it no longer has to be equal, because it is measured and can be accounted for when the performances are aligned.

This is the inversion. The system stops trying to make the network fast and starts making the network’s speed irrelevant to synchronization.

What can serve as the shared reference?

Several things, and the choice shapes the musical experience more than any other decision.

• A backing track or accompaniment
• A rehearsal piano or click track
• A previous pass by the ensemble itself
• A strong singer or a lead section
• A live leader whose part everyone else follows

The third one matters more than it looks, and it is worth showing rather than describing.

For our Los Angeles to Australia test of Amazing Grace, the first take used a prerecorded piano track. Everyone heard the piano and sang their part against it. They did not hear each other, and the singers did not particularly enjoy the experience. The moment the take ended we played the combined result back to the whole group, so the feedback was near instant, but the singers said what you would expect them to say: it was fine, and they missed hearing each other.

So we ran a second take using the first take as the backing track. Now every singer heard the full ensemble while singing. It is a bit of a trick, since what they were hearing was the previous pass rather than the current one. It worked anyway, and it worked better than we expected. The director who sang in that session described it without being prompted:

It emulated an actual choir experience where you listen to certain voices doing certain things and you match them. The alignment with all our consonants came together in that second take because everyone heard what I was doing.

Alex Siegers, choir director

That is the part most descriptions of shared-reference synchronization leave out, and it is not a refinement. People do not perform well against a metronome. They perform well against other people. A reference built from the ensemble itself gets you a musical result rather than a merely synchronized one.

There is no reason to stop at two. The second take can become the reference for a third, and so on.

Why every input is recorded on its own track

One consequence of this architecture is easy to miss and turns out to matter a great deal in practice.

Because performances are aligned to a common reference rather than mixed live, each input can be kept as a separate track. Nothing is baked together at capture time.

That gives you a set of capabilities that are awkward or impossible in a live-mix system. A director can listen to any single singer alone, immediately or a week later. Balance can be changed after the fact. And elements of the reference can be removed from the final result, or even from an intermediate one.

The Amazing Grace recording is an example. Every singer heard piano while performing the first take. But because the reference for the second take was the first take with each track recorded separately, we could choose whether the singers heard the piano in the second take at all. The piano exists on its own track, so we suppressed it in the final result, and what you hear is a cappella.

The same mechanism means the reference could be a recording of a famous choir or soloist that an ensemble wants to sing with and then remove, leaving their own performance behind, which they can then iterate on to make increasingly their own.

What this approach does not do

It does not repeal relativity, and being explicit about that is more useful than any feature list.

If a singer in California produces an unplanned ritardando, a singer in Australia cannot follow it inside that take. The information has not arrived. No architecture changes that.

But notice how narrow that statement actually is, because the usual shorthand overstates it badly.

A conductor can conduct. We run a mode where singers see and hear the director, including hand motions and lip movements, and follow her. She reaches them a few seconds late, and it does not matter, because direction is one way. What she cannot do is respond to a singer inside the same take, since that would be a round trip. She hears the result immediately afterward instead.

Improvisation is likewise not excluded as such. A great deal of playing together is improvising over an agreed frame, whether that frame is a progression, a groove or a track. A frame is a shared reference. Jazz musicians soloing over the same backing track are doing something this architecture accommodates naturally, and it is a direction we expect to take. What they cannot do is one of them deciding mid-chorus to double the time and the other following.

The boundary is not “composed music yes, improvised music no.” It is “one-way yes, unscripted round trip no.”

What happens when musicians are 8,000 miles apart?

It keeps working, which is the interesting result rather than the impressive one.

We ran four singers across Los Angeles, Canberra and Melbourne, roughly eight thousand miles of separation, on ordinary consumer connections over Wi-Fi, in browsers, with nothing installed. We have also run it over Starlink. They performed as an ensemble and the result is in sync.

Look back at the table. Under approach one, that session is not a hard case, it is an impossible one. Roughly half a second of round-trip delay before jitter buffers, against a budget of twenty to thirty milliseconds, is not a gap that better code closes.

A working four-part ensemble at that distance is not a demonstration of unusually good latency minimization. It is a demonstration that the architecture does not depend on the thing everyone else is optimizing.

That is why the distance number matters. Not because customers need eight thousand miles, but because it is the experiment that separates the two approaches. You can hear the result and read how the session ran.

Can remote musicians stay synchronized when their Internet delays differ?

Under the shared-reference approach, yes, and this is where the architectural difference has its most practical consequence.

Under approach one, unequal delays are corrosive. If A hears B 30 milliseconds late and B hears A 70 milliseconds late, they cannot converge on a common beat, and the asymmetry produces exactly the mutual dragging that makes videoconference singing collapse. Worse, the delays wander during a session as networks reroute and buffers refill, so even a group that settles does not stay settled.

When every performer is timed against a common reference instead, unequal delay stops being a synchronization problem. Each performer’s offset from the reference is a measured quantity, and measured quantities can be corrected for. What was a fatal asymmetry becomes bookkeeping.

Participants notice this working. From one of our cross-continental sessions:

The playback sync worked as well. It adjusted the timing correctly, which was great. That was very in sync.

Alex Siegers, choir director

Why ordinary hardware becomes sufficient

This is the part that turns architecture into practice, and it is the reason the distinction is worth caring about if you are not an engineer.

When success depends on shaving every recoverable millisecond, every millisecond is worth money and effort. That is why the low-latency world converged on wired Ethernet, external audio interfaces, native applications with direct driver access, and participants who can debug their own audio stack.

Look at the table one more time. Our browsers cost 122 milliseconds of round trip inside Los Angeles. In a low-latency system that number would be a catastrophe worth eliminating at almost any cost to usability. In ours it is a design choice we can afford, because the singers are not waiting on a round trip.

The entire requirements chain relaxes with it:

• Wi-Fi becomes acceptable, because jitter is absorbed rather than fatal.
• Ethernet stops being mandatory.
• Consumer devices are adequate. No audio interface is required.
• The browser becomes a viable platform, because its extra milliseconds no longer matter.
• Phones work.

That last set is not a footnote. It is the whole commercial difference. A tool requiring per-participant installation and configuration is usable by ensembles of technically confident musicians and unusable by a volunteer choir, regardless of how good the audio is, because the organizer becomes the support desk for forty people.

Concede the latency. Win on accessibility. That is the trade, made deliberately.

Which approach should you choose?

Ask what has to make a round trip.

If musicians must react to each other’s unscripted choices in the moment, if the structure of the music is being invented while it is played, you need low latency and you need to stay geographically close. Use approach one or meet in person. It is mature and it works.

If the ensemble is working from known material against something that could serve as a reference, and the binding constraint is that people are in different places and cannot easily install anything, approach two removes distance from the problem entirely. That is the case we walk through in how to rehearse a choir online.

The mistake is treating these as competing answers to one question. They are answers to two different questions that happen to share a vocabulary.

For most of the remote music world, the question everyone asks is how to minimize latency. If you must be remote, we think a better question is: how do you stop fighting physics and maximize the effectiveness and enjoyment of the ensemble experience? That is Lyrekos.