MacInTouch Amazon link...
Channels
Other
Echo problems on a voice call (land line or wireless, but not over IP) probably indicate a problem with the carrier's network being unable to enforce the delay and jitter QoS needed for the call. If it persists, consider calling customer service.

I'm not surprised that LTE has worse latency and jitter than FiOS. As a wireless technology, it is not going to pre-reserve resources, since that would take them from other customers. but the packet prioritization rules used to enforce QoS should have been sufficient. Also note that an IP traffic report is not going to tell you anything about latencies for voice calls - those packets should be tagged as voice for different handling by the network.

I would expect Wi-Fi calling to be worse than either of the above. In the best case (where your router is connected to a network with Verizon service), the packets go to a Verizon IP router before they get send to the voice network. In the worst case (traveling to a location where the router is on someone else's network), then those packets will have to cross the Internet (where QoS is generally not enforced) on their way to the voice network.

The echoes you encounter are (generally) feedback - sound played on your speaker is picked up by your microphone. Your calling/conferencing software is designed to look for this and remove the echoed sound, but there's a limit to this because it needs to maintain a buffer of audio data in order to make it work. That buffer is a fixed size and not for a very long time (probably not much more than 200 ms or so). Any echoing with a longer duration between the transmit and receive won't be canceled.

The reason good headphones help is because an insulated earpiece and a directional microphone can reduce the level of audio in that feedback loop to the point where it can be treated as background noise by the software and therefore ignored on that basis.

This is also why conferencing using a computer's built-in microphone and speaker almost never works well. Despite all the promises of hardware manufacturers, their echo/noise cancellation software is almost never good enough to deal with the latencies of a voice call except under ideal conditions – which is why I tell people without headsets to at least wear a pair of earbuds while in calls – even if you continue to use the computer's microphone, using the earbuds to listen is usually sufficient to break that feedback loop. Still far too much delay to have an argument over the call, but at least the echoes go away.
 


Ric Ford

MacInTouch
Also note that an IP traffic report is not going to tell you anything about latencies for voice calls - those packets should be tagged as voice for different handling by the network.
Thank you for pointing that out – I'd completely missed that! I wonder if there's any easy way to test performance with "voice" packets vs. the usual "data" packets? And this could be a big factor in remote music instruction.
The echoes you encounter are (generally) feedback - sound played on your speaker is picked up by your microphone.
In this case (Jabra Elite 85h), the headphones are over-the-ear with an array of microphones and internal processing. They're rated for very good mic performance (for voice calls), but apparently there's enough leakage with the volume turned up that the mics pick up some leakage and that turns into echo on the other end of the other person's voice. Turning down the volume on the headphones apparently resolves the problem for the other person.

I'm also getting some very slight echo on my end of the other person's voice, which needs more testing. In addition to the complexities of the "phone" connections - carriers, WiFi Calling options, different phones, different headsets, etc. - the Jabra headphones have different modes, including noise-cancellation on/off/"hear through", plus the ability to connect to two Bluetooth devices (so Bluetooth is part of the very complex calling system, as well).
 


It's worth noting that professional musicians have been having remote recording sessions for decades using ISDN, which is (by today's standards) a pretty low bandwidth technology (between 128 Kbps and 1.5 Mbps, depending on the number of B-Channels used for the connection). This is because ISDN calls are point-to-point (circuit-switched) connections between two ISDN modems and the network guarantees a fixed bit-rate with low latency and jitter over these connections.
Internet standards provide for priority codes for time-sensitive signals, such as voice and ISDNso the transmission would simulate circuit-switched connections by guaranteeing a fixed bit rate for transmission.

I was dismayed to find that the Electronic Frontier Foundation did not understand how essential those priority codes are for quality of service for voice and audio; back during the debate over net neutrality they adamantly insisted on no priority codes for any type of traffic. The actual net neutrality plan that was adopted (and later revoked) retained those priority codes. I wonder if they now are being ignored.

The current FCC seems to lack a deep understanding of telecommunications technology – for example, in the way they are handing spectrum needed for GPS over to the mobile phone industry.
 


Ric Ford

MacInTouch
... Latency was 3-4 times worse on Verizon LTE vs Verizon FiOS. Jitter was more than 10 times worse on LTE....
Some more Internet performance testing for what it's worth, using the FCC Test app on an iPhone X:

Verizon
FiOS​
Granite Digital
fiber​
Verizon 4G
(4 bars)​
Verizon 4G
(1 bar)​
Verizon 3G*
(1 bar)
Upload (Mbps)​
26.6​
25​
7.3​
0.045
test fail
Download (Mbps)​
27.9​
26​
11.5​
0.26
test fail
Latency (ms)​
17​
70
65​
366
test fail
Jitter (ms)​
2​
0.8​
13
5​
test fail
Packet loss (%)​
0​
0​
0​
0​
test fail
*unusably weak signal

I got a good-quality audio phone conversation via Verizon WiFi Calling on the Granite State fiber service to someone on a regular phone (connected via a different fiber network bundle package) with no echo (not using any headphones, just an iPhone X) in a location with very poor cellular service.

(I haven't tried conferencing or video but am very interested in learning more about remote musical collaboration platforms/issues.)
 


... First, a question: Are all these FaceTime and Zoom sessions with video, or are some audio-only (especially the banjo example)?
i suspect the banjo success comes from high-speed connections between your two houses rather than any other component. But I'd be curious to know if you tried any different software, settings, or hardware along the way, or if you've found any other variables affecting results for you and your students. …
These are all regular sessions using both audio and video together. Since I was concentrating on the audio, listening to us playing, I didn't pay much attention to the video on the FaceTime call that worked well. I'll try it again this week.

My Zoom interactions with other musicians, even ones who are a mile or less from my home, have significant delay and other audio artifacts that prohibit playing together. One of the things we've tried is to have two musicians in a Zoom meeting play together while everyone else is muted, so that they can hear how well it works between the two people playing together.

It usually sounds like the two different sources are competing with each other to be heard. I'll hear a string of melody notes from one player and then perhaps a chord or two from the other player, which partially obscures the melody notes. And then maybe the melody can be heard better for a few seconds, and so forth, plus the time delay. And that's what the observers hear. It's usually worse for the two people playing together.

Everything always sounds better with Original Sound enabled and disabled Background Noise Suppression. In one of my weekly musician groups, one of the participants uses an Android phone. We haven't figured out how to adjust those settings on his phone. Perhaps it's not possible on an Android phone?

In that same group, when the fellow with the Android phone is unmuted and listening to others play, loud echoing is heard by everyone else in the group. So he has to stay muted, unless he is playing and we are listening to him. But because he can't adjust his audio settings, it sounds somewhat garbled when he plays. Some notes come through, some don't.
 


Internet standards provide for priority codes for time-sensitive signals, such as voice and ISDNso the transmission would simulate circuit-switched connections by guaranteeing a fixed bit rate for transmission.
Yes. When I say "circuit switched" connections, these days it is always in the form of a virtual circuit, where packet switched data is queued and prioritized in order to provide the same characteristics as a circuit-switched connection. Even if you lease a high bandwidth line like a T3/DS3 or an OC-3, the physical line you see at your site terminates at a router or switch (in the telco's network) that forwards all your traffic into/out of a packet-switched virtual circuit.

In the (not-too-distant) past, telcos architected their core (voice and leased circuits) around technologies that were designed for virtual circuit switching (e.g. ATM and Frame Relay). Today, most (but not all) of it has been migrated to "layer 2" technologies like Ethernet and PoS (Packet over SONET/SDH), often in conjunction with higher-layer technologies like IP and MPLS – all using a variety of QoS standards that have been defined for those technologies in order to provide the bandwidth/latency/jitter guarantees required for their "circuit switched" service offerings.

But even when the core network is based on IP, they don't try to provision virtual circuits over the public Internet - there's simply no way to guarantee anything over there. Instead, the telcos run a separate network for virtual circuits, in order to retain the kind of control necessary to provide the required bandwidth/latency/jitter guarantees. Internet traffic may pass through this network, but only via an "overlay" mesh of virtual circuits that connect the Internet routers that are on the edge of the core network - the Internet traffic remains blissfully unaware of the existence of the core network that implements those virtual circuits.
 


This may be off-kilter from the real reason why banjo is working, but my first reaction as an audio/mastering engineer is that the banjo is, ahem, very strong in the midrange frequencies that are typically used to detect the human voice (~400–1 kHz), and this may be triggering the DSP at Zoom et al to prioritize the audio as voice (as opposed to background noise or complex music, which web meeting systems often seem to mangle). Here is a typical frequency profile for a snippet of Bruce Molsky's fine banjo picking, for instance.
 


Everything always sounds better with Original Sound enabled and disabled Background Noise Suppression. In one of my weekly musician groups, one of the participants uses an Android phone. We haven't figured out how to adjust those settings on his phone. Perhaps it's not possible on an Android phone?
This has been my experience with the Zoom client on Linux, which is still version 3.xxx. It makes sense that Android will also have an earlier version, as it is based on Linux. I think you need version 4.something to use 'Original Sound.'

I have been playing music once a week over Zoom for the last six weeks. The biggest improvement for me was booting my MacBook [from] an SD card with Sierra installed, and having an up-to-date version of Zoom, where I could access the advanced audio settings and use 'Original Sound.'
 



Interesting and helpful discussion, thanks to all.

FWIW, in these past weeks I have been doing testing with other colleagues in Australia who work for universities and now need to do music teaching (of various formats and persuasions) online, and over reasonably large distances. Have worked locally in Brisbane, then to Sydney, Canberra and Melbourne.

A couple of things – we all have been working out of 'home recording studios' and so a few things to note here, further below. We also are not especially wired into macOS, and the workstations vary across Windows and Mac. Bottom line with the conferencing software – we tested most, including FaceTime (for Mac users), Cisco, Google, Sykpe, BlueJeans, and Zoom – no question, by a long shot, Zoom outperformed everything else.

To sum that up in the most extreme case, one colleague does drum teaching online, and you might imagine just how important latency is here, e.g., the visual of the snare hit should be 'in time' with the audio of the snare hit. Everything else was hopeless, and Zoom was by far the fastest - nothwithstanding other ISP provider issues that may come up from time to time.

Here, we checked before the session and all of us had ~47 Mbps down and 17 Mbps up from local telco Telstra. Overall, Zoom had noticeably better latency, sound quality and video quality. Apple's FaceTime was the worst overall.

The other hardware tweaks: high quality microphone, lav or wireless lav to studio mic pre-amp, then in some cases with a little real-time compression, EQ etc. In the case of the drum teaching mentioned, this was also sharing and switching between three camera feeds and several mic sources mixed through that studio's audio interface (a MOTU in that case).

On the video side, aside from the drummer, most were using a single LogicTech webcam. mostly a variant on the 920 series, but the Logictech StreamCam was also a feature because of its 1080p 60fps capability - in general, quite a bit smoother because of the higher frame rates.

Otherwise - aside from the more uncontrollable external ISP bandwidth issues that can arise from time to time, inside the WiFI and LAN there is quite a lot that can be tweaked in terms of prioritising ports and IPs - also dependant on how much other competition there might be in the household: phones, TVs, other computers, etc.

Our Synology router, mesh wifi points and a NightHawk switch all allow that to be monitored and prioritised, in this case, for the primary video conferencing /studio workstation and our NAS. Port priortisation, of course, can also be tweaked on other routers as well, your milage may vary.

Hope that is of use.
 



FWIW, in these past weeks I have been doing testing with other colleagues in Australia who work for universities and now need to do music teaching (of various formats and persuasions) online, and over reasonably large distances. Have worked locally in Brisbane, then to Sydney, Canberra and Melbourne.
Wow. I'm surprised that any Internet-based solution was able to work over those distances.

The fact that Zoom worked best probably indicates that the server your calls are going through is relatively close to both parties and that the end-to-end connection through that server is low-latency (at least as low as you can hope to get over an Internet connection).
 


This fascinating discussion reminds me that in the March 2000 issue of Technology Review I predicted that fiber to the home would be carrying 100 to 200 megabits per second, making it possible for a family of four to all use lots of bandwidth simultaneously. I described a scene where the daughter "is practicing with a choir made up of people who live in a dozen cities in North and South America; a computer in Mexico City merges their voices and transmits the music back to their computers in real time, while creating an array of their faces on a single screen." It wasn't a bad prediction 20 years ago, but back then I didn't understand how latency could make that kind of scene impossible.
 


The fact that Zoom worked best probably indicates that the server your calls are going through is relatively close to both parties and that the end-to-end connection through that server is low-latency (at least as low as you can hope to get over an Internet connection).
There is a presumption there that Zoom does end-to-end encryption. I don't think it does. There's a pretty good chance that is a contributing factor as to why the latency is lower (and Apple's is the worse).

Encryption with low latency is technically possible, but it will probably be substantively more expensive. That these are all "free" (no charge) solutions means there is a tension of trade-offs that may persist for a while for multipoint sessions.
 


Ric Ford

MacInTouch
I received this tip via email* about a low-latency music collaboration system called "JamKazam":
I wanted to comment on the ‘latency’ issue people are writing about in your forums section concerning webconferencing for musicians. That’s where I thought I’d mention JamKazam (Mac, Windows, iOS etc.). It’s a funky name for an app, I know. Yes, my girl and I are musicians, and we think your forum friends might wish to check it out.

There are some good JamKazam YouTube videos where one can see that it really does work. Latency is reduced significantly enough to allow musicians to play in ‘real time’ over the internet. I do recommend ISP speeds of at least 200 Mbps.
JamKazam apparently requires an account (but may be used free of charge). Here's the JamKazam privacy policy. Here are its terms of service (with a physical address listed at the end).

Confusingly, the JamKazam home page touts a JamBlaster hardware device that's supposed to provide low latency, but its link is broken. Following up further, I found more information buried in a support page. The lack of professionalism here is slightly concerning.
JamBlaster Support said:
JamBlaster support is discontinued (per an email from Jamkazam support)
OK - I sent an email to support@jamkazam.com asking about pairing a JamBlaster, and whether the device was going to be supported.

I received a reply from peter@jamkazam.com (Peter Walker, according to the pop-up I get in Gmail) that simply states "It is no longer supported".

I replied, and asked them to post something official on this forum and the website,

* FWIW, I tried to reply to the email, but my reply was blocked by an obnoxious Earthlink "anti-spam" mechanism, a game I will not play on principle, since Earthlink has itself hypocritically been an unmitigated source of vast amounts of spam/scam abuse in the past....
 
Last edited:


Ric Ford

MacInTouch
Here's some worthwhile discussion about JamKazam (and JamBlaster), which goes in chronological order from old at the top to new at the bottom:
Gearslutz said:
Here's a recent article in Billboard:
Billboard said:
Band Practice at Home Alone, Together With JamKazam & Other Apps
After four years of investing their own money, raising $80,000 through Kickstarter and working nights and weekends, David Wilson and his two colleagues ran out of capital for their new venture: JamKazam, which allows musicians to perform together over the Internet. They drew 5,000 band sessions per month, but that wasn’t enough; Wilson, an Austin, Texas, tech-company product executive, dropped user support and stopped updating social media by 2017. All the founders returned to their day jobs.

Until March. When the coronavirus pandemic forced millions of musicians into their homes, JamKazam’s free monthly sessions jumped to 135,000. In the first week of April alone, that number grew by another 100,000. Wilson was shocked. “As soon as social-distancing started happening, it just started going nuts," he says. "We don't have the ability to provide end-user support. We have some help videos and it's like, 'Good luck.'"
Here's some more about the now-discontinued JamKazam with interesting discussion of latency:
Kickstarter said:
JamBlaster
... We've found that both cable broadband and fiber broadband Internet services work well, while low-end DSL service sometimes doesn't work well, and satellite/wireless Internet doesn't work due to high latency.

To play in groups of 4 musicians (audio only), you'll want to have Internet service rated at 1Mbps of uplink/upload bandwidth. To play in groups of 4 musicians (video + audio), you'll want to have Internet service rated at 2.5Mbps of uplink/upload bandwidth. ...

And finally, to play in online, real-time sessions, we recommend avoiding WiFi connections. For the best performance, connect the JamBlaster to your router using an Ethernet cable.
JamKazam said:
And here are some other things I found in a web search (note that the JamBlaster is "no longer supported", according to the quote posted above):
MakeUseOf said:
Systweak said:
 


* FWIW, I tried to reply to the email, but my reply was blocked by an obnoxious Earthlink "anti-spam" mechanism, a game I will not play on principle...
Ric, I found your email and have allowed access. (The Earthlink block is an option I've chosen. Has prevented lots of spam for me. And they have a great spam filter. I must admit that I've gotten <support@earthlink.net> spam lately. I blocked this address using their own filter. Problem solved. They're aware that there's a problem. Hope it's fixed. Today, I've been having problems getting online with Spectrum. Should be fixed now. So I'm sorry for the email miscommunication.)

Latency: Take a look at some of the videos at YouTube to see how JamKazam works.
 


Ahem... 1996...
Rocket Launchers
As the only dedicated jamming network, Res Rocket offers a free public-access area to amateur musicians worldwide....
This doc is partially incorrect and definitely incomplete.
History of the Rocketears
The Res Rocket Surfer Project. The first "virual online band" with 1,000 members that communicated through a mailing list and FTP server. Band founders were the band founders, Willy Henshall an award-winning songwriter, producer and member of the band Londonbeat, and Tim Bran, a successful engineer, producer, and member of the band Dread zone from London. They began posting messages and sound files on Usenet (the Internet's bulletin-board)-and later on an ftp and Web site-from their West London studio. People from all over the world started replying with song ideas and sound files.
Sorry but the above was working back in the '90's.

I know, because I was contracted to lead the development team based on my Mac developer experience. In fact, Canton and Matt flew to Toronto and interviewed me at the airport.

My wife and I moved down to Santa Fe and stayed there for 3 months. In that time I led the design and wrote the Mac client, Matt wrote the Windows client, Canton did the server / client protocol and coding, along with Jason, who also ran the Sun Server. Gregory and Liz did documentation and QA. It was a MOO, and coding was all C++ on Mac and Windows.

We returned home at the end of our contract, even though we were asked to stay on for a year – circumstances made that impossible. But the project was one of the most enjoyable of our contracting years. I still have the video tapes we made during the 3 months.

It went live and, at one point, we had online jamming with players from Europe, North America and all the way to Australia.

After we left, it moved to SFO and got funding from Paul Allen's VC Fund (Vulcan Capital).

FWIW

OrgName: Res Rocket Surfer, Inc​
OrgId: RRS-2​
Address: 699 8th Street, Suite 5262​
City: San Francisco​
StateProv: CA​
PostalCode: 94103​
Country: US​
RegDate: 1997-07-21​
Updated: 2011-09-24​
 




Ric Ford

MacInTouch
Out of curiosity, were these tests with a WiFi local connection at your end, or direct ethernet?
I tried both and couldn't see any differences between them, to my surprise.

I'm currently trying to resolve some FiOS problems (some outages, and only 1/3 the promised bandwidth) – I hope to do more testing by next week.

Verizon insisted that I needed to test using their own speed test:


but I got exactly the same upload/download speeds with their test that I've gotten with a variety of others.
 


I'm currently trying to resolve some FiOS problems (some outages, and only 1/3 the promised bandwidth) – I hope to do more testing by next week.
Verizon insisted that I needed to test using their own speed test:
but I got exactly the same upload/download speeds with their test that I've gotten with a variety of others.
Hmmm... I tried the Verizon test, which started out low but toward the end (at 99%) climbed to 75% and suddenly jumped to 83 MB.

Then I tried TestMyNet and got download speeds of 43, 58, and 48, and an upload rate of 44.

I have nominally 75 Mbps service. Nothing else was online, except my wife on the digital voice line.
 


Ric Ford

MacInTouch
Then I tried TestMyNet and got download speeds of 43, 58, and 48, and an upload rate of 44. I have nominally 75 Mbps service.
I supposedly have 75 Mbps, too, but I'm only getting 26/27 Mbps, never higher, even direct into the router via Gigabit Ethernet after a factory reset.
 



Ric Ford

MacInTouch
It's worth noting that professional musicians have been having remote recording sessions for decades using ISDN, which is (by today's standards) a pretty low bandwidth technology (between 128 Kbps and 1.5 Mbps...
Also note that an IP traffic report is not going to tell you anything about latencies for voice calls - those packets should be tagged as voice for different handling by the network.
So... what about using VoIP for music teaching and playing, instead of software like Zoom and FaceTime? Would VoIP and SIP phones with HD audio be a much better alternative, I wonder?

Here are a few things I found in a quick search for more information:
OnSIP said:
HD VoIP and HD Voice Codecs
The first standardized wideband codec, G.722, was developed in the late 1980s. It is now freely available because its patent has expired. Another HD voice codec, G.722.2, offers high-definition voice quality over smaller bandwidth transfer rates. However, the current “gold standard” of the wideband audio codecs is called Opus, and it's the default codec that's used by all WebRTC technology and apps currently available.
OnSIP said:
VoIP Test
  • We recommend 100 kbps per simultaneous call.
  • We recommend < 3 ms in jitter.
  • We recommend latency is <150 ms.
OnSIP said:
VoIP Fundamentals

HD Voice

Today's premier VoIP platforms are giving users voice quality that make phone calls seem more like in-person conversations. HD voice is a wideband audio technology that uses codecs of a greater frequency range on the audio spectrum than conventional telephone calls. The PSTN and a majority of VoIP codecs capture at 8 kHz, while a wideband VoIP implementation might capture at 16 kHz.

Our own HD voice operates on the G.722 wideband codec. G722 provides enhanced speech quality because of a wider speech bandwidth (50–7000 Hz) when compared to narrowband codecs like G.711 (optimized for landline use at 300–3400 Hz).
OnSIP said:
OnSIP Now Supports High Definition Voice
Phones from Polycom, Cisco, Linksys, SNOM, Aastra and countless others currently come with support for HD voice, meaning they support calls in HD using some flavor of the G.722 codec, the “wideband codec”.
...
BTW, we also support ultra-wideband voice, HD video... on any device that supports it. :-)
 


Ric Ford

MacInTouch
My IEEE Spectrum article on cellular voice quality is a few years old now, but it explains the problem in more detail. Sadly, the improvements promised then have yet to make their way through the phone system.
Here's a discussion about similar problems on the VoIP front:
VoIP Supply said:
HD Voice and Your Network
So you've got some HD voice-capable phones and you want to make the move from G.711 narrowband into wide world of wideband. How will that affect your network? How do HD voice phones "talk" to other HD voice phones in your organization, other organizations, and the PSTN/POTS? The three keywords here are bandwidth, transcoding, and interconnection.
 


Technically, VoLTE (and ViWiFi) is VoIP; however Mobile calls G.722.2 "AMR-WB" (because, why would 3GPP use an existing acronym?).

If you use Skype or WebEx (which are VoIP), you'll see that the codec is generally SILK or Opus. Opus is better than G.722 (it's close to CD quality) and is free. So, of course, most mobile carriers don't support it.
 


Here's a discussion about similar problems on the VoIP front:
Looking through the VoIP information and checking out specifications on VoIP phones makes me almost nostalgic for the old Bell System's stringent rules on phone lines half a century ago. The features look good, but I couldn't find any clear instructions on how to install the phones so I could get the features.

Today's voice network includes a lot of old, poorly maintained, and poorly installed equipment, and I suspect that's why voice quality varies so widely. All it takes is one malfunctioning connection to ruin a call. Yesterday I called a utility company and got a robotic voice system that crackled so badly I could barely understand it, and I started saying "human" "agent" "Operator" and such until I finally got through to a woman whose voice came though quite clearly.
 


Amazon disclaimer:
As an Amazon Associate I earn from qualifying purchases.

Latest posts