Video Transcript
Q-SYS Networking and Topologies (Part 6)
7m 12s
00:07
Once all the other issues on the network are taken care of, we’re often left
00:11
with the third category of problem…network performance issues.
00:16
These are usually indicated by streaming faults and PTP clocking faults
00:20
in the Q-SYS eventlogs and device status blocks.
00:23
Remember the network requirements
00:24
for Q-LAN are very tight …280us end to end latency with 30us of allowable jitter.
00:31
We need audio packets to travel as quickly and consistently as possible.
00:36
When thinking about solving these problems, remember
00:39
that if the devices are not locked to the same PTP grandmaster,
00:42
you cannot hope to have consistent streaming.
00:45
Always concentrate on this problem first.
00:48
The important information for troubleshooting PTP problems is in the Core and peripheral status blocks.
00:54
The first thing of note is exactly what device is the PTP grandmaster.
00:58
If a Q-SYS device is named in the grandmaster field, then it’s easy to know which device that is.
01:04
If you see a MAC address instead, that would indicate that a 3rd party device is the grandmaster.
01:09
You could use the OUI of that device to find out exactly which device that is.
01:14
A good example would be the use of a 110f in conjunction with a SHURE mxa 910.
01:20
The Shure device uses a lower number for PTP priority 1, so it is elected the grandmaster by default.
01:26
You would see that device’s MAC in the grandmaster field for both the Core and peripheral in this case.
01:32
In general you want to first observe that the device serving as grandmaster makes sense.
01:38
Red PTP clocking errors will show up in the status bar at the top of the blocks.
01:43
Observe the Clock offset around the time these errors occur.
01:47
If you see sudden changes in this offset, especially if it’s going from a positive to a negative number abruptly,
01:53
this indicates jitter on the network.
01:55
Jitter is the enemy of PTP, so you’ll want to concentrate
01:58
on getting clock packets delivered consistently on the network.
02:01
This often means the proper QoS policies haven’t been put in place from end to end on the network.
02:08
When troubleshooting Q-LAN packet delivery problems,
02:11
it’s very useful to enable the verbose statistics in the device status blocks.
02:16
We’d often recommend this be done first thing, as it requires redeployment of the design to enable.
02:22
If you’re having problems that crop up only after some time,
02:25
you may have to wait a long time for the problem to resurface.
02:28
A Q-LAN stream consists of 3000 packets per second per stream, so it’s a lot to keep track of.
02:35
The verbose statistics can tell you exactly how many did or did not arrive,
02:40
whether they were completely missing or late.
02:42
Of course late packets would indicate excess latency on the network.
02:46
Missing packets indicate that packets are being dropped by the network.
02:50
You can also get a good idea here of how bad the problem is:
02:53
If you have one late packet per million, that’s something that users aren’t even likely to hear.
02:58
I you have high numbers of late packets that are rapidly counting up, it’s an issue you want to deal with quickly.
03:04
You might recall some of the scenarios that can get us into trouble from the previous trainings.
03:09
We often have isolated room systems connected to the corporate network for Q-SYS softphone registration,
03:14
which can cause all the individual room systems to try and sync to each other over a non-compliant network.
03:20
As long as these systems don’t need to share Q-LAN audio,
03:24
the solution here is to simply disable PTP on the LANB network ports.
03:29
This is done in the network services tab of Q-SYS Core Manager.
03:33
If the systems in question must share Q-LAN or AES67 audio,
03:38
they must be synchronized to a single PTP grandmaster, meaning they must also be on the same PTP domain.
03:45
It’s still valid to manipulate the PTP priority of one system over the other to get the right results.
03:51
For example, if one system is mission critical such as the bowl of a stadium seating area,
03:57
it makes sense to prioritize that Core as the PTP grandmaster.
04:01
In other cases, the grandmaster might be on the very edge of the network making it hard to sync to all the other edges.
04:07
Prioritizing a more centralized device to be grandmaster minimizes how far clock packets must travel,
04:14
so that can often be a big help.
04:16
Once all the basics are covered and there are still PTP problems,
04:20
then the switch configuration will need to be examined.
04:23
You’ll want to make sure the recommendations are being followed:
04:26
Make sure a strict priority queueing model is being used.
04:30
Make sure you know which DSCP value clock packets are using and that value is assigned to the
04:36
highest-priority queue. Remember sometimes that will be two values…56 for PTPv1 and 46 for PTPv2.
04:45
Make sure you know which DSCP value audio packets are using
04:48
and make sure that’s assigned the second highest priority queue.
04:51
Keep in mind these settings must be consistent on every switch in the network to know that traffic
04:57
will be prioritized properly.
04:58
This must be investigated for not just PTP clocking problems,
05:02
but for missing and late audio packet reports, too.
05:05
Here’s an illustration of these points in the setup for a commonly used switch: The DLink DGS1210-10P.
05:12
At the top left, we can see the switch is in DSCP QoS mode and using a strict priority queueing model.
05:19
This is exactly what we’d like. We have to look a little more closely to examine the queue assignments.
05:24
In most network switches the highest number queue represents the highest priority.
05:29
In this switch that’s queue 7. We see here that DSCP value 46 is assigned to that queue.
05:36
That would correspond to the PTPv2 DCSP marking in the Q-LAN QoS Preset.
05:41
We know the DSCP value associated with audio in that case is 34.
05:46
We see here that 34 is assigned to queue 6 as it should be. That's the second-highest queue.
05:51
Finally, DSCP value 26, which is Q-SYS video, is placed in queue 5.
05:56
Note that all other DSCP values are associated with priority zero, so they are not prioritized at all.
06:03
This is what we would expect in this situation
06:06
Remember these settings would be valid for the QLAN QoS preset in the Q-SYS design.
06:11
In the Audinate QoS Preset, we’d see 56 for queue 7, 46 for queue 6 and 26 for queue 5.
06:19
We also discussed the case with a mixture of new and old firmware Audinate devices where we have to account
06:25
for DSCP values 56, 46, 34 and 26 to make sure all traffic is properly prioritized.
06:31
This is what we’d expect in that case.
06:34
Going back to our troubleshooting mindset, let’s say you’ve looked at all your Q-SYS
06:40
and network configuration and it all looks ok.
06:43
You still have some late packets reported in your Q-SYS devices that you just can’t remedy.
06:48
There’s ONE last thing you can try.
06:50
Extending the Q-SYS network receive buffer in the Q-SYS Core properties.
06:55
This increases the overall audio latency by the amount your extend the buffer,
07:00
but it does make the Q-SYS network more tolerant to end to end latency.
07:05
This will not help, however, if your primary problem is network jitter.
Downloads and Links
Q-SYS Networking and Topologies (Part 6)
7m 12s