Q-SYS Networking and Topologies (Part 5)
Video Transcript
00:08
To begin that conversation of network troubleshooting,
00:11
this quote from Albert Einstein lays out a very important idea.
00:15
Network-related problems can seem very difficult and
00:19
sometimes it’s tempting to just start trying things and hope for the best.
00:22
It’s not usually a very fruitful, as if you don’t understand the root cause
00:26
you probably won’t be completely successful.
00:28
Before you act, you’ll want to look at the information you can gather
00:32
from various points of the system to truly define the problem.
00:36
Once you know that, you can devise the lowest-impact solution.
00:39
When troubleshooting these kinds of problems, we’re typically dealing with one of three common problems.
00:45
The first layer is that of basic connectivity…making sure we have the devices all connected the correct way,
00:51
they’re all configured correctly and they’re all running properly.
00:54
After all those challenges are overcome, the next layer tend to be related to how the devices are discovered,
01:01
how they interact with each other and how we interact with them.
01:04
Once all those issues are sorted out, the final layer involves
01:08
the finer detail of troubleshooting media streaming issues.
01:11
At this layer, we’re looking at network performance issues and proper prioritization of packets.
01:17
In troubleshooting basic connectivity problems, Q-SYS configurator is our first stop.
01:22
Does the device in question show up? If it doesn’t show up at all, then check the switch connections?
01:27
Does the device respond to touch of the front panel buttons?
01:31
If not, it may indicate a hardware problem.
01:33
If you know the device is running and you know its IP address,
01:37
you might try a simple PING to make sure you have point to point connectivity to that device.
01:42
If that is a success, try adding a hardlink to the device in Q-SYS designer.
01:46
If it suddenly appears, that could indicate that multicast traffic is being blocked on the network.
01:53
If it does show up, does it do so in green or red?
01:56
Red typically indicates a device is configured for another IP subnet,
02:01
either the computer or the Q-SYS device is configured incorrectly.
02:05
Remember that the connection from the computer running Q-SYS Configurator is one pathway,
02:10
while the connection from a Core running the system to a peripheral is different.
02:15
In some cases you’ll see the device as ‘green’ in configurator,
02:18
but see a report of device ‘missing’ when connected to the running system in Q-SYS Designer Software.
02:24
Another indicator of this is that the ‘design’ is shown as ‘idle’ on that peripheral in the configurator view.
02:29
That tells you that the Core can’t find that peripheral to get it configured.
02:33
There are a few different potential problems:
02:36
One - maybe the peripheral could have the wrong IP configuration and be on a different subnet than the Core.
02:42
Two - The peripheral could have a name that doesn’t match the Q-SYS inventory item in the design file.
02:47
Or three - Multicast discovery packets may not be getting from the Core
02:52
to the network segment the peripheral is on.
02:54
The point here is to remember that there are multicast and unicast elements at play,
02:59
so try to separate those two elements when troubleshooting.
03:02
It’ll help you find solutions much faster.
03:05
Sometimes you might be looking to Q-SYS eventlogs to lead you to understand a problem
03:10
but there seems to be too much information to grasp.
03:13
In older event viewers, you can use Control – F to ‘find’ the items you’re interested in very quickly.
03:19
In Core Manager there’s the ‘search messages’ field with the same functionality.
03:24
This can help you quickly when the other filters aren’t showing you the entries you’d like to see.
03:29
Remember the built in PING, ARP and TRACE route commands
03:34
can give you a quick assessment of network connectivity.
03:36
A PING helps you make sure something is at the IP address you expect the device to respond to.
03:41
If you have a response but something still doesn’t seem right,
03:44
you might try the ARP command to make sure the correct MAC address is associated with that IP.
03:50
If not, you know right away you have a conflict on the network.
03:54
The trace route command can help find problems with routed paths like those to hosted SIP providers.
04:00
One of the most powerful troubleshooting tools in Q-SYS in regard to the network
04:05
is the ability to run a PCAP capture directly on the Core or peripheral itself.
04:11
This isn’t going to help you troubleshoot Q-LAN or other streaming protocols,
04:15
but it's very useful for troubleshooting problems with a Q-SYS softphone,
04:19
external control issues and a number of other things.
04:22
Remember that Q-SYS peripherals don’t have a lot of memory to store large PCAP files, so plan accordingly.
04:28
When it comes time to download and analyze these files,
04:31
remember you’ll need the Wireshark program to see and understand them.
04:35
Once our devices are properly configured and behaving on the network,
04:39
we often uncover other unicast and multicast issues related more to switch configuration.
04:45
As mentioned in other lectures, sometimes it’s to our advantage
04:49
to create multiple virtual networks on the same set of switches.
04:53
For example, we might want Q-SYS and Q-LAN to have a dedicated VLAN
04:57
on a corporate network to separate it from the other common data network.
05:01
When this configuration is applied, it can add a layer of complication to troubleshooting.
05:06
One network port may be a part of one VLAN while the very next is on a different one.
05:12
Plugging the Q-SYS device into the wrong one could be placing it on the wrong network.
05:16
When there are connection problems,
05:18
you might have to check the port configuration to make sure that it’s assigned to the correct VLAN.
05:24
Network switches organize VLANs using 802.1q tags,
05:29
which are numbers embedded into network packets so they know what packet should ride on which VLAN.
05:35
Sometimes packets leave the switch with those tags still in place,
05:39
in which case it seems like Q-SYS didn’t get the message at all.
05:42
Q-SYS does not process VLAN tagged packets.
05:46
We want to make sure that packets are configured as ‘untagged’ on egress in the switch configuration for Q-SYS.
05:52
We also want to make sure the switch is applying the tag from a Q-SYS port into the infrastructure,
05:57
as Q-SYS doesn’t do that either.
05:59
When multiple VLANs must cross a link between switches, the group of VLANs is called a trunk.
06:05
Another common issue when all the devices are missing on another switch
06:09
is that the appropriate VLAN wasn’t trunked from switch to switch.
06:13
This is another aspect that can only be found by looking at the configuration of the switches themselves.
06:19
One other important thing to remember is that a 1Gb link
06:23
between switches can carry only a maximum of 1Gb combined for ALL VLANs.
06:29
Very often one designer is working on the equipment for one VLAN…maybe video, while another is
06:34
working on the audio VLAN thinking they have the entire uplink speed to work with.
06:39
That is not true.
06:40
This can especially become a choke point for networks
06:43
that must carry 4K video streams over uplinks between switches.
06:48
If we see Q-SYS devices disappearing on the network or complete loss of network connectivity
06:53
to everything on the network, it may be related to IGMP problems on the switch.
06:58
This is especially true for networks carrying a lot of multicast video data.
07:03
Have you ever connected two high-bandwidth multicast video encoders onto a network without IGMP filtering?
07:09
The result is often a complete flood of the network.
07:13
Connectivity is lost to everything until you unplug an encoder.
07:17
Remember that a few things must be true to make IGMP snooping and filtering functionality work properly.
07:24
There must be one and ONLY one IGMP querier on each network or VLAN requiring IGMP functionality.
07:31
If there isn’t one, devices will drop off after a time when the IGMP tables are periodically cleared in the switch.
07:38
If there are multiples, the tables in each switch might become incorrect,
07:43
which can cause the network to act unpredictably.
07:45
Remember also that IGMP filtering must often be enabled in a separate step of switch configuration.
07:52
If that step isn’t taken, ports can still be saturated by multicast traffic.
07:58
When you’re not sure if you’re dealing with an IGMP related issue it’s often good to take these steps.
08:03
Temporarily unplug high-bandwidth devices and see if the network recovers.
08:08
If so, check the IGMP configuration immediately.
08:12
If that step doesn’t help, try disabling IGMP to see if the remaining devices start to react more predictably.
08:19
This is often proof that either there isn’t a querier on the network in question.
08:24
If you know where to look in the switch configuration, you can often see
08:27
the table of multicast groups and the ports assigned to them.
08:30
If all are ports our assigned to every group, then multicast filtering isn’t working.
08:35
You’ll need to check that part of the configuration.
08:38
Finally, if nothing else works, you can take a PCAP capture at a Q-SYS device
08:43
and look for more advanced problems with IGMP configuration.