Angelia networking weirdness - using Ethernet broadcast address
Posted: Sun Jan 06, 2019 4:21 pm
All,
I purchased an Anan Angelia board from India around three years ago and use it in my home station for VHF DXing on receive with Herman's CuSDR principally on 144MHz with a feed from my transverter IF output but also use it on 6m during the ES season.
When I first had the board it was reliable. Some time around October 2016 it became unreliable on my home network - this may be a coincidence but it was a few weeks after I upgraded to Netgear GS724TP managed switches and fibre between the shack and the garage.
Here's some videos of it misbehaving back in 2016:
https://www.youtube.com/watch?v=EJye6ehOxdk
https://www.youtube.com/watch?v=n_LL03NLYMk
https://www.youtube.com/watch?v=AVgX37c5YBg
Basically it was nearly impossible for it to obtain an IP address via DHCP from my firewall/gateway in the garage, but the strange thing I noticed is that it always works first time when we use it at our contest site - here its plugged into an old Netgear JGS516 unmanaged 16-port Gigabit switch. Just power up the Anan, power up the PC, switch on and go - CuSDR finds it first time.
So, a couple of nights ago I finally got to grips with it, a new Window 10 PC, a Dlink router with a 4-port switch and DHCP server and an old Netgear FS105 10/100 switch.
I tested several scenarios while following Rob W1AEX's guide that I found on the 'net. Three network scenarios:
1. Angelia and PC connected to main Netgear GS724TP switch (Dlink and FS105 not used)
2. Angelia and PC connected to Dlink and routed onward to main network (separate layer-3 domains)
3. Angelia and PC connected to FS105 and uplinked to main network
Testing scenario 1
I can get the Angelia to work about one time in fifty - when it works the port on the GS724TP shows link up, 100Mbps, full-duplex, MAC address 00:04:A3:6A:67:F1 learned and my Linux firewall/gateway (two Ethernet L2 hops away) hands off an IP address via DHCP, CuSDR comes up and works.
About 98% of the time (the other 49 in 50) nothing happens, the GS724TP shows link up, 100Mbps, full-duplex, no learned MAC address, no DHCP request reaches gate.tubby.org and hence no reply, no IP address, no connectivity.
Testing scenario 2
This worked a couple of times and failed a couple of times. I quite quickly moved on to scenario 3.
Testing scenario 3
In scenario 3 I found that CuSDR running on my new PC attached to the same switch as the Angelia could find the SDR and worked. Attempting to access the Angelia from my old PC through the GS724TP then the FS108 failed.
I found this odd so installed Wireshark on the new PC and found that CuSDR was talking to the Angelia on IP address 169.254.255.255 which made me suspicious. Delving deeper I found that on start-up CuSDR sends a UDP/IP broadcast to discover the Angelia to IP address 255.255.255.255 and the Angelia responds from 169.254.255.255. The PC then performs an ARP 'Who is 169.254.255.255' and gets the response '169.254.255.255 is-at FF:FF:FF:FF:FF:FF'.
This tells us that the Angelia is using FF:FF:FF:FF:FF:FF as it's source address and explains why it is reachable on the same (dumb) switch but won't pass through a managed switch and hence why DHCP doesn't work.
Finding a lost Anan
I followed Rob's notes and after many attempts couldn't get the HPSDRProgrammer version 2.0.4.10 to find my Angelia in any combination of the network scenarios - even with Windows firewall disabled and running as Administrator I cannot find the board.
I switched to the HPSDRBootLoader program and it found the Angelia almost instantly when the PC and Angelia were both connected to the FS105 (dumb) switch, however BootLoader reports that the MAC address is FF:FF:FF:FF:FF:FF - see here:
https://www.youtube.com/watch?v=FitKHImFsJ4
Reflashed the Angelia
I was able to use HPSDRBootLoader to re-flash the Angelia 5.4 RBF - this worked first time - however its made no difference to the reliability of the board.
Conclusions
For some reason my board is coming up without its MAC address around 49 times in 50 and the Ethernet broadcast address FF:FF:FF:FF:FF:FF is used instead.
The Ethernet broadcast address FF:FF:FF:FF:FF:FF cannot be used in the Source MAC field through managed switches and this explains why I cannot use the Angelia on my normal network.
The Angelia is probably attempting DHCP but failing using Ethernet address FF:FF:FF:FF:FF:FF as it falls back to the Microsoft blackhole address 169.254.xxx.yyy. The actual address is always 169.254.255.255 which I believe further confirms the problem with the MAC address as xxx and yyy are usually taken from the last two bytes of the MAC address.
On the rare occasions when the Angelia uses MAC address 00:04:A3:6A:67:F1 it successfully obtains an IP address via DHCP on my main network through two managed switches.
I suspect that CuSDR is working by accident rather than by design when it probes for the Anan/Angelia/Hermes at start-up via UDP and gets a response from 169.254.255.255 on the Ethernet broadcast address as this works with the PC and Angelia on the same (dumb) switch.
What next?
Firstly I assume that the MAC address 00:04:A3:6A:67:F1 which I see once in a blue moon is actually the correct address for my Angelia board? The OUI-preix 00:04:A3 appears to be assigned to Microchip which makes sense for the KSZ9021 if that is where the MAC address is stored.
Next I need to find out the root cause for the MAC address problem... hunches and guesses:
1. I have a faulty board?
2. Power on reset timing issue?
3. Power supply power-up sequencing problem?
4. Flakey firmware/marginal timing issue?
Assuming that my board isn't faulty then the other choices are that there is some sort of hardware or software issue that is marginal - perhaps between the FPGA and the KSZ9021?
My Angelia is powered from a Meanwell PT-4503 45W triple output power supply, directly into the three supply inputs on the Angelia. All three supplies are within 50mV of their specified values although I can't say for sure in which order they come up...
Does the Ethernet implementation in the FPGA ensure that the PHY is set up correctly? Is it possible that there is a race condition or something flakey in the initialisation sequence that just shows up a lot on my board?
Any thoughts or suggestions appreciated?
Mike G8TIC/M0VHF/VK2SHF/9M6BAA
I purchased an Anan Angelia board from India around three years ago and use it in my home station for VHF DXing on receive with Herman's CuSDR principally on 144MHz with a feed from my transverter IF output but also use it on 6m during the ES season.
When I first had the board it was reliable. Some time around October 2016 it became unreliable on my home network - this may be a coincidence but it was a few weeks after I upgraded to Netgear GS724TP managed switches and fibre between the shack and the garage.
Here's some videos of it misbehaving back in 2016:
https://www.youtube.com/watch?v=EJye6ehOxdk
https://www.youtube.com/watch?v=n_LL03NLYMk
https://www.youtube.com/watch?v=AVgX37c5YBg
Basically it was nearly impossible for it to obtain an IP address via DHCP from my firewall/gateway in the garage, but the strange thing I noticed is that it always works first time when we use it at our contest site - here its plugged into an old Netgear JGS516 unmanaged 16-port Gigabit switch. Just power up the Anan, power up the PC, switch on and go - CuSDR finds it first time.
So, a couple of nights ago I finally got to grips with it, a new Window 10 PC, a Dlink router with a 4-port switch and DHCP server and an old Netgear FS105 10/100 switch.
I tested several scenarios while following Rob W1AEX's guide that I found on the 'net. Three network scenarios:
1. Angelia and PC connected to main Netgear GS724TP switch (Dlink and FS105 not used)
2. Angelia and PC connected to Dlink and routed onward to main network (separate layer-3 domains)
3. Angelia and PC connected to FS105 and uplinked to main network
Testing scenario 1
I can get the Angelia to work about one time in fifty - when it works the port on the GS724TP shows link up, 100Mbps, full-duplex, MAC address 00:04:A3:6A:67:F1 learned and my Linux firewall/gateway (two Ethernet L2 hops away) hands off an IP address via DHCP, CuSDR comes up and works.
About 98% of the time (the other 49 in 50) nothing happens, the GS724TP shows link up, 100Mbps, full-duplex, no learned MAC address, no DHCP request reaches gate.tubby.org and hence no reply, no IP address, no connectivity.
Testing scenario 2
This worked a couple of times and failed a couple of times. I quite quickly moved on to scenario 3.
Testing scenario 3
In scenario 3 I found that CuSDR running on my new PC attached to the same switch as the Angelia could find the SDR and worked. Attempting to access the Angelia from my old PC through the GS724TP then the FS108 failed.
I found this odd so installed Wireshark on the new PC and found that CuSDR was talking to the Angelia on IP address 169.254.255.255 which made me suspicious. Delving deeper I found that on start-up CuSDR sends a UDP/IP broadcast to discover the Angelia to IP address 255.255.255.255 and the Angelia responds from 169.254.255.255. The PC then performs an ARP 'Who is 169.254.255.255' and gets the response '169.254.255.255 is-at FF:FF:FF:FF:FF:FF'.
This tells us that the Angelia is using FF:FF:FF:FF:FF:FF as it's source address and explains why it is reachable on the same (dumb) switch but won't pass through a managed switch and hence why DHCP doesn't work.
Finding a lost Anan
I followed Rob's notes and after many attempts couldn't get the HPSDRProgrammer version 2.0.4.10 to find my Angelia in any combination of the network scenarios - even with Windows firewall disabled and running as Administrator I cannot find the board.
I switched to the HPSDRBootLoader program and it found the Angelia almost instantly when the PC and Angelia were both connected to the FS105 (dumb) switch, however BootLoader reports that the MAC address is FF:FF:FF:FF:FF:FF - see here:
https://www.youtube.com/watch?v=FitKHImFsJ4
Reflashed the Angelia
I was able to use HPSDRBootLoader to re-flash the Angelia 5.4 RBF - this worked first time - however its made no difference to the reliability of the board.
Conclusions
For some reason my board is coming up without its MAC address around 49 times in 50 and the Ethernet broadcast address FF:FF:FF:FF:FF:FF is used instead.
The Ethernet broadcast address FF:FF:FF:FF:FF:FF cannot be used in the Source MAC field through managed switches and this explains why I cannot use the Angelia on my normal network.
The Angelia is probably attempting DHCP but failing using Ethernet address FF:FF:FF:FF:FF:FF as it falls back to the Microsoft blackhole address 169.254.xxx.yyy. The actual address is always 169.254.255.255 which I believe further confirms the problem with the MAC address as xxx and yyy are usually taken from the last two bytes of the MAC address.
On the rare occasions when the Angelia uses MAC address 00:04:A3:6A:67:F1 it successfully obtains an IP address via DHCP on my main network through two managed switches.
I suspect that CuSDR is working by accident rather than by design when it probes for the Anan/Angelia/Hermes at start-up via UDP and gets a response from 169.254.255.255 on the Ethernet broadcast address as this works with the PC and Angelia on the same (dumb) switch.
What next?
Firstly I assume that the MAC address 00:04:A3:6A:67:F1 which I see once in a blue moon is actually the correct address for my Angelia board? The OUI-preix 00:04:A3 appears to be assigned to Microchip which makes sense for the KSZ9021 if that is where the MAC address is stored.
Next I need to find out the root cause for the MAC address problem... hunches and guesses:
1. I have a faulty board?
2. Power on reset timing issue?
3. Power supply power-up sequencing problem?
4. Flakey firmware/marginal timing issue?
Assuming that my board isn't faulty then the other choices are that there is some sort of hardware or software issue that is marginal - perhaps between the FPGA and the KSZ9021?
My Angelia is powered from a Meanwell PT-4503 45W triple output power supply, directly into the three supply inputs on the Angelia. All three supplies are within 50mV of their specified values although I can't say for sure in which order they come up...
Does the Ethernet implementation in the FPGA ensure that the PHY is set up correctly? Is it possible that there is a race condition or something flakey in the initialisation sequence that just shows up a lot on my board?
Any thoughts or suggestions appreciated?
Mike G8TIC/M0VHF/VK2SHF/9M6BAA