Angelia networking weirdness - using Ethernet broadcast address

Can't talk to your radio? This is the place to ask!
G8TIC
Posts: 7
Joined: Sat Jan 05, 2019 9:45 pm

Angelia networking weirdness - using Ethernet broadcast address

Postby G8TIC » Sun Jan 06, 2019 4:21 pm

All,

I purchased an Anan Angelia board from India around three years ago and use it in my home station for VHF DXing on receive with Herman's CuSDR principally on 144MHz with a feed from my transverter IF output but also use it on 6m during the ES season.

When I first had the board it was reliable. Some time around October 2016 it became unreliable on my home network - this may be a coincidence but it was a few weeks after I upgraded to Netgear GS724TP managed switches and fibre between the shack and the garage.

Here's some videos of it misbehaving back in 2016:

https://www.youtube.com/watch?v=EJye6ehOxdk

https://www.youtube.com/watch?v=n_LL03NLYMk

https://www.youtube.com/watch?v=AVgX37c5YBg

Basically it was nearly impossible for it to obtain an IP address via DHCP from my firewall/gateway in the garage, but the strange thing I noticed is that it always works first time when we use it at our contest site - here its plugged into an old Netgear JGS516 unmanaged 16-port Gigabit switch. Just power up the Anan, power up the PC, switch on and go - CuSDR finds it first time.


So, a couple of nights ago I finally got to grips with it, a new Window 10 PC, a Dlink router with a 4-port switch and DHCP server and an old Netgear FS105 10/100 switch.

I tested several scenarios while following Rob W1AEX's guide that I found on the 'net. Three network scenarios:

1. Angelia and PC connected to main Netgear GS724TP switch (Dlink and FS105 not used)

2. Angelia and PC connected to Dlink and routed onward to main network (separate layer-3 domains)

3. Angelia and PC connected to FS105 and uplinked to main network


Testing scenario 1

I can get the Angelia to work about one time in fifty - when it works the port on the GS724TP shows link up, 100Mbps, full-duplex, MAC address 00:04:A3:6A:67:F1 learned and my Linux firewall/gateway (two Ethernet L2 hops away) hands off an IP address via DHCP, CuSDR comes up and works.

About 98% of the time (the other 49 in 50) nothing happens, the GS724TP shows link up, 100Mbps, full-duplex, no learned MAC address, no DHCP request reaches gate.tubby.org and hence no reply, no IP address, no connectivity.


Testing scenario 2

This worked a couple of times and failed a couple of times. I quite quickly moved on to scenario 3.


Testing scenario 3

In scenario 3 I found that CuSDR running on my new PC attached to the same switch as the Angelia could find the SDR and worked. Attempting to access the Angelia from my old PC through the GS724TP then the FS108 failed.

I found this odd so installed Wireshark on the new PC and found that CuSDR was talking to the Angelia on IP address 169.254.255.255 which made me suspicious. Delving deeper I found that on start-up CuSDR sends a UDP/IP broadcast to discover the Angelia to IP address 255.255.255.255 and the Angelia responds from 169.254.255.255. The PC then performs an ARP 'Who is 169.254.255.255' and gets the response '169.254.255.255 is-at FF:FF:FF:FF:FF:FF'.

This tells us that the Angelia is using FF:FF:FF:FF:FF:FF as it's source address and explains why it is reachable on the same (dumb) switch but won't pass through a managed switch and hence why DHCP doesn't work.


Finding a lost Anan

I followed Rob's notes and after many attempts couldn't get the HPSDRProgrammer version 2.0.4.10 to find my Angelia in any combination of the network scenarios - even with Windows firewall disabled and running as Administrator I cannot find the board.

I switched to the HPSDRBootLoader program and it found the Angelia almost instantly when the PC and Angelia were both connected to the FS105 (dumb) switch, however BootLoader reports that the MAC address is FF:FF:FF:FF:FF:FF - see here:

https://www.youtube.com/watch?v=FitKHImFsJ4


Reflashed the Angelia

I was able to use HPSDRBootLoader to re-flash the Angelia 5.4 RBF - this worked first time - however its made no difference to the reliability of the board.


Conclusions

For some reason my board is coming up without its MAC address around 49 times in 50 and the Ethernet broadcast address FF:FF:FF:FF:FF:FF is used instead.

The Ethernet broadcast address FF:FF:FF:FF:FF:FF cannot be used in the Source MAC field through managed switches and this explains why I cannot use the Angelia on my normal network.

The Angelia is probably attempting DHCP but failing using Ethernet address FF:FF:FF:FF:FF:FF as it falls back to the Microsoft blackhole address 169.254.xxx.yyy. The actual address is always 169.254.255.255 which I believe further confirms the problem with the MAC address as xxx and yyy are usually taken from the last two bytes of the MAC address.

On the rare occasions when the Angelia uses MAC address 00:04:A3:6A:67:F1 it successfully obtains an IP address via DHCP on my main network through two managed switches.

I suspect that CuSDR is working by accident rather than by design when it probes for the Anan/Angelia/Hermes at start-up via UDP and gets a response from 169.254.255.255 on the Ethernet broadcast address as this works with the PC and Angelia on the same (dumb) switch.


What next?

Firstly I assume that the MAC address 00:04:A3:6A:67:F1 which I see once in a blue moon is actually the correct address for my Angelia board? The OUI-preix 00:04:A3 appears to be assigned to Microchip which makes sense for the KSZ9021 if that is where the MAC address is stored.

Next I need to find out the root cause for the MAC address problem... hunches and guesses:
1. I have a faulty board?
2. Power on reset timing issue?
3. Power supply power-up sequencing problem?
4. Flakey firmware/marginal timing issue?

Assuming that my board isn't faulty then the other choices are that there is some sort of hardware or software issue that is marginal - perhaps between the FPGA and the KSZ9021?

My Angelia is powered from a Meanwell PT-4503 45W triple output power supply, directly into the three supply inputs on the Angelia. All three supplies are within 50mV of their specified values although I can't say for sure in which order they come up...

Does the Ethernet implementation in the FPGA ensure that the PHY is set up correctly? Is it possible that there is a race condition or something flakey in the initialisation sequence that just shows up a lot on my board?

Any thoughts or suggestions appreciated?


Mike G8TIC/M0VHF/VK2SHF/9M6BAA
User avatar
w-u-2-o
Posts: 5540
Joined: Fri Mar 10, 2017 1:47 pm

Re: Angelia networking weirdness - using Ethernet broadcast address

Postby w-u-2-o » Sun Jan 06, 2019 7:42 pm

Wow, long post, and I have to admit that I did not read it in complete detail.

First thing that bothers me: there is no reason for any addresses in the 169.x.x.x range unless you are using APIPA.

- In all of your scenarios was the radio hardware set up using bootloader or programmer to use DHCP (program address of 0.0.0.0)?
- In all of your scenarios was the radio hardware theoretically able to access a DHCP server?

Second thing: DHCP and RARP work fine in the radio MAC in all Protocol 1 firmware (the MAC is in FPGA code, the PHY is a Microchip Ethernet IC). And it works through managed switches just fine. The Microchip PHY is using a Microchip MAC address as it should. The only known illegal MAC address occurs when bringing the radio up in bootloader mode. In bootloader mode the bootloader FPGA MAC code forces the PHY to use a MAC of 01:02:03:04:05:06 which is, of course, an illegal MAC. This is why you have to use a direct connection or a dumb switch when using bootloader.

Third thing: the CuSDR (or PowerSDR, or piHPSDR, or whatever) discovery process has nothing to do with successful DHCP. The radio should get an address by itself every time, either static, DHCP or APIPA.

Suggestions:

- Do NOT use APIPA.
- Confirm the radio hardware is properly configured for DHCP and has access to a DHCP server on the port it is on (put a laptop on that port and see if it gets DHCP).
- You should be able to confirm the radio has a good IP without using any application software (e.g. CuSDR) by pinging the radio and looking at your ARP tables and DHCP lease tables.
- Failing all that, use programmer or bootloader to assign a static IP to the radio and see if it all works then.

And if that all fails I can only surmise that you have a failing PHY chip or FPGA. The only time I've ever seen a broadcast MAC or IP was during alpha testing of Protocol 2. And that was exciting, not only was it a broadcast, but it was a jabber as well and it would take down my entire un-managed subnet (couldn't use managed because of the stupid bootloader MAC) and it would stay like that until I powered down the radio unit.

73,

Scott
G8TIC
Posts: 7
Joined: Sat Jan 05, 2019 9:45 pm

Re: Angelia networking weirdness - using Ethernet broadcast address

Postby G8TIC » Tue Jan 08, 2019 10:36 pm

Hi Scott,

Thanks for the reply.

I have attached a sketch of the network as set up currently (Test Scenario 3).

I am not using APIPA - at least not deliberately! There are three Win 10 Pro machines on my network and I haven't configured APIPA, two use static IP and the one local to the Angelia is on DHCP.

My firewall/server in the garage (gate.tubby.org) has ISC-DHCP-Server running on it and this assigns IP addresses correctly to machines on my LAN - including the Windows 10 PC adjacent to the Angelia on the same FS105 dumb switch.

When put J17 on the Angelia, power it up and hit the [Test for BootLoader] button it reports that it finds a board with MAC address FF:FF:FF:FF:FF:FF.

I have re-flashed the Angelia with Protocol 1 firmware 5.9 RBF and with firmware 6.0 RBF using BootLoader and tested with the same issue.

I have used BootLoader to set the IP to 192.168.144.14 which it says it it has done but when I remove J17 and power cycle I cannot ping it.

Going back in to BootLoader and reading the IP address it says 255.255.255.255 ?

If I set the IP address back to 0.0.0.0 it says that DHCP has been enabled.

If I power cycle the Angelia and use BootLoader again, [Test for BootLoader] -> Tools -> Read IP it reports the IP address as 255.255.255.255 and Read MAC says FF:FF:FF:FF:FF:FF

In all cases CuSDR on the PC that is local to the Angelia, i.e. same dumb switch, works (probably not by design) because the PC emits a probe to the IP broadcast address 255.255.255.255 and the Angelia responds using IP address 169.254.255.255. The PC then does an ARP who-is 169.254.255.255 and the Angelia appears to answer ARP 169.254.255.255 is-at FF:FF:FF:FF:FF:FF after which comms comes up but clearly this odd since (a) the MAC address in use is the broadcast address and the IP address in use is an APIPA address related to the broadcast address. In this state it works on a dumb switch but won't pass through my managed switches.

Equally, since the Angelia insists on using FF:FF:FF:FF:FF:FF as it's MAC address it's DHCP Discover doesn't reach ISC-DHCP-Server on gate.tubby.org so it cannot complete a DHCP negotiation and doesn't get a proper IP address.

If I power cycle often enough then once in a blue moon (maybe one time in 50) the Angelia uses MAC address 00:04:A3:6A:67:F1 and then it's DHCP request does reach gate.tubby.org, it gets assigned a proper IP address and works as intended. It can take 15+ mins of power cycling to hit the one time when it works correctly ...

Is there a definitive list of the jumpers that I can check, just in case one is incorrect?

Looking at the schematic, U15 is a 25AA02E48T 2Kb EEPROM, connected to the FPGA - I'm guessing this is used to store the MAC address, IP address and other settings? Is there a way to perform a test on it and/or set it back to factory defaults or otherwise re-program it in case something is corrupt?

Regards

Mike G8TIC/M0VHF
Attachments
G8TIC_Angelia_Problem.pdf
(233.29 KiB) Downloaded 365 times
User avatar
w-u-2-o
Posts: 5540
Joined: Fri Mar 10, 2017 1:47 pm

Re: Angelia networking weirdness - using Ethernet broadcast address

Postby w-u-2-o » Wed Jan 09, 2019 1:44 am

Mike,

If you truly believe there is nothing wrong with your network, then we have to assume that your Angelia board is broken somehow. There are no LAN related jumper settings other than the bootloader jumper.

However, we at least know that the radio does work occasionally and that it does accept firmware. This would seem to rule out any faulty memory. Interestingly, bootloader does work. That means that the magnetics, the PHY, and the FPGA all have to be working. This sort of points the finger back at the network.

I'm not saying this would be easy, but it would be nice to throw another Apache radio on the network and see if it works or if you observe the same problems.

73,

Scott
G8TIC
Posts: 7
Joined: Sat Jan 05, 2019 9:45 pm

Re: Angelia networking weirdness - using Ethernet broadcast address

Postby G8TIC » Wed Jan 09, 2019 9:44 pm

Hi Scott,

I've love to try another Angelia, Hermes or Orion but don't know anyone with one locally. That said, the reason that I had multiple test scenarios in my first email was to rule out network related issues - which I believe I did.

I can't see how any external issues, such as my network, would cause the Angelia to insist on using the broadcast MAC address FF:FF:FF:FF:FF:FF when it should be using its real MAC address (stored in its EEPROM).

I have attached a screen grab from the new PC adjacent to the Angelia - you can see that BootLoader says that the board is present but the MAC address is FF:FF:FF:FF:FF:FF ... you can also see that from CMD prompt I performed an 'arp -a' and that showed the IP address 169.254.255.255 mapped to the broadcast address.

I believe that the Angelia using the broadcast MAC address to be the root cause of my problems and this has to be internal to the Angelia ... possible causes might be:

* power supply power up sequence issue
* corrupt data in the EEPROM (U15)
* bad connection between the EEPROM and the FPGA
* faulty FPGA

Is there a way to read/write the EEPROM (U15) ? i.e. a programming/verification tool for it? I'm guessing that U15 stores the MAC address and other settings? Is the data in U15 stored with a checksum? Is the EEPROM copied to to the FPGA at power up? If there is a checksum on the data in the EEPROM? If there is a checksum and it fails does the RAM in the FGPA default back to all 0xFFs?

I'm asking because we used to make a vehicle based product that used a 2Kb I2C EEPROM and we had issues with reading out the data reliably during engine cranking and ended up having to read the EEPROM multiple times in a for() loop and exit if it was read correctly ...

Any way I look at this I don't think the Angelia is going to work as expected until it reliably uses its proper MAC address.


Mike G8TIC
Attachments
Angelia using MAC Broadcast Address.png
Angelia using MAC Broadcast Address.png (551.54 KiB) Viewed 12527 times
User avatar
w-u-2-o
Posts: 5540
Joined: Fri Mar 10, 2017 1:47 pm

Re: Angelia networking weirdness - using Ethernet broadcast address

Postby w-u-2-o » Wed Jan 09, 2019 10:37 pm

Mike,

I'm not questioning any of your observations.

As I noted previously, if the network is healthy we have to assume your board is broken.

There is a JTAG connector on the board. Most people use an Altera USB "Byteblaster" pod when working with it as it is known compatible with the free Quartus Prime Lite development tool. That is the environment used by the dev's to work with the FPGA.

I don't know if your skill set runs towards firmware development. However, even as a non-developer, I found it quite easy to install Quartus, download the rbf files from the Git repo, and import them into the environment for inspection. If you are handy with such things it might not be that much more difficult to do a dump of the memory.

You might also consider reloading the Bootloader partition. If it is corrupt that might be a source of your difficulty. For most people a corrupt Bootloader image is well and truly "bricked". However, Doug, W5WC should be able to send you a procedure to reload it, but only if you've got the JTAG pod.

73,

Scott
G8TIC
Posts: 7
Joined: Sat Jan 05, 2019 9:45 pm

Re: Angelia networking weirdness - using Ethernet broadcast address

Postby G8TIC » Thu Jan 10, 2019 7:04 pm

Hi Scott,

Thanks for the reply.

One thing I did do was go to work and have a rummage and found a second hand 300W ATX PSU which I plugged in, instead of the Meanwell PT-4503, and it came it with the proper MAC address and DHCP'd first time ... success! or so I thought ... however after I switched it off and back on I was back at the "only works one time in XX" (where XX is a number like 20~50) so this was a wild goose chase!

I have an Altera USB Blaster at hand and while I haven't worked with the FPGAs used in the Angelia I have worked with a range of embedded systems and am familiar with In System Programming (ISP) so am prepared to give it a try - afterall I cannot make it much worse.

If you could point be at the Quartus software, the Git Repo and pass be contact details for Doug I'll give it a go and see what I can find.

Regards

Mike G8TIC
User avatar
w-u-2-o
Posts: 5540
Joined: Fri Mar 10, 2017 1:47 pm

Re: Angelia networking weirdness - using Ethernet broadcast address

Postby w-u-2-o » Thu Jan 10, 2019 7:27 pm

Simply Google "quartus prime lite" and the link will present itself.

Everything you need to know about the normal tools and the repo is in the first three "sticky" posts here:

https://apache-labs.com/community/viewforum.php?f=18

I mis-spoke above. You want the .qar files for importation into Quartus, of course.

What is missing are the procedures for programming the Bootloader and normal partitions via JTAG. For that you will have to apply to Doug. I will send his contact info via separate cover.

73,

Scott
G8TIC
Posts: 7
Joined: Sat Jan 05, 2019 9:45 pm

Re: Angelia networking weirdness - using Ethernet broadcast address

Postby G8TIC » Sat Jan 19, 2019 10:38 pm

Scott and others,

My Angelia is back from SMT rework and a new U15 (25AA02E48T) has been installed.

Good news! I have powered up the board multiple times connected to my dumb Ethernet switch and connected to my managed switch and it came up and worked 19 times out of 20. On the one occasion it didn't, unplugging and re-plugging the Ethernet cable fixed it.

So the board has a new MAC address (54:10:ec:9f:6a:ba) and works with DHCP and I've been able to tell my DHCP server to always had off the same IP address.

Happy the networking issues are now fixed, on to the rest of the issues ...

Thanks for your contributions.

Regards

Mike G8TIC
User avatar
w-u-2-o
Posts: 5540
Joined: Fri Mar 10, 2017 1:47 pm

Re: Angelia networking weirdness - using Ethernet broadcast address

Postby w-u-2-o » Sat Jan 19, 2019 11:54 pm

Hmm...bad EEPROM. If the MAC changed that implies the PHY got replaced, too.

Thanks for filling us in on the resolution.

73!
G8TIC
Posts: 7
Joined: Sat Jan 05, 2019 9:45 pm

Re: Angelia networking weirdness - using Ethernet broadcast address

Postby G8TIC » Sun Jan 20, 2019 3:24 pm

Only U15 (25AA02E48T) was replaced. Its an SPI EEPROM, but a special one since it contains a unique read-only MAC address (OUI-48) as well as 256 bytes of read/write EEPROM. The MAC addresses for Anan/Angelia/Orion boards have the Microchip prefix since the device is made by them.

Mike
User avatar
Tony EI7BMB
Posts: 651
Joined: Sun Apr 09, 2017 2:31 pm
Location: Dublin
Contact:

Re: Angelia networking weirdness - using Ethernet broadcast address

Postby Tony EI7BMB » Sun Jan 20, 2019 3:32 pm

Hi Mike, did you ship it to Doug in the US for the repair or was it done in the UK ? Just wondering in case I ever have an issue that needs servicing.
G8TIC
Posts: 7
Joined: Sat Jan 05, 2019 9:45 pm

Re: Angelia networking weirdness - using Ethernet broadcast address

Postby G8TIC » Wed Feb 06, 2019 10:47 pm

Tom,

Once we had a candidate problem (in my case U15) I had a local SMT/SMD re-work company (Reflow or Malvern, run by Lee Mason G1CBL) replace the device.

Mike
User avatar
Tony EI7BMB
Posts: 651
Joined: Sun Apr 09, 2017 2:31 pm
Location: Dublin
Contact:

Re: Angelia networking weirdness - using Ethernet broadcast address

Postby Tony EI7BMB » Thu Feb 07, 2019 12:21 pm

Thanks Mike, useful info

Return to “Network Connections & Network Hardware”