[Closed] Network / Pi experts - WTF is going on?

IP address clash, perhaps

Nope, that was my first thought but it is on a fixed address in a reserved range.

By way of an update, after around an hour the router rebooted again.
I disconnected the power to the pi but left the network cable attached and it has been stable for a couple of hours now

Try a different build on the Pi, see if it's the software it's running or the cabling / hardware itself.

Any error logging you can set up / review on the router?

Well it’s been up for the past 5 hours with the pi connected by Ethernet but not powered on, so it looks like a pi software issue.

I did grab the syslog from the router before I’d worked out which device was causing me issues. I’ve sent that off to draytek support as there was nothing meaningful that I could see (exception codes that I couldn’t find a decoder for)...

Fortunately it’s dead simple to flash the fr24 image and start again, although I wouldn’t mind working out the problem properly.

The next worry of course is whether there is something nefarious doing something akin to a DoS attack, although when I was watching the data flow nothing was apparent.
I might have to investigate the security of the pi, although I’m certain that I changed to a secure password when I set it up.

Sorry wrong room, thought you said pie experts....

Try a standard rasbian image on the pi.
Sounds like it's the FR24 image that's somehow crappy.

Whatever the Pi is doing it's bad that it can cause the router to crash by doing it.

Any chance of sticking something "in line" between the Pi and router and watching the network traffic to see what the problem packets are?

I’d argue that if the pi is doing something that causes the router to reboot, the problem is with the router - it should be able to handle pretty much anything being thrown at it from a consumer perspective. Best bet to find out what’s going on is a packet capture, you should be able to set a tcpdump going on the pi and then look at it to see what was going on around the time of the router reboot.

I’d argue that if the pi is doing something that causes the router to reboot, the problem is with the router – it should be able to handle pretty much anything being thrown at it from a consumer perspective.

Was on the latest firmware, tried rolling back to a previous version and now gone back to the latest.

The fr24 image has not been changed (to my knowledge) at the time the problem started - I believe there is an upgrade back door for fr24 to maintain the feeder, but it is unlikely that they sent an update just at the time that I carried out a factory reset of the router.

Ditto any hardware issues.

Capturing tcp packets is beyond my skill set - any hints on how to accomplish that?

"connected but off" does not rule out hardware issues. Though I'd agree with the above, if that's the case it's a problem with the router.

For packet capture, use tcpdump (it’s built into Linux). Open terminal and run this:

tcpdump not port 22 -U -n -w - | tee filename.$(date +%Y-%m-%d.%Z.%H.%M.%S).pcap | tcpdump -r -

When the router reboots you can stop the capture and have a look through the output. Happy to take a look at the put out file if you like.

The fr24 image is set up without booting to desktop, I normally access it through SSH.
Can I run tcpdump through that?

Just out of interest, why is the Pi in the loft??Scrub that - better reception I presume.

Could you retrieve it and plug it in using a different ethernet cable (into a different router port) just to rule out those two things?

Edit: I see that Pi24 is "Pi24 image is based on Raspbian lite" - shouldn't be any issues there.

^^ good idea also.

Basically, troubleshooting an "it could be anything" problem is to start ruling things out. It could be the application, the OS, the Pi itself, the cable run, the switchport on the router, coincidence, something else... ask yourself, "how do I cross these things off?"

I must have read your minds, I pulled the pi from the loft this morning as it would be easier to work with.

Plugged directly into the router using the same patch cable and within minutes (before I even connected the hdmi) the router had rebooted after over 24hrs stable running.

Swapped over the patch cable for a longer one to get me closer to the tv (the hdmi cable I found was a 1m tiddler!) and resumed testing - As an aside I was right the pi24 image is set up without desktop. It hasn’t been long enough to be certain, but there is definitely a good chance that it is working.

Bearing in mind that the problem was initiated by only a software factory reset of the router, it still seems weird that it should be a hardware (cable) fault.

May have spoken too soon, 5 hours after reinstalling in place with a new Ethernet cable it has just rebooted again...

Short in Pi network socket?

And again...
Best fetch it back out of the loft!

Have you got a spare network switch of any kind lying around?

Aha, plugged pi direct into the router again and reboot initiated in <5 mins...
With the different cable.

Will reflash.

Can’t see any short in the socket.

I’ve got dumb hubs but no switches.

Rats. A dumb hub won't cut it. Can you fire up the router and see if there's a setting in there for any kind of broadcast storm control? If so, enable it. It would normally be on by default but the reset you did may have changed this.

Yes you can run tcpdump via ssh. The command I gave above will filter out all the traffic generated by opening the ssh session.

Nothing like that I can see in the manual for the router.

The thick plottens though... my SSH client can’t log in to the pi with it’s stored username and password.

Assuming you can log in via SSH, what time/date does the Pi think it is?

I’m not sure of your application particularly, but I’ve had issues with the lack of a real time clock on the Pis. Not that that should take down your router...

The thick plottens though… my SSH client can’t log in to the pi with it’s stored username and password.

Questions....
Have you reflashed the pi {with a newly downloaded} image?
If so have you changed the UN and password from the default ones?
Your client may have an issue with the SSH fingerprint changing.
Did you try a different port on the router?
Did you have eveything else disconnected from the router when you were testing the Pi?
Can you swap the pi for another?

(RF24 is based on rasbian light which doesn't install a desktop - headless only AFAIA)

Assuming you can log in via SSH, what time/date does the Pi think it is?

As the pi has been rebooted and reconnected to the internet it would have updated it clock.

Well I've checked the passwords that could be in there and SSH won't let me in at all.

I've flashed the image to a spare SD card and will set up as a new station, we'll soon find out if it is a Pi software thing.

Have you reflashed the pi image? Not yet - setting up a spare SD card so the funny one will remain intact.
If so have you changed the UN and password from the default ones? I believe so (keychain has a password) but there is a possibility that might not be the case
Your client may have an issue with the SSH fingerprint changing. I've reset the security on that having experienced that before
Did you try a different port on the router? yes
Did you have eveything else disconnected from the router when you were testing the Pi? no, but system ran fine for 24hrs+ with pi disconnected, are you suspecting ip conflict etc?
Can you swap the pi for another? I've got another 2B sat around somewhere to try if it looks like hardware

(RF24 is based on rasbian light which doesn’t install a desktop – headless only AFAIA) correct

SSH isn't switched on by default in rasbian light.

I think the fix is to create an empty file called SSH and put it in the boot directory on the pi card.
The pi will see at boot, switch on SSH and delete the file.

Edit:
Wait..... SSH is switched on isn't it 🤦🏻‍♂️

Have you tried the default (pi, raspberry)?

Have you got a spare network switch of any kind lying around?

Wouldn't the old router work as a switch?

Wouldn’t the old router work as a switch?

It might, but it might insist on having some layer 3 stuff in there too; the routers with inbuilt switch modules usually do. I was thinking that slow-building outages like this are often some kind of broadcast storm.

As the pi has been rebooted and reconnected to the internet it would have updated it clock.

That’s what I was curious about, has the Pi actually connected to the internet? I re-read the thread and I wasn’t sure that the Pi application was confirmed to be functional or not.

....well the router ran for 5 hours with the pi attached before falling over.

yes it was connecting to the internet and showing as active on the FR24 server.

yes I checked the standard pi/raspberry combo.

I think the PI24 image has SSH turned on as standard - certainly the new image just let me in without having to activate it.

I've got it all up and running with the new image now so we'll see in the next few hours.

Am I imagining a broadcast storm right? I'm imagining a device spamming nonsense traffic down the line. Would that be a fault or done on purpose by scallywag's?

At this point, given you've seemingly ignored most of what I've suggested to date,

How long does it run without the Pi attached at all? Do you have a reliable baseline? Like, stability for days? Have you ruled out the notion that router might be sodded?

Have I misread your advice Cougar? It was following your advice and disconnecting bits that led me to the pi in the first place, was that not what you were getting at?

The router ran most of yesterday through to this morning with the pi disconnected with no issues, granted not days - but a night and day difference and a fairly good indication I thought.

I haven't completely ruled out a router issue, especially as Draytek support are unable to tell me anything about why it reboots from their own log files.

We'll see if there are any more reboots overnight...

The router ran most of yesterday through to this morning with the pi disconnected with no issues, granted not days – but a night and day difference and a fairly good indication I thought.

So the Pi is inside the firewall (remains to be seen just how effective that is), so it "should" be protected from outside nasties. Plus you've put a new image on it so it should be clean of nasties also.
I'm surprised that a router will fall over "so easily" - I do suspect that the router is the issue.

Edit: a quick google shows plenty of history for Draytek random reboots. Have you disabled IPV6?

Have I misread your advice Cougar?

Sorry, I was being needlessly grumpy.

Try the Pi with a new OS build or even just powered up without an SD.

Well it’s been up and stable for 21hrs so far with a new build of pi24 connected directly to the router.

Actually it did have a port open through the firewall, I think that was a hangover from a previous adsb receiver project as I’ve closed it and the pi is still online according to the fr24 server.

Although the traffic from the old build of the pi didn’t seem high enough to be noteworthy, the new build is a single session and not high enough nitrate to register.

Actually it did have a port open through the firewall

Just to confirm.... the pi was connected to an open port through the router - but not in a DMZ.
You've now closed the port, are running a newly burnt image on the pi and it all seems to be stable?
Hmmm.

Yes that’s right.

I don’t even want to put the SD card with the original image into my computer to erase it, never mind interrogate it!

I assume you would have similar worries unless you have a VM set up for testing such things?

Yes that’s right.

Eeek! 😬

Well there's a decent chance that it was compromised via the open port I guess.
(I think you'll be fine formatting it on a Windows machine).

Hopefully your system will now run OK.
Maybe no more open ports eh?!

If I need SSH access from outside of my LAN I usually port forward a different port to 22.
Another good idea is to run fail2ban on the Pi

(later worked out that if you enter an SSID passphrase that keychain saves this over the username

haven’t found this when setting up an ASUS router. Keychain stores the Wi-Fi password in an appropriate keychain item. It stores the admin credentials in a website keychain item related to the router’s IP address and its URL.

though is the problem you describe that you’re in the admin site for the router and then setting the SSID password? I can see that would overwrite your URL/IP-related password. Next time just say ‘no, do not save this password’ and your login credentials will be preserved.

yes exactly that - easy enough to say 'no' when you realise what it's going to save!

I assume you would have similar worries unless you have a VM set up for testing such things?

Why would you need a port open anyway - isn't/wasn't all the data flowing out of your network?

I used to use a different image for a different service and they were doing something that had data flowing in. When I rebuilt, I used my notes which included that old port.