config.xml empty all of a sudden
-
Just had a crash of pfsense it seams.
I network went down, and I couldn't even reach the web admin page.
Got terminal access to it, and found error messages, that pointed my to the issue.
My config.xml had been cleared, and was now empty.I've seen other report the issue as well in the forum, and some pointing to bad that change WAN IP connections frequently, running out of disk space, or loosing power.
None of these appear to be case in for me.
I additionally have an UPS on the firewall.
Anyone have any suggestions to why this appear to be happening? -
Anything logged at that time?
Did you have any packages installed that make automated changes/updates like pfBlocker?
Even if the config becomes damaged so it can no longer be loader pfSense should normally revert to the last good config instead of ending up without anything.
-
@stephenw10 I added a static ip mapping on the DHCP service, about 30 - 1 hour prior to the crash.
I still had the the web admin open, as I was referring to static bindings, but made no further changes.I also check the harddrive. It's only 18% full, and I don't see any hardware errors reported on the drive.
I've had this fiirewall for quite a long time, and have never seen this before. -
Do you still see the old configs in /conf/backup ? Do the timestamps look correct?
-
@stephenw10 Yes, I copied one of the backup configs over to restore. After I made the edit, I stepped away for about 20 minutes, and network was down when I came back.
I saw that several backup files was 0 bytes in size.
Several files, because I restarted the firewall a few times, and re-plugged the WAN link before I released, I had to connect a monitor and keyboard to the firewall to resolve the issue.Hmm, just had another look. Something clearly was happening, that I don't think was my single edit.
-rw-r--r-- 1 root wheel 218625 Jan 22 04:36 config-1737542176.xml
-rw-r--r-- 1 root wheel 218882 Jan 22 04:36 config-1737542180.xml
-rw-r--r-- 1 root wheel 218890 Jan 22 04:36 config-1737542181.xml
-rw-r--r-- 1 root wheel 218882 Jan 22 04:36 config-1737542186.xml
-rw-r--r-- 1 root wheel 218884 Jan 22 04:36 config-1737542187.xml
-rw-r--r-- 1 root wheel 218890 Jan 22 04:37 config-1737542188.xml
-rw-r--r-- 1 root wheel 218871 Jan 22 04:37 config-1737542232.xml
-rw-r--r-- 1 root wheel 218884 Jan 22 04:37 config-1737542233.xml
-rw-r--r-- 1 root wheel 218890 Jan 22 04:37 config-1737542234.xml
-rw-r--r-- 1 root wheel 218625 Jan 22 04:37 config-1737542237.xml
-rw-r--r-- 1 root wheel 218882 Jan 22 04:37 config-1737542238.xml
-rw-r--r-- 1 root wheel 218890 Jan 22 04:37 config-1737542239.xml
-rw-r--r-- 1 root wheel 218871 Jan 22 04:37 config-1737542241.xml
-rw-r--r-- 1 root wheel 218887 Jan 22 04:37 config-1737542242.xml -
Hmm, yeah. Try diff'ing those and see what was changing.
-
@stephenw10 Looks like it's WireGuard.
I tried posting example of the changes, but I was not allowed to.
It contains applying default settings, installing earlysheelcmd(s) and de-installing and then installing ACL group WireGuardWireguard was locked down to only accept connections from 2 specific IP's. I don't know if my phone ( one of the ip's ) could have triggered these. A software update maybe?
-
Ah, that can happen at boot with Wireguard so I imagine it was repeatedly rebooting? Though it's a lot inside 1 min....
-
@stephenw10
hmmm.
In system.log, I see 2 boots at 04:36:33. As soon as the first boot completed, it booted again.
I don't know why it booted though. I have a UPS providing power.
I do see the same pattern on the Jan 13th.
Maybe the UPS, was testing the battery, and briefly interrupted power?
Don't know that I have a method of verifying that though.I can understand the config file is lost, if WireGuard updates the config file, and then a second boot occur, right in between the old config is moved aside, but before the new file is written.
Maybe something to discuss with dev's. If the process can be optimized somehow, so the config file is not actually deleted, but the firmware keep a pointer to the most recent file instead.
Or maybe split the config file, separating out 3rd party add-on's, as well as more dynamic content, like DHCP address updates into separate files. Ensuring that the core configuration is never lost?
