The iss1 data manager display froze this afternoon, but Bill was able to ssh from thatcher and reboot it.  However, on reboot the networking was broken, no Internet and no ping to lan hosts.  It turned out that the networking configuration had been lost.  Worse, it looked almost correct because one of the interfaces (looking at ifconfig  output) showed the correct lan address, 192.168.0.56.

It looks like NetworkManager assigned the LAN connection configuration to the WAN ethernet device, and then left the LAN connection disconnected.  So ping to the profiler would show "network unreachable".

For reference, here is the ifconfig output when the ports are configured correctly:

[root@iss1 network-scripts]# ifconfig
enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.155.56  netmask 255.255.255.0  broadcast 192.168.155.255
        ether d4:ae:52:a7:f3:d5  txqueuelen 1000  (Ethernet)
        RX packets 11233  bytes 2713246 (2.5 MiB)
        RX errors 0  dropped 30  overruns 0  frame 0
        TX packets 15283  bytes 11810144 (11.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 16  

enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.56  netmask 255.255.255.0  broadcast 192.168.0.255
        ether d4:ae:52:a7:f3:d4  txqueuelen 1000  (Ethernet)
        RX packets 265383  bytes 370343046 (353.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 66966  bytes 6893951 (6.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17

Interface enp3s0 is the WAN port, while enp4s0 is the LAN port.

The fix was to use the system network configuration to change the port configurations.  Click on the network icon in the lower right of the screen, and try to match up the correct network connection on the left with the correct configuration on the right.  This particular time, we clicked the top connection on the left and assigned it the "WAN Static" configuration on the right.  Then click on the bottom connection on the left, and assign it the remaining configuration on the right "ISS enp4s0".

I have seen NetworkManager confuse connection configurations before, but I don't know what causes it.  Perhaps there is some configuration setting that is missing which would force the right configuration to each port.  NM should be using the "ifcfg-rh" storage backend, so it is using the ifcfg network scripts in /etc/sysconfig/network-scripts .  Looking at those scripts, I noticed that only the ifcfg-WAN_DHCP  script has a HWADDR  setting, associating that configuration with the WAN device, enp3s0, with hardware address as shown above in the ifconfig output, d4:ae:52:a7:f3:d5.  (Note the ifcfg-WAN_DHCP script unfortunately has a name leftover from past configurations, but it does in fact contain the configuration with the "WAN Static" description.)

I'm guessing that NM can initialize the ports in different orders, and in this case it configured the enp3s0 port first, but assigned it the ifcfg-enp4s0 config because that one did not have a HWADDR setting.  I have since fixed that file to contain a HWADDR setting for the LAN port, and for good measure I added the DEVICE setting also.  I don't know why that was missing either...

[root@iss1 network-scripts]# cat ifcfg-enp4s0
DEVICE=enp4s0
ONBOOT=yes
BOOTPROTO=none
NM_CONTROLLED="yes"
DEFROUTE=no
NAME="ISS enp4s0"
IPADDR=192.168.0.56
NETMASK=255.255.255.0
TYPE=Ethernet
NETWORK=192.168.0.0
BROADCAST=192.168.0.255
DOMAIN=field.eol.ucar.edu
DNS1=127.0.0.1
PROXY_METHOD=none
BROWSER_ONLY=no
PREFIX=24
IPADDR1=192.168.0.1
PREFIX1=24
NETMASK1=255.255.255.0
IPV4_FAILURE_FATAL=no
IPV6INIT=no
UUID=b325fd44-30b3-c744-3fc9-e154b78e8c82
HWADDR=D4:AE:52:A7:F3:D4

I added the device setting to the WAN config also.

[root@iss1 network-scripts]# cat ifcfg-WAN_DHCP 
DEVICE=enp3s0
ETHTOOL_OPTS="autoneg on"
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="WAN static"
UUID=2d18da56-d5b6-4e8e-9873-21e35ace2439
ONBOOT=yes
HWADDR=D4:AE:52:A7:F3:D5
IPADDR=192.168.155.56
PREFIX=24
GATEWAY=192.168.155.1
DNS1=192.168.155.1

Hopefully that will prevent future mix-ups on reboots.


  • No labels