[ILUG] NFS Issues on network (RPC timeouts)

[Lance Dryden]

Walter Faleiro wrote:
> Folks,
> I have been trying to troubleshoot a network issue for the past couple of
> days without much success.

[...]

> Now the NFS error does not appear for the Netapp Filer.

Just for fun, what's the output of "rpcinfo -p 192.168.11.38" from one 
of the Linux NFS clients?

> What I did was rebooted all servers, Netapp and the Infrant boxes. Kept only
> the linux servers, Filer and Infrant boxes on the backbone switch and
> started all the devices. No RPC errors.  When I selectively plugged in the
> uplinks, it started the RPC timeouts when I plugged in one of the network
> switches that connects the desktops.
> Now its a laymans guess that something in the network is flodding. But if so
> then why only the Infrant boxes give RPC error.
> df -k does not exit normally after this.
> Also unplugging the uplink in question does not get the linux boxes to
> respond normally to df -k.

Chances are, not by default.  Have the linux clients mount with 
"soft,intr", and for debugging, set a very short timeout.  They still 
won't recover "quickly", but a default "hard" mount means "NFS 
filesystem driver will try to satisfy requests to read or write blocks, 
forever, until it gets them."

> What type of attack can this be that affects only a type of NFS server? Also
> are there any tools I can use to get the culprit.

As for the exact condition you're dealing with, I'm not sure.

> One option would be to try to setup management on the Cisco switch that will
> get the traffic on each port. Any suggestion for packet analyzers?

I believe you're looking for the CLI command "monitor session," but 
check the Cisco software configuration guides for the switches you are 
using.  Look for the "SPAN" feature.

I've had to do this with tcpdump on the SPAN destination port.  It works 
to just link-up the interface that connects to the Cisco switch's SPAN 
port; an actual IP address on the interface isn't necessary, just L2 
link.  So "ifconfig $INTERFACE up" or "ip link set up dev $INTERFACE" 
are fine.

-- Lance