Discussion:
NFS error: rpc.nfsd: writing fd to kernel failed: errno 5 (Input/output error)
Carles Acosta
2013-02-05 15:02:32 UTC
Permalink
Dear all,

I am new to the list, so, forgive me I am doing something wrong.

I want to ask you about a strange issue on a SL6.2 server related to the
NFS service.

So, after a while working properly, we tried to restart NFS server one
week ago and we obtained:

Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS daemon: rpc.nfsd: writing fd to kernel failed: errno 5
(Input/output error)
rpc.nfsd: writing fd to kernel failed: errno 5 (Input/output error)
rpc.nfsd: unable to set any sockets for nfsd
[FAILED]

dmesg:
svc: failed to register nfsdv2 RPC service (errno 5).
svc: failed to register nfsaclv2 RPC service (errno 5).
nfsd: last server has exited, flushing export cache

We changed /etc/exports before the restart but after the error, we
restored the original /etc/exports file with no results.

We have tried to restart rpcbind, nfslock and nfs in the correct order.
Rpcbind and nfslock are working fine in principle:

[***@XXXX~]# rpcinfo -p
program vers proto port service
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper

Selinux is disabled and there is no iptables. The hardware is also ok.

These are our rpm packages:
[***@XXXX ~]# rpm -qa | grep nfs
nfs-utils-1.2.3-15.el6.x86_64
nfs-utils-lib-1.1.5-4.el6.x86_64

And our kernel:

[***@XXXX ~]# uname -a
Linux pbs04.pic.es 2.6.32-220.13.1.el6.x86_64 #1 SMP Tue Apr 17 15:16:22
CDT 2012 x86_64 x86_64 x86_64 GNU/Linux

We are using Puppet and there are no modifications in the config files.

And moreover (and more strange!) NFS is also failing as client. These
are the errors when we try to mount something with autofs:

*NFS4*
NFS: nfs mount
opts='hard,intr,tcp,rsize=32768,wsize=32768,timeo=600,sloppy,vers=4,addr=192.168.101.104,clientaddr=193.109.174.14'
NFS: parsing nfs mount option 'hard'
NFS: parsing nfs mount option 'intr'
NFS: ignoring mount option 'intr'
NFS: parsing nfs mount option 'tcp'
NFS: parsing nfs mount option 'rsize=32768'
NFS: parsing nfs mount option 'wsize=32768'
NFS: parsing nfs mount option 'timeo=600'
NFS: parsing nfs mount option 'sloppy'
NFS: relaxing parsing rules
NFS: parsing nfs mount option 'vers=4'
NFS: parsing nfs mount option 'addr=192.168.101.104'
NFS: parsing nfs mount option 'clientaddr=193.109.174.14'
NFS: MNTPATH: '/vol/vol_users'
--> nfs4_try_mount()
<-- nfs4_try_mount() = -5 [error]

*NFS3*
NFS: nfs mount
opts='hard,intr,rsize=32768,wsize=32768,timeo=600,sloppy,addr=192.168.101.104,vers=3,proto=tcp,mountvers=3,mountproto=tcp,mountport=4046'
NFS: parsing nfs mount option 'hard'
NFS: parsing nfs mount option 'intr'
NFS: ignoring mount option 'intr'
NFS: parsing nfs mount option 'rsize=32768'
NFS: parsing nfs mount option 'wsize=32768'
NFS: parsing nfs mount option 'timeo=600'
NFS: parsing nfs mount option 'sloppy'
NFS: relaxing parsing rules
NFS: parsing nfs mount option 'addr=192.168.101.104'
NFS: parsing nfs mount option 'vers=3'
NFS: parsing nfs mount option 'proto=tcp'
NFS: parsing nfs mount option 'mountvers=3'
NFS: parsing nfs mount option 'mountproto=tcp'
NFS: parsing nfs mount option 'mountport=4046'
NFS: MNTPATH: '/vol/vol_users'
NFS: sending MNT request for nfshome.pic.es:/vol/vol_users
NFS: failed to create MNT RPC client, status=-5
NFS: unable to mount server nfshome.pic.es, error -5

Maybe rebooting the machine will solve this problem, but this machine is
a batch system server and we cannot reboot it. We would really
appreciate your help.

Thank you in advance.

Regards,

Carles
Carles Acosta
2013-02-11 08:30:47 UTC
Permalink
Hello again,

After few days, the NFS server started to work again (it is difficult to
know how, we were starting and stopping services and then, magically,
the NFS server ran again...) but only for ten minutes. Trying to solve
the problem of the NFS client, we restarted the autofs service and,
then, the annoying "rpc.nfsd: writing fd to kernel failer: errno5
(Input/ouput error)" error returned on the NFS server side. So, it seems
that the issue with the NFS server is closely related (or maybe the
consequence) of the problem with NFS client.

Any idea? Could someone help us?

Thank you very much.

Regards,

Carles
Post by Carles Acosta
Dear all,
I am new to the list, so, forgive me I am doing something wrong.
I want to ask you about a strange issue on a SL6.2 server related to
the NFS service.
So, after a while working properly, we tried to restart NFS server one
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS daemon: rpc.nfsd: writing fd to kernel failed: errno 5
(Input/output error)
rpc.nfsd: writing fd to kernel failed: errno 5 (Input/output error)
rpc.nfsd: unable to set any sockets for nfsd
[FAILED]
svc: failed to register nfsdv2 RPC service (errno 5).
svc: failed to register nfsaclv2 RPC service (errno 5).
nfsd: last server has exited, flushing export cache
We changed /etc/exports before the restart but after the error, we
restored the original /etc/exports file with no results.
We have tried to restart rpcbind, nfslock and nfs in the correct
program vers proto port service
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
Selinux is disabled and there is no iptables. The hardware is also ok.
nfs-utils-1.2.3-15.el6.x86_64
nfs-utils-lib-1.1.5-4.el6.x86_64
Linux pbs04.pic.es 2.6.32-220.13.1.el6.x86_64 #1 SMP Tue Apr 17
15:16:22 CDT 2012 x86_64 x86_64 x86_64 GNU/Linux
We are using Puppet and there are no modifications in the config files.
And moreover (and more strange!) NFS is also failing as client. These
*NFS4*
NFS: nfs mount
opts='hard,intr,tcp,rsize=32768,wsize=32768,timeo=600,sloppy,vers=4,addr=192.168.101.104,clientaddr=193.109.174.14'
NFS: parsing nfs mount option 'hard'
NFS: parsing nfs mount option 'intr'
NFS: ignoring mount option 'intr'
NFS: parsing nfs mount option 'tcp'
NFS: parsing nfs mount option 'rsize=32768'
NFS: parsing nfs mount option 'wsize=32768'
NFS: parsing nfs mount option 'timeo=600'
NFS: parsing nfs mount option 'sloppy'
NFS: relaxing parsing rules
NFS: parsing nfs mount option 'vers=4'
NFS: parsing nfs mount option 'addr=192.168.101.104'
NFS: parsing nfs mount option 'clientaddr=193.109.174.14'
NFS: MNTPATH: '/vol/vol_users'
--> nfs4_try_mount()
<-- nfs4_try_mount() = -5 [error]
*NFS3*
NFS: nfs mount
opts='hard,intr,rsize=32768,wsize=32768,timeo=600,sloppy,addr=192.168.101.104,vers=3,proto=tcp,mountvers=3,mountproto=tcp,mountport=4046'
NFS: parsing nfs mount option 'hard'
NFS: parsing nfs mount option 'intr'
NFS: ignoring mount option 'intr'
NFS: parsing nfs mount option 'rsize=32768'
NFS: parsing nfs mount option 'wsize=32768'
NFS: parsing nfs mount option 'timeo=600'
NFS: parsing nfs mount option 'sloppy'
NFS: relaxing parsing rules
NFS: parsing nfs mount option 'addr=192.168.101.104'
NFS: parsing nfs mount option 'vers=3'
NFS: parsing nfs mount option 'proto=tcp'
NFS: parsing nfs mount option 'mountvers=3'
NFS: parsing nfs mount option 'mountproto=tcp'
NFS: parsing nfs mount option 'mountport=4046'
NFS: MNTPATH: '/vol/vol_users'
NFS: sending MNT request for nfshome.pic.es:/vol/vol_users
NFS: failed to create MNT RPC client, status=-5
NFS: unable to mount server nfshome.pic.es, error -5
Maybe rebooting the machine will solve this problem, but this machine
is a batch system server and we cannot reboot it. We would really
appreciate your help.
Thank you in advance.
Regards,
Carles
Loading...