Network file system for "exclusive use" by one client only
Hi, I am looking for a "slimmer" NFS when I have a file server with a lkot of space - while clients with not much. I would like to export parts of the server file systems but do not need to share them between the clients. So I don't need all the overhead of leases that expire, server-side locking etc. Instead it would be good enough to do locking on the client side and write a journal which can be send to the file server. The file server needs to provide read access to files and be able to write what's given to it from the client. UIDs etc. could be send through, the permission check would happen at the client side. I guess that's all needed and would be faster and less complex than NFS. Does this kind of network file system exist? Any other ideas for very fast access on file system level over the net welcome. Regards Peter
On Mon, 21 Sep 2015 07:54:55 PM Peter Ross wrote:
I would like to export parts of the server file systems but do not need to share them between the clients. So I don't need all the overhead of leases that expire, server-side locking etc.
Instead it would be good enough to do locking on the client side and write a journal which can be send to the file server.
Isn't this basically NFSv4 delegations? If you're exporting it to just a single client then I'd expect no other client can interfere with them. https://www.ietf.org/rfc/rfc3530.txt 1.4.6. Client Caching and Delegation [...] The major addition to NFS version 4 in the area of caching is the ability of the server to delegate certain responsibilities to the client. When the server grants a delegation for a file to a client, the client is guaranteed certain semantics with respect to the sharing of that file with other clients. At OPEN, the server may provide the client either a read or write delegation for the file. If the client is granted a read delegation, it is assured that no other client has the ability to write to the file for the duration of the delegation. If the client is granted a write delegation, the client is assured that no other client has read or write access to the file. Delegations can be recalled by the server. If another client requests access to the file in such a way that the access conflicts with the granted delegation, the server is able to notify the initial client and recall the delegation. This requires that a callback path exist between the server and client. If this callback path does not exist, then delegations can not be granted. The essence of a delegation is that it allows the client to locally service operations such as OPEN, CLOSE, LOCK, LOCKU, READ, WRITE without immediate interaction with the server. -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
Chris Samuel wrote:
On Mon, 21 Sep 2015 07:54:55 PM Peter Ross wrote:
I would like to export parts of the server file systems but do not need to share them between the clients. So I don't need all the overhead of leases that expire, server-side locking etc.
Instead it would be good enough to do locking on the client side and write a journal which can be send to the file server.
Isn't this basically NFSv4 delegations? If you're exporting it to just a single client then I'd expect no other client can interfere with them.
https://www.ietf.org/rfc/rfc3530.txt
1.4.6. Client Caching and Delegation [...]
The major addition to NFS version 4 in the area of caching is the ability of the server to delegate certain responsibilities to the client. When the server grants a delegation for a file to a client, the client is guaranteed certain semantics with respect to the sharing of that file with other clients. At OPEN, the server may provide the client either a read or write delegation for the file. If the client is granted a read delegation, it is assured that no other client has the ability to write to the file for the duration of the delegation. If the client is granted a write delegation, the client is assured that no other client has read or write access to the file.
Delegations can be recalled by the server. If another client requests access to the file in such a way that the access conflicts with the granted delegation, the server is able to notify the initial client and recall the delegation. This requires that a callback path exist between the server and client. If this callback path does not exist, then delegations can not be granted. The essence of a delegation is that it allows the client to locally service operations such as OPEN, CLOSE, LOCK, LOCKU, READ, WRITE without immediate interaction with the server.
Yes, that sounds like it. I have to investigate how to configure this behaviour. I could not find it in the nfs(5) manpage of CentOS6 servers. Yes, Russell, I thought of iSCSI but in this case I want to make use of the server filesystem. Regards Peter On Mon, Sep 21, 2015 at 9:06 PM, Chris Samuel <chris@csamuel.org> wrote:
On Mon, 21 Sep 2015 07:54:55 PM Peter Ross wrote:
I would like to export parts of the server file systems but do not need to share them between the clients. So I don't need all the overhead of leases that expire, server-side locking etc.
Instead it would be good enough to do locking on the client side and write a journal which can be send to the file server.
Isn't this basically NFSv4 delegations? If you're exporting it to just a single client then I'd expect no other client can interfere with them.
https://www.ietf.org/rfc/rfc3530.txt
1.4.6. Client Caching and Delegation [...]
The major addition to NFS version 4 in the area of caching is the ability of the server to delegate certain responsibilities to the client. When the server grants a delegation for a file to a client, the client is guaranteed certain semantics with respect to the sharing of that file with other clients. At OPEN, the server may provide the client either a read or write delegation for the file. If the client is granted a read delegation, it is assured that no other client has the ability to write to the file for the duration of the delegation. If the client is granted a write delegation, the client is assured that no other client has read or write access to the file.
Delegations can be recalled by the server. If another client requests access to the file in such a way that the access conflicts with the granted delegation, the server is able to notify the initial client and recall the delegation. This requires that a callback path exist between the server and client. If this callback path does not exist, then delegations can not be granted. The essence of a delegation is that it allows the client to locally service operations such as OPEN, CLOSE, LOCK, LOCKU, READ, WRITE without immediate interaction with the server.
-- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
_______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main
I have to investigate how to configure this [NFSv4 delegation]
behaviour. For FreeBSD I could find this in the nfsv4(4) manpage: If the file systems you are exporting are only being accessed via NFSv4 there are a couple of sysctl(8) variables that you can change, which might improve performance. vfs.nfsd.issue_delegations when set non-zero, allows the server to issue Open Delegations to clients. These delegations permit the client to manipulate the file locally on the client. Unfortunately, at this time, client use of delegations is limited, so performance gains may not be observed. This can only be enabled when the file systems being exported to NFSv4 clients are not being accessed locally on the server and, if being accessed via NFS Version 2 or 3 clients, these clients cannot be using the NLM. vfs.nfsd.enable_locallocks can be set to 0 to disable acquisition of local byte range locks. Disabling local locking can only be done if neither local accesses to the exported file systems nor the NLM is operating on them. CLIENT MOUNTS If the NFSv4 server that is being mounted on supports delegations, you can start the nfscbd(8) daemon to handle client side callbacks. This will occur if nfsuserd_enable="YES" nfscbd_enable="YES" are set in rc.conf(5). I looked around on CentOS 7 - I could not find similar functionality. That is where I would like to have it at the moment.. Regards Peter On Tue, Sep 22, 2015 at 9:41 AM, Peter Ross <petrosssit@gmail.com> wrote:
Chris Samuel wrote:
On Mon, 21 Sep 2015 07:54:55 PM Peter Ross wrote:
I would like to export parts of the server file systems but do not need to share them between the clients. So I don't need all the overhead of leases that expire, server-side locking etc.
Instead it would be good enough to do locking on the client side and write a journal which can be send to the file server.
Isn't this basically NFSv4 delegations? If you're exporting it to just a single client then I'd expect no other client can interfere with them.
https://www.ietf.org/rfc/rfc3530.txt
1.4.6. Client Caching and Delegation [...]
The major addition to NFS version 4 in the area of caching is the ability of the server to delegate certain responsibilities to the client. When the server grants a delegation for a file to a client, the client is guaranteed certain semantics with respect to the sharing of that file with other clients. At OPEN, the server may provide the client either a read or write delegation for the file. If the client is granted a read delegation, it is assured that no other client has the ability to write to the file for the duration of the delegation. If the client is granted a write delegation, the client is assured that no other client has read or write access to the file.
Delegations can be recalled by the server. If another client requests access to the file in such a way that the access conflicts with the granted delegation, the server is able to notify the initial client and recall the delegation. This requires that a callback path exist between the server and client. If this callback path does not exist, then delegations can not be granted. The essence of a delegation is that it allows the client to locally service operations such as OPEN, CLOSE, LOCK, LOCKU, READ, WRITE without immediate interaction with the server.
Yes, that sounds like it.
I have to investigate how to configure this behaviour. I could not find it in the nfs(5) manpage of CentOS6 servers.
Yes, Russell, I thought of iSCSI but in this case I want to make use of the server filesystem.
Regards Peter
On Mon, Sep 21, 2015 at 9:06 PM, Chris Samuel <chris@csamuel.org> wrote:
On Mon, 21 Sep 2015 07:54:55 PM Peter Ross wrote:
I would like to export parts of the server file systems but do not need to share them between the clients. So I don't need all the overhead of leases that expire, server-side locking etc.
Instead it would be good enough to do locking on the client side and write a journal which can be send to the file server.
Isn't this basically NFSv4 delegations? If you're exporting it to just a single client then I'd expect no other client can interfere with them.
https://www.ietf.org/rfc/rfc3530.txt
1.4.6. Client Caching and Delegation [...]
The major addition to NFS version 4 in the area of caching is the ability of the server to delegate certain responsibilities to the client. When the server grants a delegation for a file to a client, the client is guaranteed certain semantics with respect to the sharing of that file with other clients. At OPEN, the server may provide the client either a read or write delegation for the file. If the client is granted a read delegation, it is assured that no other client has the ability to write to the file for the duration of the delegation. If the client is granted a write delegation, the client is assured that no other client has read or write access to the file.
Delegations can be recalled by the server. If another client requests access to the file in such a way that the access conflicts with the granted delegation, the server is able to notify the initial client and recall the delegation. This requires that a callback path exist between the server and client. If this callback path does not exist, then delegations can not be granted. The essence of a delegation is that it allows the client to locally service operations such as OPEN, CLOSE, LOCK, LOCKU, READ, WRITE without immediate interaction with the server.
-- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
_______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main
Hi Peter, On 22/09/15 10:09, Peter Ross wrote:
I looked around on CentOS 7 - I could not find similar functionality.
According to: http://www.citi.umich.edu/projects/nfsv4/linux/features.html delegation support was added to the NFSv4 client in 2.6.9-rc1 and to the NFS server in 2.6.10, so my suspicion is that it's enabled by default. The sysctl you are looking for (on RHEL6 at least) is: [root@snowy-m ~]# sysctl fs.leases-enable fs.leases-enable = 1 So yes, it should be enabled by default in RHEL6 (haven't got any RHEL7 boxes to check). All the best, Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
Hi Chris, On Tue, Sep 22, 2015 at 10:38 AM, Chris Samuel <chris@csamuel.org> wrote:
Hi Peter,
On 22/09/15 10:09, Peter Ross wrote:
I looked around on CentOS 7 - I could not find similar functionality.
According to:
http://www.citi.umich.edu/projects/nfsv4/linux/features.html
delegation support was added to the NFSv4 client in 2.6.9-rc1 and to the NFS server in 2.6.10, so my suspicion is that it's enabled by default.
The sysctl you are looking for (on RHEL6 at least) is:
[root@snowy-m ~]# sysctl fs.leases-enable fs.leases-enable = 1
So yes, it should be enabled by default in RHEL6 (haven't got any RHEL7 boxes to check).
I am afraid the leases are not the same. I came across leases first when looking for a NFSv3 extention integrated in FreeBSD, NQNFS (not quite NFS). http://docs.freebsd.org/44doc/papers/nqnfs.html These leases expire after a short period so have to be renewed regularly to "keep" the file. In NFSv4 a callback server receives a revocation of the exclusive file access instead.The callback server, nfs4cbd, is needed according to the FreeBSD implementation, and Oracle's ( http://docs.oracle.com/cd/E19253-01/816-4555/rfsrefer-140/index.html) I cannot find any reference on Linux for it, though. Trent, NFS is called Nightmare File System for a reason. My experience started with crashing Nextstep machines with block sizes of 8192 (when trying to connect them with Solaris machines), and it went downhill from there. NFSv4 is a step forward, I hope.. I did not touch NFS for ca. 10 years, I think. I did not miss it much;-) Regards Peter
On Tue, Sep 22, 2015 at 8:34 PM, Peter Ross <petrosssit@gmail.com> wrote:
Hi Chris,
On Tue, Sep 22, 2015 at 10:38 AM, Chris Samuel <chris@csamuel.org> wrote:
Hi Peter,
On 22/09/15 10:09, Peter Ross wrote:
I looked around on CentOS 7 - I could not find similar functionality.
According to:
http://www.citi.umich.edu/projects/nfsv4/linux/features.html
delegation support was added to the NFSv4 client in 2.6.9-rc1 and to the NFS server in 2.6.10, so my suspicion is that it's enabled by default.
The sysctl you are looking for (on RHEL6 at least) is:
[root@snowy-m ~]# sysctl fs.leases-enable fs.leases-enable = 1
I have to correct myself. You are right, it is part of Linux NFSv4 delegation implementation. http://wiki.linux-nfs.org/wiki/index.php/Cluster_Coherent_NFSv4_and_Delegati... How the recall gets transmitted is still a mystery to me. If there is no daemon, it must be "inbound" using the TCP port 2049? No, that would cause real trouble when interacting with other NFS servers. Or?? The issues on the page below scare me a bit, especially http://wiki.linux-nfs.org/wiki/index.php/Server_4.0_and_4.1_issues#Callback_... May all not really important for "my case" but in general it looks a bit like a shortcut. I still think about an alternative way for me. It's all too complicated. Imagine an asynchronic connection to the server. The server does not have any leases, callbacks etc. It does not lock on its end because I don't want to have it changed by anything else. All what it has to do is supplying data, as from a local disk, and return "data written". So, a journal written for another filesystem, e.g. ext4, could be used to provide the local cache of changes. Is there a "generic" journal which could be used for this? Would it be feasible to implement such a network filesystem? Why doesn't it exist? Or does it? Regards Peter
On Tue, 22 Sep 2015 09:08:01 PM Peter Ross wrote:
I have to correct myself. You are right, it is part of Linux NFSv4 delegation implementation.
http://wiki.linux-nfs.org/wiki/index.php/Cluster_Coherent_NFSv4_and_Delegati ons
How the recall gets transmitted is still a mystery to me. If there is no daemon, it must be "inbound" using the TCP port 2049? No, that would cause real trouble when interacting with other NFS servers. Or??
It's set via a kernel module parameter I believe - see: http://comments.gmane.org/gmane.linux.nfsv4/11560 Sorry for brevity, I've developed a nasty cough and am heading for some rest now.. :-/ -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
On Tue, Sep 22, 2015 at 10:42 PM, Chris Samuel <chris@csamuel.org> wrote:
On Tue, 22 Sep 2015 09:08:01 PM Peter Ross wrote:
How the recall gets transmitted is still a mystery to me. If there is no daemon, it must be "inbound" using the TCP port 2049? No, that would cause real trouble when interacting with other NFS servers. Or??
It's set via a kernel module parameter I believe - see:
http://comments.gmane.org/gmane.linux.nfsv4/11560
Sorry for brevity, I've developed a nasty cough and am heading for some rest now.. :-/
Thanks for the enlightment, and all the best Peter
On Tue, Sep 22, 2015 at 9:08 PM, Peter Ross <petrosssit@gmail.com> wrote:
Is there a "generic" journal which could be used for this?
BTW: From FreeBSD I know gjournal, and thanks to the GEOM framework, it is easy to stack: https://www.freebsd.org/doc/en/books/handbook/geom-gjournal.html https://www.freebsd.org/cgi/man.cgi?query=gjournal&sektion=8&apropos=0&manpa... In Linuxland seems to be JBD but is it that easy to apply or only through extensive programming? Regards Peter P.S. Sorry about my enlightment spelling. I got enlightened;-)
On Mon, 21 Sep 2015 07:54:55 PM Peter Ross wrote:
I am looking for a "slimmer" NFS when I have a file server with a lkot of space - while clients with not much.
I would like to export parts of the server file systems but do not need to share them between the clients. So I don't need all the overhead of leases that expire, server-side locking etc.
Instead it would be good enough to do locking on the client side and write a journal which can be send to the file server.
Have you considered iSCSI or one of the other NBD options? -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
Peter Ross <petrosssit@gmail.com> writes:
Instead it would be good enough to do locking on the client side and write a journal which can be send to the file server.
Does locking *ever* work properly on a network filesystem? On sshfs (SFTP) it's not implemented. On AFS it's "advisory", i.e. clients can ignore it. On Samba3/4 I usually have to set it somewhere between "off" and "more off" to appease MS Office (& LO). What I've seen of rpc.statd (NFSv3) looks pretty hinkey, too.
On Tue, 22 Sep 2015 12:20:16 PM Trent W. Buck wrote:
Does locking *ever* work properly on a network filesystem?
I believe both GPFS and Lustre work well (GPFS certainly does it as we use it in production at work). https://en.wikipedia.org/wiki/IBM_General_Parallel_File_System#Architecture https://en.wikipedia.org/wiki/Lustre_(file_system)#Locking GPFS is proprietary of course, and Lustre is open source (kind-of-sort-of premiumware with a bunch of people selling appliances and consultancy). Neither could be characterised as simple.. -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
participants (4)
-
Chris Samuel -
Peter Ross -
Russell Coker -
trentbuck＠gmail.com