Excellent tool, excellent article
|
| Author | Content |
|---|---|
| caitlyn Aug 17, 2009 12:29 PM EDT |
A while back I did some work for a company which had downsized and consolidated their six R&D locations into a single office with a much smaller team. They wanted to consolidate their servers as well which made sense under the conditions. Anyway, to make a long story short, we needed to archive the filesystems, transfer them via NFS to their new home, and unarchive them in the appropriate place. The tool that I used to do this was cpio. It was far and away the fastest and most efficient way to handle large amounts of data. cpio has been around in *nix forever. It's a pity more people don't know about it and use it. I'm amazed how many sysadmins I've talked to who just never tried using cpio. |
| gus3 Aug 17, 2009 12:39 PM EDT |
Dumb question time. Why not use SFTP (or FTP if it was trusted) to transfer the archives, and just run cpio locally? In my experience, even network tar/gzip pipes is more efficient than NFS for bulk transfers. EDIT: Fixed semantic error. |
| caitlyn Aug 17, 2009 1:06 PM EDT |
@gus3: We tried it and on the network in question cpio proved to be faster. Obviously, YMMV. Their network had, um... interesting issues that may have played a part. In any case, even locally, cpio is faster than tar. |
| gus3 Aug 17, 2009 2:29 PM EDT |
Apparently, I didn't fix enough of the semantic error to be clear. Seems to be a habit lately. My impression from your first comment, is that you had an NFS mount, and were running cpio on the NFS client, while the files you unpacked were sent via NFS to the server. What I'm wondering is, why you didn't use a streaming (not datagram) protocol to transfer the archive to the server, then run cpio directly on the server? |
| caitlyn Aug 17, 2009 2:44 PM EDT |
I think I was the one who didn't communicate clearly. We tried what you suggest. It wasn't faster in their environment. There were a lot of issues with that network which might have contributed to that result but the fact is that using cpio across an NFS mount worked better. |
| Steven_Rosenber Aug 17, 2009 5:20 PM EDT |
What advantages does this method have over using rsync? |
| gus3 Aug 17, 2009 5:27 PM EDT |
Rsync uses TCP. The dodgy network in question apparently deals better with datagram transmission than with streams. |
| caitlyn Aug 17, 2009 6:23 PM EDT |
@Steven: In addition to the issues regarding transmission of large arcives that gus3 and I have been discussing there is the issue of creating the archive itself. cpio is much faster than tar as I mentioned before. The combination of the two speed improvements made the choice a no-brainer. Oh, and the script to do this was two lines long: one to do the archive, transfer, and unarchive and the second to send a notification that the process was completed. |
Posting in this forum is limited to members of the group: [ForumMods, SITEADMINS, MEMBERS.]
Becoming a member of LXer is easy and free. Join Us!