Issues with copying large directories
- SCP is VERY slow with large numbers of files
- Data bandwith can be compressed to speed data transfer over slower networks
- Maintaining file permissions
- Not enough disk space to create a backup (2nd copy of the data)
Solutions
(Q) How can I make a backup of my home directory?
The following shows how to make a backup and copy the data to another system (without creating a 2nd copy of the data). This is valuable if your disk doesn’t have enough storage for a 2nd copy of the data being backed up.
Copy all of the files in my current directory but skip Downloads and Videos to another system.
tar --exclude ./Downloads --exclude ./Videos -czf - . | ssh backup-host tar xvf - -C /tmp/$USER
The above copies all of the files but skips two directories that have large files that I don’t need backed up
(Q) How can I copy a directory from remote:/local/apps/ to my system?
The following will work
rsync -avz remote-host:/local/apps/ /local/apps/
A more comprehensive copy is shown next. It is more useful if you’re copying system level files which might have ACLs on them, or if it contains hard links that need to be maintained.
rsync -aHAXvz host-remote:/local/dir1/ /local/dir/
For rsync -aHAXvz:
-a — archive mode; preserves most normal file properties and copies recursively
-H — preserve hard links
-A — preserve ACLs
-X — preserve extended attributes
-v — verbose output
-z — compress data during transfer