Issues with copying large directories
- SCP is VERY slow with large numbers of files
- Data bandwith can be compressed to speed data transfer over slower networks
- Maintaining file permissions
- Not enough disk space to create a backup (2nd copy of the data)
Solutions
(Q) How can I make a backup of my home directory?
The following shows how to make a backup and copy the data to another system (without creating a 2nd copy of the data). This is valuable if your disk doesn’t have enough storage for a 2nd copy of the data being backed up.
Copy all of the files in my current directory but skip Downloads and Videos to another system.
tar --exclude ./Downloads --exclude ./Videos -czf - . | ssh backup-host tar xvf - -C /tmp/$USER
The above copies all of the files but skips two directories that have large files that I don’t need backed up