{"id":329,"date":"2005-09-21T15:40:32","date_gmt":"2005-09-21T22:40:32","guid":{"rendered":"http:\/\/lee.org\/blog\/?p=329"},"modified":"2011-11-09T16:38:42","modified_gmt":"2011-11-10T00:38:42","slug":"using-cygwin-rsync-ssh-and-the-internet-to-backup-my-xp-computer","status":"publish","type":"post","link":"https:\/\/www.lee.org\/blog\/2005\/09\/21\/using-cygwin-rsync-ssh-and-the-internet-to-backup-my-xp-computer\/","title":{"rendered":"Using Cygwin, Rsync, SSH and the internet to backup my XP computer"},"content":{"rendered":"

update 11-9-11:<\/strong> consider using DeltaCopy<\/a> instead of my home-brewed method below. I haven’t used it yet, but it looks nice!<\/p>\n

I want to back up my Windows XP computer to another Windows XP computer using Rsync and SSH. Since Rsync only runs under *nix, I’m running it under Cygwin. <\/p>\n

There are a couple hurdles to doing this. The one I got stuck on was getting SSH to work without a password. <\/p>\n

It’s so freaking simple to do. You just have to know which instructions do not help accomplish your goal.<\/p>\n

<\/p>\n

First, ignore these instructions from pigtail.net<\/a>. They don’t work. Ignore these instructions from Berkeley.edu<\/a>. They don’t work. Follow these instructions from Freebsd.org<\/a>. They work. Thank you Mike!!<\/p>\n

\n1) Generate your keys on your local machine:
\n\t– ssh-keygen -t rsa
\n2) When prompted for a password, do not enter one. This will generate a password-less key called id_rsa, and a public key id_rsa.pub
\n3) Copy the id_rsa.pub key over to the machine you want to ssh to. NOTE: change the filename first or you may overwrite your existing RSA key for your remote host. Like this:
\n\t– local: cp id_rsa.pub local.key
\n\t– scp local.key remote:
\n\t– cat local.key >> .ssh\/authorized_keys<\/p>\n

At this point, your remote machine should accept a passwordless login from that “local” machine.<\/p>\n

Mike
\nMichael K. Smith NoaNet\n<\/p><\/blockquote>\n

\n

Here’s the rest of it….<\/p>\n

Let’s say you want to back up your computer… The most protected thing you could do would be to make encrypted offsite backups on a regular or even constant basis. But toting around offsite backups are a real bother.<\/p>\n

The best (inexpensive and easy to maintain) system I’ve come up with is thus:<\/p>\n

Two WinXP computers over high speed internet. Both computers have Cygwin (installed: ssh, sshd, rsync) and Truecrypt. Every so often, I rsync a backup to the Truecrypted file system. A full 50 gigabyte backup takes about 2 weeks of continuous file transfer (at 50KB\/sec) but subsequent backups take a few hours overnight. It’s better if the initial backup is done locally but what the hey, it’s only bandwidth.<\/p>\n

I’ve only recently gotten the system going so I’ll keep you informed as to how it’s working for me.<\/p>\n

To protect the backup machine, I use Truecrypt<\/a>. I also use “Windows Tweakui | My Computer | Drives” to hide the Truecrypted partition on the backup machine. The best I could figure for using Truecrypt is to VNC into the backup machine and decrypt the partition before running a backup process. I’ve discussed this topic previously in my blog. Go searching<\/a>.<\/p>\n

update 9-22-05<\/strong>
\nI had to restart the rsync session several times because the following error would stall things<\/p>\n

…Lullaby.mp3
\nReceived disconnect from [IP removed]: 2: Corrupted MAC on input.<\/p><\/blockquote>\n

I googled a little and found some folks on Redhat.com or someplace like that saying how they don’t have a fix for it but believe it to be a bug in a Linksys router. My Netgear router has the problem as well so phoey. <\/p>\n

4.4 gig moved in 20 hours. Yup. It looks like the initial backup will be a 2 week affair. The 75 KB\/sec transfer isn’t hindering my normal system usage which is a good thing. When I move files over my local network with rsync at about 1.5 MB\/sec, both computers are virtually unusable during the transfer. (update<\/strong>, there’s a fix for that. Check the comments)<\/p>\n

Stopping and restarting the rsync is solid proof that I’m on the right track. It takes 30-45 seconds for the computers to sift through the first 4.4 gigabytes for changed files and then resume the transfer.<\/p>\n

update 9-23-05<\/strong> I had some trouble trying to install cron (so I could restart failed rsync processes) so I reinstalled Cygwin from scratch. I observe 2 things: 1) Reinstalling this entire pseudo operating system was a SNAP 2)I haven’t had the failure in several hours even though I didn’t do anything.<\/p>\n

update 9-24-05<\/strong>I installed cron and I run this every hour to restart everything:<\/p>\n

\n#!\/bin\/bash<\/p>\n

# kill all of the rsync processes (actually, shoot all the tcsh processes… they are running rsyncs. Then kill the rsync processes<\/p>\n

ps | grep tcsh | awk ‘{print $1}’ | xargs kill -9
\nps | grep rsync | awk ‘{print $1}’ | xargs kill -9
\nps | grep ssh | awk ‘{print $1}’ | xargs kill -9<\/p>\n

# start a backup or two
\n\/home\/lee\/backup.txt >> \/home\/lee\/backup-log.txt &
\n#\/home\/lee\/backup2.txt >> \/home\/lee\/backup2-log.txt &\n<\/p><\/blockquote>\n

But grrr. After a restart, the file it was working on is restarted from the beginning. That means if it takes more than 1 hour to move an individual file (at my speed, about a 0.25 gig file), my cron job will spin around in circles forever… …<\/p>\n

Rsync Options to the rescue! “–partial-dir=.rsync-partial” fixes that problem.<\/p>\n

Here is the script file I run (I’ve stripped out some of the boring folders that I backup):<\/p>\n

\n#!\/bin\/csh<\/p>\n

echo “———— Begin Rsync ———— ”
\necho -n “start on: ” ; date<\/p>\n

# notes:
\n# If you want to push the output to a file, you might do something like this:
\n# .\/backup-booty.txt > booty-log.txt 2>&1
\n#
\n# This is being run with Cygwin on WinXP on both sides. Several folders initially
\n# (for some unknown reason) had default permissions of 000.
\n# This caused the first rsync to work and subsequent transfers to crash due to file permission errors.
\n# To fix this, I used Cygwin to set file permissions on the client-side to 700.
\n#
\n# If the file transfer gets mucked up and the files on the server won’t delete,
\n# run a ‘chmod -R 777 *’ on them
\n#
\n# It might be useful to run it with the output to a file for analysis later.
\n# IE: backup-booty.txt > bootylog.txt<\/p>\n

setenv Rsync “nice +1 rsync”
\nsetenv OPTS “–verbose –partial-dir=.rsync-partial –compress –recursive –times –delete –rsh=\/usr\/bin\/ssh”<\/p>\n

# useful options:
\n# –bwlimit=50 limit bandwidth to 50KB\/sec
\n# –progress: good for interactive sessions, bad for logged sessions
\n# –verbose –verbose –verbose: shows exactly what’s going on. You might want to pipe the output to a file.
\n# –compress: good for slow connections, bad for high speed
\n# –whole-file: good for fast connections. It won’t use the rsync algorythm so it won’t bog down the CPU.<\/p>\n

# I don’t run with “-a” because it could be good to strip all that permission stuff when restoring the backup<\/p>\n

setenv BackupDestination “Owner@mydomain.com:\/cygdrive\/f\/”
\n#setenv BackupDestination “\/cygdrive\/f\/”
\n#setenv BackupDestination “cg:\/cygdrive\/d\/”<\/p>\n

setenv BackupFolder “current-backup”<\/p>\n

echo “rsyncing to: $BackupFolder and $BackupFolderD”
\necho “Options specified: $Rsync, $OPTS”<\/p>\n

# ———————————————————————<\/p>\n

echo “Set up backup folder”
\ndate
\n# … by dropping this very program into the folder. We do this because rsync can’t create sub-sub-folders blindly
\n$Rsync $OPTS $0 \\
\n\t$BackupDestination\/$BackupFolder\/<\/p>\n

#echo “Outlook”
\ndate
\n$Rsync $OPTS \/cygdrive\/c\/Documents\\ and\\ Settings\/Lee\/Local\\ Settings\/Application\\ Data\/Microsoft\/Outlook* \\
\n\t$BackupDestination$BackupFolder<\/p>\n

echo “My Documents sans My Music and My Videos”
\ndate
\n$Rsync $OPTS –exclude=”My Music” –exclude=”My Videos” \/cygdrive\/c\/Documents\\ and\\ Settings\/Me\/My\\ Documents* \\
\n\t$BackupDestination$BackupFolder<\/p>\n

echo “Firefox bookmarks”
\ndate
\n$Rsync $OPTS \/cygdrive\/c\/Documents\\ and\\ Settings\/Lee\/Application\\ Data\/Mozilla\/Firefox\/Profiles\/41e9243n.default\/bookmarks.html \\
\n\t$BackupDestination$BackupFolder\/Firefox-Bookmarks\/ <\/p>\n

echo “Firefox extentions”
\ndate
\n$Rsync $OPTS \/cygdrive\/c\/Documents\\ and\\ Settings\/Lee\/Application\\ Data\/Mozilla\/Firefox\/Profiles\/41e9243n.default\/extensions* \\
\n\t$BackupDestination$BackupFolder\/Firefox-Extentions\/ <\/p>\n

echo “Start Menu”
\ndate
\n$Rsync $OPTS \/cygdrive\/c\/Documents\\ and\\ Settings\/Lee\/Start\\ Menu* \\
\n\t$BackupDestination$BackupFolder<\/p>\n

echo -n “— finished rsync on ” ; date\n<\/p><\/blockquote>\n

update 9-25-05 8pm<\/strong> 19 gig moved. I get about 5 GB\/day when it runs continuously… my estimates were correct :-)<\/p>\n

update 9-26-05 9am<\/strong> 22 gig moved. If I run 2 rsync sessions at once, I get 6 GB\/day. The computer lags a bit though. 3 sessions doesn’t improve performance. It takes about 60 seconds for Rsync to power-up so I changed the cron job to restart the download every 2 hours instead of every hour. “* *\/2 * * * \/home\/lee\/restart-backup-cron.txt”<\/p>\n

update 9-29-05<\/strong> 12:26am, enabled –compression. 33.4 GB on the remote machine. . . . . 8:30pm 9-29-05, 37.9 GB… thats 5.4 GB per day. A slight improvement over the previous 5GB\/day. I keep most of my files compressed as jpg, mp3, avi, zip, etc…<\/p>\n

I think it might be that the first time a folder is checked, it bogs down the CPU, subsequent checks are 100 times faster and don’t bog down the machine. <\/p>\n

update 10-3-05 12:00am<\/strong> 53.7 GB.. actually, I’ve been PCAnywhere-ing into the machine and getting that read with Windows Explorer. When I SSH in, ‘df’ says 56.6 GB. Bigger numbers are better numbers so there. I set the cron to restart every hour instead of every 2.<\/p>\n

update 10-7-05<\/strong>: To make cron run as a service and start automatically when I start my computer (so I can run the cron jobs that start the rsyncs that swallow the flies) I opened a Cygwin shell prompt and typed this<\/p>\n

cygrunsrv -I cron -p \/usr\/sbin\/cron -a -D<\/p><\/blockquote>\n

It makes it an automatically starting Service. :-( (Thanks<\/a>)<\/p>\n

10-8-05 update<\/strong> after 15 days and 77 GB, I have a backup in New Jersey. When doing updates, if there are no changes, Rsync verifies this fact in less than 4 minutes. That is astoundingly phenominally wonderfully fast. All hail the Rsync. And good night!<\/p>\n","protected":false},"excerpt":{"rendered":"

Using Cygwin, Rsync, and SSH to backup my computer<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-329","post","type-post","status-publish","format-standard","hentry","category-geekery"],"_links":{"self":[{"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/posts\/329","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/comments?post=329"}],"version-history":[{"count":0,"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/posts\/329\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/media?parent=329"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/categories?post=329"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/tags?post=329"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}