{"id":9603,"date":"2022-08-27T17:09:22","date_gmt":"2022-08-28T00:09:22","guid":{"rendered":"https:\/\/www.lee.org\/blog\/?p=9603"},"modified":"2022-09-03T11:01:48","modified_gmt":"2022-09-03T18:01:48","slug":"rclone-is-a-great-way-to-move-files","status":"publish","type":"post","link":"https:\/\/www.lee.org\/blog\/2022\/08\/27\/rclone-is-a-great-way-to-move-files\/","title":{"rendered":"Rclone is a Great Way to Move Files"},"content":{"rendered":"

Rclone<\/a> \"\"<\/a>is the right software to copy multi-terabyte file repositories from one place to another! It’s also very capable at rearranging files in lots of other different ways. Check it out. It’s something of a sister program to rsync on unix.<\/p>\n

I had to fiddle with rclone a bit. Here is the command I ended up using:<\/p>\n

c:\\aaa\\rclone\\rclone sync –modify-window=2s -v –exclude=”System Volume Information\/**” d:\/ f:\/[my new repository]<\/p><\/blockquote>\n

It was so simple (after a two weeks of fiddling and yelling at my computer). Just run that command, wait 3 days (yipe, I’ve got some bandwidth\/scale problems) and profit!<\/p>\n

(If you copy-paste that text, make sure you don’t use smartquotes, use regular quotes)<\/p>\n

Let’s look at that command one argument at a time:<\/p>\n

c:\\aaa\\rclone\\rclone<\/p><\/blockquote>\n

rclone isn’t installed, perse, on a windows computer. You just unzip it and go! That means it doesn’t set PATH variables (unless you do it yourself. So I open up a Windows Powershell with Admin rights, specify the folder to run in (c:\\aaa\\rclone in my example) and go!<\/p>\n

sync<\/p><\/blockquote>\n

It’s so magically simple. Sync is better than copying because it will delete extra files on the target side.<\/p>\n

–modify-window=2s<\/p><\/blockquote>\n

The FAT file system doesn’t accurately capture file modification times. So every time you run rclone, it’ll compare file modification times and get it wrong. If you turn on very verbose mode with “-vv” you’ll see errors like “Modification times differ by 1.490000000s”, and it’ll copy files over and over and over :-(. This “modify-window” setting says “if the files were modified within 2 seconds of each other, don’t sweat the difference.<\/p>\n

-v<\/p><\/blockquote>\n

“-v” is “verbose” mode, “-vv” is very verbose mode, for when you want to figure out why your rclone session is failing. You can eliminate it when you are sure of your sessions.<\/p>\n

 <\/p>\n

–exclude=”System Volume Information\/**”<\/p><\/blockquote>\n

I’m copying from the root (d:\\) If I don’t specify this, rclone will try (and fail) to grab files from the system volume information folder.<\/p>\n

d:\/ f:\/[my new repository]<\/p><\/blockquote>\n

Note that I’m using slashes instead of Window’s backslashes. It might not be essential but rclone (and I) prefers slashes.<\/p>\n

————————————————–<\/p>\n

What doesn’t work:<\/p>\n

Windows File Explorer<\/strong> drag-and-drop copy doesn’t have enough checking, the ability to recover a failed copy, or the ability to “sync”.<\/p>\n

Teracopy<\/strong> can copy but not sync. It recovers a bit better from failed copies but not well enough for my 2TB file moves. I think it created system instability, crashing a few times at inopportune times.<\/p>\n

Robocopy<\/strong> not enough checking, mediocre recovery from failed copies, not enough feedback.<\/p>\n

Windows File History<\/a><\/strong> might actually be useful as a local backup tool. I played with it a bit and seemed to work well on a local network. Though it kept nagging me when a computer wasn’t connected (yes, I know, I turned the other computer off an hour ago). Though I’ll likely stick with rclone and Backblaze instead of learning 2 tools.<\/p>\n

Minio<\/a><\/strong> on a server with some fancy buy-once Amazon AWS compatible backup software for the client like Arqbackup<\/a> was a good idea until I found that Minio needed to be manually updated every couple weeks or it would refuse to start up.<\/p>\n

keywords: backup<\/p>\n","protected":false},"excerpt":{"rendered":"

Rclone is the right software to copy multi-terabyte file repositories from one place to another! It’s also very capable at rearranging files in lots of other different ways. Check it out. It’s something of a sister program to rsync on unix. I had to fiddle with rclone a bit. Here is the command I ended […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,14],"tags":[],"class_list":["post-9603","post","type-post","status-publish","format-standard","hentry","category-geekery","category-product-recommendations"],"_links":{"self":[{"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/posts\/9603","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/comments?post=9603"}],"version-history":[{"count":10,"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/posts\/9603\/revisions"}],"predecessor-version":[{"id":9621,"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/posts\/9603\/revisions\/9621"}],"wp:attachment":[{"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/media?parent=9603"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/categories?post=9603"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lee.org\/blog\/wp-json\/wp\/v2\/tags?post=9603"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}