Copy paste files?

[demarpaint]

I'm curious. I back up files between a few different computers. For the sake of argument I copy and paste 27,000 files in 32 folders with sub folders onto a USB stick. I paste them into another PC and end up with more or less files in the total count, but everything appears to be there. Then I'll do it from the same USB stick onto another computer and the number changes again along with the GBs. I am certain I am copying and pasting everything. Can someone explain what is going on? Thanks

 

[HangFire]

Each hard drive/thumb drive/SSD has a hardware defined block size, usually 512 bytes or 4K. When formatted into a file system, space on the disk is allocated via low-level software into clusters, where each cluster is made up of a fixed number of blocks. Then, each file is written into as many clusters as it needs.

Each (non-empty) file fits into at least one cluster (a small file) or more. Unless it randomly just happens to be a perfect fit, each file has some wasted empty space at the end of it in the last sector it uses. On disks with large cluster sizes, the average wasted space at the end of each file is larger. That accounts for the difference in the "used space" you see.

While this is "waste" it's not necessarily a bad thing. If the file is appended to with a small amount of data (very common on computers), or re-written slightly larger, the space at the end will probably be used without going through the process of finding and linking another free sector. This makes it faster to write, and writing to disk is the second slowest thing a computer does. (The first is waiting for the user.)

Large disks with certain filesystems tend to have large sectors, and therefore the wasted space in the last sector tends to be larger.

Windows uses a variety of filesystems, depending primarily on the size of the disk and how old the Windows is. More gory detail:

https://support.microsoft.com/en-us/help/140365/default-cluster-size-for-ntfs-fat-and-exfat

 

[HangFire]

OK That didn't answer your original question. (Why is the number of files different? Not space used.)

I suspect it is meta-data. Windows tends to create hidden index files for photographs, user preference files for folders, and other files that can customize or speed-up later access. However not all of these meta-data files are supported on non-boot, removable drives or certain filesystems.

 

[demarpaint]

Thanks HangFire for the explanation. I figured it had something to do with space allocation, it was the file count itself that was confusing me. Hidden files, etc. makes sense.

I was pasting files from a Linux system to Win 10 using NTFS format on the USB stick, and many of the files weren't showing on the Win 10 machine. I formatted the USB stick to FAT32 and I was able to quickly resolve that issue.

 

[StevieC]

Open a command prompt and use X-Copy with the verification switch. I do this when there is multiple directories and files and I can't verify each one but want to be sure it copied everything.

Example of what you would type...

xcopy c:\myfiles\*.* z:\folder /E /V /Y

(E= Copies all subdirectories including empty ones, V= Verify each file written, Y= Don't ask me to confirm)

 

[Garak]

That's on Windows, I take it, demarpaint? What are you using to back up? Something like tar grabs all the hidden files, but I haven't paid a lot of attention to file counts. How things are backed up or archived is important, given the way different programs handle files (and preserve permissions) in different ways. If I want to get something to someone in a [vain] hope of them being able to use it, I'll stick to zip, or rar, or 7z, depending on the level of simplicity versus security I'm looking for. If it's actual backing up, it's tar with some compression after.

 

[demarpaint]

Originally Posted by Garak
That's on Windows, I take it, demarpaint? What are you using to back up? Something like tar grabs all the hidden files, but I haven't paid a lot of attention to file counts. How things are backed up or archived is important, given the way different programs handle files (and preserve permissions) in different ways. If I want to get something to someone in a [vain] hope of them being able to use it, I'll stick to zip, or rar, or 7z, depending on the level of simplicity versus security I'm looking for. If it's actual backing up, it's tar with some compression after.

My towers have three HDD's in them. An SSD and two spindle drives, all data is saved on one of the spindle drives and then copied and pasted to the other drive. Then any files that I create or edit during the day are copied and pasted to a thumb drive and transferred to my other tower and laptop computers once or twice a week. Occasionally I will delete all the documents from my wifes tower, and paste a fresh copy of the documents onto her machine or one of the laptops that wasn't used in a while, just so I am certain nothing is missed. That's when I notice the issue. I run Linux Mint on some machines and Win 10 on others, and the count can differ even going from Windows to Windows, Linux to Linux, or Windows to Linux. But I have never found a file missing, or corrupt data.

On my newest tower I also have a copy of all my data created from Win 10 back up. And on the Linux machines a backup created with Lucky Backup. HDD space is not an issue.

My method might sound insane, but It works. I was only questioning the file count and why it can vary, yet everything is there and in order on the copies. I hope this makes some sense.........Thanks for all the replies.

Edit: If I backup my music, which isn't often, it is on three of my computers and external drives, the file/GB count can be crazy. I have close to 350GB of music.

 

[Garak]

Yes, for what you describe, and many purposes, that works just fine. I have done copy and paste for a lot of home directory contents on Linux, like things in a documents folder. If I want to go more in depth with the home folder, and don't want to miss the Firefox settings, for instance, then I make sure I use tar, or specifically grab the appropriate hidden directory.

 

[demarpaint]

Originally Posted by Garak
Yes, for what you describe, and many purposes, that works just fine. I have done copy and paste for a lot of home directory contents on Linux, like things in a documents folder. If I want to go more in depth with the home folder, and don't want to miss the Firefox settings, for instance, then I make sure I use tar, or specifically grab the appropriate hidden directory.

Thanks! All I'm after is backing up documents, pictures, and occasionally music. I have the system images backed up, as virgin, and after I have everything tweaked and working.

 

[Garak]

That's pretty much what I do. I get it set up the way I like, then tarball the system. My documents are generally used cross platform, so I'll use rar or something to back them up as needed, or copy and paste them to a thumb drive if I'm going back and forth, and for backup. I don't keep a lot of music on the computer.