Cross-platform Integrity Checker (Java version) — icj
IntegrityChecker java (icj)is a cross-platform (Mac, Windows, Linux, NAS, etc) file data integrity validation program. Implemented in Java, it can prove, regardless of operating system or storage medium, whether your files are intact and undamaged.
So-called “bit rot” is rare but there are so many things that can cause problems. Why not be sure (prove!) that everything is OK? Stuff happens.
- Peace of mind knowing that your data is absolutely bit-for-bit good.
- Save time—runs fast as the storage allows.
- Flags changes that might otherwise go un-noticed from “bit rot” or software errors or malicious software or your own mistakes (!), or storage or connectivity issues, or nasty stuff like ransomware or malware.
- Know as a hard fact when your files are intact-detects even a single bit change no matter how many files or how large.
- Platform-neutral—works on Mac, Windows, Linux, NAS, etc. Change platforms? No problem!
- Backup validation, so that an 10th or 100th generation copy can be validated just as easily as the original.
- Archival integrity—ensure that critical archive data has zero issues.
How it works
The IntegrityChecker engine relies on the SHA-512 cryptographic hash for each file. Essentially this means that the odds of not detecting even a single-bit change are about as likely as you winning the lottery a million times in a row. IntegrityChecker stores these hashes for every file as reference values.
Even single-bit errors are detected, anywhere in a file. A single-bit error might seem trivial, but in can be a “canary in the coal mine” indicating impending “data doom” for much more.
The process is simple:
IntegrityChecker is the perfect tool when changing systems or moving data (backing up, switching systems, etc), because you can verify that all data has 100% fidelity to the original copy, and that ALL of it is there. That cannot be take for granted, given numerous bugs in things like macOS Finder, which has serious bugs when copying files.
This validation can be done a day, a week or a month or a year later, an important consideration if a system meltdown requires restoring all your data.*
It doesn’t matter if the data is on a hard drive, SSD, BluRay or NAS, the only prerequisite is to 'icj verify' is that 'icj update' has been done before/
Example scenario — swapping hard drives
Workflow challenge: you want to upgrade a hard drive to a larger one. This scenario is the same as transferring files between machines or anything similar.
Follow this process:
- Update the original/older volume / drive.
- Make backups of the original; verify the backups (to be certain they are perfect copies).
- Remove the old drive (set it aside for safekeeping), install the new drive, transfer the backup to the new drive.
- Verify the new volume / drive.
Any similar scenario works the same way, and is in fact even more flexible: partial copies are fine; all that matter is that any folder to be verified has its hidden “.icj” file in place from a previous update. So it doesn’t matter if that folder is a 2nd or 3rd or even 10th generation copy of the original.
Commands
IntegrityChecker™ offers the following commands:
- verify — verify files for integrity, flagging those changes by date or size, and especially those that have changed content;
- update — update files that are new or have changed date or size;
- update-all — update all files, whether or not they already have validation info;
- update-new — update new files (only);
- sync — syncs file attributes
- status — quickly show status eg new files and/or size/date change (does not validate contents);
- dupes — find duplicates bases on hashes, options to remove or clone duplicates to save storage space
- info — shows general information about files
- compare — compare two folders, based on hashes (very fast)
- clean — remove “.ic” validation files;
- matches — shows file globbing matches
- pref — shows and opens preferences
- sha — test hashing speed
FAQ
IntegrityChecker validates only those files that were previously hashed using update or a variant thereof; it cannot validate new files that were never before processed (no validation info).
Is it hard to use?
Easy to use: all you have to do is type a few chararacters. Entire volumes or folder(s) can be dealt with easily with the volume name, such as:
icj update MyData
icj verify MyData
Why is it better than a backup program that checks files?
Validate any time — IntegrityChecker can check any backup containing the validation info at any time. No originals needed for comparison—any backup of duplicate can be verified. This is incredibly important if a main drive fails, or transferring to a new machine, etc.
Any media — works on any media, including hard drives, SSDs, CD or DVD or flash drive. Since the validation information is carried in the same folder as the files themselves, it “goes along for the ride”, to check today, or a year from today so long as each of those top-level transferred folders has had an icj update. See Using Hierarchy Hash Files vs Per-Folder Hash Files.
Ultra-efficient — IntegrityChecker also runs as fast as your drives can go, so validating your files is not a tedious experience.
Future validation —backups made far in the past can be validated without any need for original files for comparison.
Backup validation
Before making a backup, run icj on all the folders to be backed-up with update. When the backup is complete, run IntegrityChecker on the backup to verify that all files are 100% accurate. You can repeat this process a day or a month or a year later to validate older backups, including those on CD or DVD. Since validation reads all the files, it is also a good “sanity check” on the health of the drive or CD or DVD itself.
Extending this procedure, a clone of a clone of a backup can be validated, which can prove that all copies (at least when made) are all 100% intact, bit-for-bit.
When restoring from a backup, you can verify the backup first, or simply restore your files and then verify the restored files. In this manner, you can know for certain that the files you’ve restored are perfect.
Data transfer
Data transfer is essentially the same as backup, but it’s about moving files to another computer, to another drive, etc. Follow the same process as for backup, and be assured that your files are perfect after the transfer.
What’s changed on the system
Especially with multiple users and/or large data sets, you might want to see which files have changed. Changes include not just legitimately-changed files, but insidious corruption by software or flaky hardware or malware, or a CD or DVD that is damaged.
Malware heads-up
Malware and ransomware can damage or encrypt your files. IntegrityChecker will flag all files that are different and will barf if its own files are damaged.
Copyright © 2022 diglloyd Inc, all rights reserved