IntegrityChecker java (icj): sync command [version 3.0]

The sync command syncs stored icj hash info as extended attributes with the name user.diglloyd.icj.HashInfo#S and user.diglloyd.icj.ID#N. These attributes are in addition to the regular icj hash information.

Operating system differences: macOS (only) makes use of the suffixes as follows:
  #N indicates "do not preserven when copying"
  #S is a hint for "preserve when copying"

The sync command uses existing hash information without performing hashing. Therefore, update prior to a sync, because only current hash info will be synced.

The update command care of all syncing issues as a 2nd phase after the hashing phase, so it is never necessary to sync after an update.

Backup programs and attributes

Most backup programs also backup extended attributes (unless the user has disabled that feature). However, if a file has not changed, but its attribute(s) have changed, a backup programs might not update attributes for those files on the backup. Unless the file itself has changed, and it therefore is backed-up again along with its attributes.

To ensure that backups previously- made contain up-to-date attributes, run sync on the backup to be sure. All files that lack attributes or have stale ones will have the current attribute added without any hashing needed.

Usage (general)

Unless this is a backup scenario (operating on a backup, see discussion above), run update prior.

Any combination of volumes or folders can be specified in a single invocation.

Files with stale hash info will not have the attribute added or updated; icj update required. Only files using the current SHA512 hashing algorithm will have the attribute added (do an update first, which upgrades the hashes). Files with conflicting icj/icjh/attribute info will be flagged and not updated; use update which can make the proper determination of authoritative info (by hashing the current file).

The com.diglloyd.icj.HashInfo attribute ends with #S, which instructs programs to maintain it across backups and copies.

About extended attributes

Extended attributes (xattr) are widely used in macOS and also on Linux. They are metadata separate from the file contents.

IntegrityChecker Java (icj) can store its hash info the extended attribute com.diglloyd.icj.HashInfo#S (in addition to its icjh hierarchy files). While extended attributes are redundant, they travel with the file in most all cases, and are thus useful if files or folders are moved around and their hash info “lost” relative to the icj hierarchy file. Attributes also make it possible to locate missing files in new locations.

Caveats: adding extended attributes may cause Time Machine to backup files all over again (TBD), even though the attribute is metadata and not stored as part of the file. Similarly but in the opposite sense, cloning may fail to copy extended attributes added after the initial clone, if the file has not otherwise changed.

Extended attributes are suitable only as a fallback/adjunct mechanism to icj hierarchy files. Limitations of extended attributes:

Locked and read-only files cannot have the attribute added.
Files that are otherwise readable and thus hashable by icj might not be writable and thus cannot support attributes.
Some file systems do not support extended attributes. However, the Windows file system ExFat when used on macOS *does* support attributes, even if Windows does not (Windows has no apparent support extended attributes as do macOS and Linux).
Some backup programs might not copy extended attributes or offer an option to not copy them. (macOS Finder and programs like Carbon Copy Cloner do copy them by default).
Windows systems lack extended attributes, or at least lack them in any way icj can manipulate them.
Saving a file might wipe-out its attributes.

IntegrityChecker java is a comprehensive data integrity solution, because it addresses all these issues by having BOTH attributes on files as well as hierarchy files.

Command line usage

icj sync [<path>]*

--save=prompt|yes|no

# --sync option applies only to user.diglloyd.icj.HashInfo#S attribute. user.diglloyd.icj.ID#N are always synced.
--sync=existing|missing|all

Examples

Lines starting with "#" are comments.

# Sync attributes for all files in current directory, recursively
icj sync --sync=all

# Sync existing attributes in files in current directory, recursively
icj sync

# Sync existing attributes for files in current directory, recursively, prompt for whether to save after listing info
icj sync --save=prompt

# Sync existing attributes for files in Folder1, Folder2, Folder3, recursively
icj sync Folder1 Folder2 Folder3

# Sync attributes for all files in Folder1, Folder2, Folder3, recursively
icj sync --sync=all Folder1 Folder2 Folder3

Previous page: icj clean

Next page: icj empty

General Topics