synctool by Walter de Jong (c) 2003-2010 synctool COMES WITH NO WARRANTY. synctool IS FREE SOFTWARE. synctool is distributed under terms described in the GNU General Public License. See the README for general information. See the INSTALL file for information on how to deploy synctool. This DOCUMENTATION file tries to explain using synctool. In 'synctool terminology', a node is a host, a computer in a group of computers. A group of computers is called a cluster. Once you've installed and setup synctool, you will have a 'masterdir' repository on the master node of your cluster. This masterdir is configured in the synctool.conf file. If you don't know what I'm talking about here, read the INSTALL file carefully. The masterdir is usually set to /var/lib/synctool synctool is usually run on the master node. It will use rsync to mirror the masterdir to all the nodes in the cluster. This has the advantage that you don't need a shared filesystem to run synctool, and it is fast too. Note that it is perfectly possible to run synctool.py "stand alone" on a node, in which case it will check it's local copy of the repository. It will not synchronize with the master repository (it's all server push, NOT client pull). The main power of synctool is the fact that you can define logical groups, and you can add these to a filename as a filename extension. This will result in the file being copied, only if the node belongs to the same group. The groups a node is in, is defined in the synctool.conf file. In the configuration file, the nodename is associated with one or more groups. The nodename itself can also be used as a group to indicate that a file belongs to that node. In the synctool masterdir there are 4 subdirectories, each having its own function: * overlay/ * delete/ * scripts/ * tasks/ The overlay/ tree contains files that have to be copied. When synctool detects a difference between a file on the system and a file in the overlay tree, the file will be copied from the overlay tree onto the system. The delete/ tree contains files that always have to be deleted from the system. Only the filename matters, so it is OK if the files in this tree are only 0 bytes in size. The executables in the scripts/ directory are executables that synctool can run when needed. By means of the on_update directive in the synctool.conf file, a designated script may be executed when a certain file is changed. For example: when /etc/inetd.conf is updated, the script 'hupdaemon.sh inetd' must be run. The executables in de tasks/ directory are run when synctool is invoked with the '-t' or '--tasks' argument. This makes it possible to run scripts on hosts, which is very convenient for doing change management. Example run: root@masternode:/# synctool -qf node3: /etc/xinetd.d/identd updated (file size mismatch) The file is being updated because there's a mismatch in the file size. Should the file size be the same, synctool will calculate an MD5 checksum to see whether the file was changed or not. By default synctool does a DRY RUN. It will not do anything but show what would happen if this would not be a dry run. Use -f or --fix to apply any changes. Now, I want the xinetd to be automatically reloaded after I change the identd file. There are two ways to do this in synctool. 1. Old, classic way, the way it was done in synctool version <= 3.0 Put in synctool.conf: on_update /etc/xinetd.d/identd /etc/init.d/xinetd reload 2. Modern way, synctool version >= 4.0, much easier: Put in file $masterdir/overlay/etc/xinetd.d/identd.post : /etc/init.d/xinetd reload The .post script will be run when the file changes: root@masternode:/# synctool -qf node3: /etc/xinetd.d/identd updated (file size mismatch) node3: running command $masterdir/overlay/etc/xinetd.d/identd.post It is possible to put a group extension on the .post script, so that you can have one group of nodes perform different actions than others. The example for /etc/xinetd.d is interesting because you can also put a "on_update" trigger or .post script on the directory itself. Whenever a file in the directory gets modified, the trigger will be called. So, we can simplify the situation for /etc/xinetd.d to: on_update /etc/xinetd.d /etc/init.d/xinetd reload or $masterdir/overlay/etc/xinetd.d.post: /etc/init.d/xinetd reload The next example shows that the nodename can be used as a group. In the example, the 'fstab' file is identical throughout the cluster, with the exception of node5 and node7. root@masternode:/# ls -F /var/lib/synctool/overlay/etc ... fstab._all motd.production._batch sudoers._all fstab._node5 nsswitch.conf._all sysconfig/ fstab._node7 nsswitch.conf.old._all sysctl.conf._all ... The group 'all' implictly applies to all nodes. Likewise, there is an implicit group 'none' that applies to no nodes. Group 'none' can be convenient when you to have a copy of a file around, but do not wish to push it to any nodes (yet). The '-v' option gives verbose output. This is another way of displaying the logic that synctool performs: # synctool -v ... node3: checking $masterdir/overlay/etc/tcpd_banner.production._all node3: overridden by $masterdir/overlay/etc/tcpd_banner.production._batch node3: checking $masterdir/overlay/etc/issue.net.production._all node3: checking $masterdir/overlay/etc/syslog.conf._all node3: checking $masterdir/overlay/etc/issue.production._all node3: checking $masterdir/overlay/etc/modules.conf._all node3: checking $masterdir/overlay/etc/hosts.allow.production._interactive node3: skipping $masterdir/overlay/etc/hosts.allow.production._interactive, it is not one of my groups ... The '-q' option of synctool gives less output. This is my favorite option. root@masternode:/# synctool -q node3: /etc/xinetd.d/identd updated (file size mismatch) node3: running command /bin/rm /etc/xinetd.d/*.saved ; /etc/init.d/xinetd reload If '-q' still gives too much output, because you have many nodes in your cluster, it is possible to specify '-a' to condense (aggregate) output. The condensed output groups together output that is the same for many nodes. synctool -qa is one of my favorite commands. synctool does this by piping the output through the synctool-aggr command. You may also use this to condense output from dsh, for example # dsh date | synctool-aggr or just # dsh -a date The '-f' or '--fix' option applies all changes. Always be sure to run synctool at least once as a dry run! (without -f). Mind that synctool does not lock the repository and does not guard against concurrent use by multiple sysadmins at once. In practice, this hardly ever led to any problems. To update only a single file rather than all files, use the option '--single' or '-1'. If you want to check what file synctool is using for a given destination file, use the '-r' or '--ref' option: root@masternode:/# synctool -q -n node1 -r /etc/resolv.conf node1: /etc/resolv.conf._somegroup To inspect differences between the master copy and the client copy of a file, use '--diff' or '-d'. synctool can be run on a subset of nodes, a group, or even on individual nodes using the options '--group' or '-g', '--node' or '-n', '--exclude' or '-x', and '--exclude-group' or '-X'. This also works for dsh and dcp. For example: # synctool -g batch,sched -X rack8 Another example: # dsh -n node1,node2,node3 date or copy a file to these three nodes: # dcp -n node1,node2,node3 -d /tmp patchfile-1.0.tar.gz You may also wish to pull a file from a node into the repository. You can do this from the masternode like this: # synctool -n node1 --upload /path/to/file It may be desirable to give the file a different group extension than the default proposed by synctool: # synctool -n node1 --upload /path/to/file --suffix=somegroup The '-t' or '--tasks' option runs the de executables that are in tasks/ (if you also supply '-f'..!) These executables can also have classnames as filename extension. They can be shell scripts or any other kind of executables. This option is particularly useful for making system changes that cannot be done easily by replacing a configuration file, like for example installing new software packages. Mind to always include a check to see whether the system change has already been made, or else it will always keep installing the same software when it was already there. Doing system changes through the tasks mechanism is highly recommended, for 2 reasons: * easy to see what changes are being done; all tasks are in tasks/ * whenever a node is down, it can do the updates later to get back in sync By default, '--tasks' is not being run. You have to explicitly specify this argument to run tasks. The '--unix' option produces unix-style output. This shows in standard shell syntax just what synctool is about to do. (Note: synctool does not apply changes by executing shell commands; all operations are programmed in Python. This option is only a way of displaying what synctool does, and may be useful when debugging). root@masternode:/# synctool --unix node3: # updating file /etc/xinetd.d/identd node3: mv /etc/xinetd.d/identd /etc/xinetd.d/identd.saved node3: umask 077 node3: cp /var/lib/synctool/overlay/etc/xinetd.d/identd._all /etc/xinetd.d/identd node3: chown root.root /etc/xinetd.d/identd node3: chmod 0644 /etc/xinetd.d/identd node3: node3: # run command /bin/rm /etc/xinetd.d/*.saved ; /etc/init.d/xinetd reload node3: /bin/rm /etc/xinetd.d/*.saved ; /etc/init.d/xinetd reload The '--skip-rsync' option skips the rsync run (that copies the repository from the master server to the client node). You should only specify this option when your repository resides on a shared filesystem. Sharing the repository between your master server and client nodes has certain security implications, so be mindful of what you are doing in such a setup. If you have a fast shared filesystem between all client nodes, but it is not shared with the master server, you may want to write a wrapper script around synctool that first runs rsync to a single node to update the shared repository, and then run synctool with the --skip-rsync option. When using "--fix" to apply changes, synctool can log the performed actions in a log file. Use the 'logfile' directive in synctool.conf to specify that you want logging: logfile /var/log/synctool.log synctool will write this logfile on each node seperately, and a concatenated log on the master node. synctool can check whether a new version of synctool itself is available by using the '--check-update' option on the master node. To download the latest version, run synctool '--download' on the master node. EOB