[CCP14 Home: (Frames | No Frames)]
CCP14 Mirrors: [UK] | [CA] | [US] | [AU]

(This Webpage Page in No Frames Mode)

Collaborative Computational Project Number 14

for Single Crystal and Powder Diffraction

CCP14

Server Security Information

Routine, flexible and convenient Windows to UNIX Web updating using Rsync

The CCP14 Homepage is at http://www.ccp14.ac.uk

[Back to CCP14 Web/Config Main Page]

[Security Links Homepage] | [To FTP secure shell Tunnelling Page] | [To X secure shell Tunnelling] | [Routine Windows to UNIX Web updating using Rsync] | [Secure Routine Windows to UNIX Web updating using Teraterm and Rsync]

What the point of this?

The point of this is to be able do very easy web updating (even over tortuously slow phone lines) in a convenient, timely and effective manner. FTP can be quite painful in this regard but Rsync can be superb. If you change a line of a 500kB ASCII file, rsync will only need to send around 2kB of the "changes" over - and it will compress, and run via secure shell if desired.

The following is based around the excellent "Installing ssh and rsync on a Windows machine" by Robert Scholten at http://optics.ph.unimelb.edu.au/help/rsync/

However, a big difference is that this does not go via secure shell as I could not get the default compiled ssh to work. The following tutorial on how to "tunnel" via "Teraterm/ttsh Secure Shell" to the rsync deamon could be more convenient: Secure Routine Windows to UNIX Web updating using Teraterm and Rsync


Notes:

Relevant pages:


Install and configure Rsync on your UNIX (VMS?) webserver. You can use the following information.

Methods may vary, but put in a user specific config in the /etc/rsyncd.conf file. One for each user.

motd file = /etc/motd
     max connections = 25
     syslog facility = local3
#  Normally you would set read only = yes - but we want the user to be
#     able to write files to his area.

[joe_blogs_web]
   comment = updating Joe Blogs's webpage via rsync
   path = /path/to/web_area/for/joe
   read only = no
   list = yes
   uid = real_user_id
   gid = real_user_group
   hosts allow = machine1.dl.ac.uk  machine2.domain.com machine2.dl.ac.uk
   auth users = rsync_specific_username
   secrets file = /etc/rsyncd.secrets

NOTE: If you change the rsyncd.secrets file, then (in inetd mode) REHUP Inetd on the server using /etc/killall -HUP inetd to get this changed noticed. Otherwise, you may get an auth error, even if you are inserting the correct rsync password.


Downloading and installing rsync for Windows

Look at the instructions at Robert Scholten's "Installing ssh and rsync on a Windows machine" by Robert Scholten at http://optics.ph.unimelb.edu.au/help/rsync/

Download the rsync/ssh package from http://optics.ph.unimelb.edu.au/help/rsync/rsync_package.zip

Extra the zip so that it creates subdirectories in the c:\etc and c:\rsync area. This is important if you are trying to use the secureshell option that comes with this distribution of "rsync for Windows". If you extract these into other directories such as c:\rsync\etc and c:\rsync\sync, the following secure shell part will not work. This is to take into account some UNIX nuances of this particular Windows implementation of rsync.

In the c:\rsync directory, rename one of the rsync_version.exe to rsync.exe. At this date, I used the rsync2.4.3.exe version. If you get problems in file transfers, you may have to try another version.

ren rsync2.4.3.exe rsync.exe


Setting up rsync command file

The method to use to start with is to always try this with --dry-run option first to see what will happen. It is possible to do nasty things with rsync - it will obey what you tell it to do (which may not be what you think you requested). Also, don't forget to turn off the --dry-run option when you want the files to actually transfer across.

Another hint is don't use the --delete option. One tired move with this option and you may accidentally delete a portion of the web-area that you did not mean to. If you need to elliminate files, it might be best to do this using a GUI FTP program.

Following is an example DOS style bat batch file to run various options depending on what is required. (it has a lot of the common tools "REMMED" ready for cut and paste) Once you get into a routine, you may create you own discrete batch files. If doing a set of work on a certain directory system, extend a command line for that directory. --exclude "*.pfe" is the backup file name that the freeware PFE for Windows text editor creates when editing the HTM and HTML file. So we don't want to transfer the backup files over. The "\\d/" stands for D: drive on the hard-disk.

We specify the rsync area we want to update (in this case, joe_blogs_web), the rsync specific user name, and the option.

@echo off
REM c:\rsync\rsync -av --delete "\\d/" torte:/rsync/data2/scholten/schizo/d >>c:\rsync\rsync.log
REM netjam --progress --safe-links --dry-run
REM http://optics.ph.unimelb.edu.au/help/rsync/#install
REM http://www.ccp14.ac.uk/ccp14admin/rsync/index.html --exclude "*.gif" --exclude "*.jpg"
set CYGWIN=tty binmode
set TERM=ansi
REM set RSYNC_PASSWORD=plaintextpassword
REM >>c:\rsync\rsync.log
set RSYNC_RSH=c:\rsync\ssh.exe
REM echo Drive D >>c:\rsync\rsync.log

c:\rsync\gnudate

REM Books test
REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --delete --stats -e ssh -z "\\d/lachlan/web/books/" rsync_user_name@www.web_server.ac.uk::joe_blogs_web/books/

REM root area HTML changes (what's new). Currently running
c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.html" --include "*.htm" --exclude "*.gif" --exclude "*.jpg" --exclude "**" --stats -e ssh -z "\\d/lachlan/web/" rsync_user_name@www.web_server.ac.uk::joe_blogs_web/

REM Images test
REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.html" --include "*.htm" --include "*.gif" --include "*.jpg" --exclude "**" --stats -e ssh -z "\\d/lachlan/web/images/" rsync_user_name@www.web_server.ac.uk::joe_blogs_web/images/

REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.html" --include "*.htm" --include "*.gif" --include "*.jpg" --exclude "**" --stats -e ssh -z "\\d/lachlan/web/" rsync_user_name@www.web_server.ac.uk::joe_blogs_web/

REM Complete update
REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --stats -e ssh -z "\\d/lachlan/web/" rsync_user_name@www.web_server.ac.uk::joe_blogs_web/

REM Root area and tutorial area
REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.html" --include "*.htm" --include "*.gif" --include "*.jpg" --include "/tutorial/" --include "/tutorial/**" --exclude "**" --stats -e ssh -z "\\c/lachlan/ccp14/web_live/" rsync_user_name@www.web_server.ac.uk::joe_blogs_web/

REM Root area and tutorial directory and no subdirectories
REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.html" --include "*.htm" --include "*.gif" --include "*.jpg" --include "/tutorial/" --exclude "**" --stats -e ssh -z "\\c/lachlan/ccp14/web_live/" rsync_user_name@www.web_server.ac.uk::joe_blogs_web/

REM ccp14admin security area + rsync + ccp14admin index page
REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.txt" --include "*.zip" --include "*gz" --include "*Z" --include "*.html" --include "*.htm" --include "*.gif" --include "*.jpg" --include "/rsync/" --include "/rsync/**" --include "/security/" --include "/security/**" --exclude "**" --stats -z "\\c/lachlan/ccp14/web_live/ccp14admin/" rsync_user_name@www.web_server.ac.uk::joe_blogs_web/ccp14admin/

c:\rsync\gnudate


Running Rsync

In this example, we have modified a file in the root web-area, whatsnew.htm. So the DOS style BATCH program we run has an rsync command line that only updates the HTML files in the root web-area, and not in any subdirectory.

c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.html" --include "*.htm" --exclude "*.gif" --exclude "*.jpg" --exclude "**" --stats -e ssh -z "\\d/lachlan/web/" rsync_user_name@www.web_server.ac.uk::joe_blogs_web/

Rsync will then prompt for the "rsync" password corresponding to this username and the transfer will continue.

You can avoid having to type in a password by using the following declaration in your batch file (but this is not a good habit to get into):

  • set RSYNC_PASSWORD=plaintextpassword

In the following screen dump, rsync has started, prompted for the password (which the user enters); then determins what has changed, and updates the file. Note that even though the whatsnew.htm file is 47 kB, rsync only had to transfer 808 bytes to update the remote file on the web server to match the local version. This means it takes far less time and bandwidth.

Rsync running

To repeat the uploading exercise, have a DOS window open with doskey installed, and just press the "up arrow" key and press enter to run the previous command.


[Back to CCP14 Web/Config Main Page]

[Security Links Homepage] | [To FTP secure shell Tunnelling Page] | [To X secure shell Tunnelling] | [Routine Windows to UNIX Web updating using Rsync] | [Secure Routine Windows to UNIX Web updating using Teraterm and Rsync]

[CCP14 Home: (Frames | No Frames)]
CCP14 Mirrors: [UK] | [CA] | [US] | [AU]

(This Webpage Page in No Frames Mode)

If you have any queries or comments, please feel free to contact the CCP14