[CCP14 Home: (Frames | No Frames)]
CCP14 Mirrors: [UK] | [CA] | [US] | [AU]
(This Webpage Page in No Frames Mode)

Collaborative Computational Project Number 14

for Single Crystal and Powder Diffraction

CCP14

WGET software for FTP and Web Auto-mirroring

The CCP14 Homepage is at http://www.ccp14.ac.uk
[Getting WGET | WGET for VMS FTP Servers | Compiling WGET | The .wgetrc Config File | The WGET Manual Files]
[Setting up Cron Scripts | Used WGET Options]
[Efficient Web/FTP Mirroring - HTTP Based | FTP Based | netlib style Co-operative Mirroring]
[rsync style Co-operative Mirroring]
[Logging Details of Nightly Incremental Updates]

Where to get WGET

WGET is freeware/Gnu UNIX based software written by Hrvoje Niksic (E-mail: hniksic@srce.hr). It is used for manual and automatic mirroring of FTP and Web sites. Various versions are available via the following links:

Using WGET from behind a firewall

GUI WGET - using a Tcl/Tk Script ("Utter Coolness")

From Paul Rahme (Mr Bogus), South Africa. E-mail: paulrahmel@hotmail.com

WGET for Grabbing files off VMS FTP Site

Patch and version by: 
Jan Prikryl - prikryl@cg.tuwien.ac.at 
         http://www.cg.tuwien.ac.at/staff/JanPrikryl.html
         Institute of Computer Graphics and Visualisation
             Vienna University of Technology, Austria

Possible Alternatives to WGET - PAVUK, FWGET and LFTP

  • FWget - FTP and HTTP Mirroring
    • Personal update of WGET. A possible alternative to WGET.
    • Refer to http://bay4.de/FWget/
    • "FWget is based on Wget, but I have replaced some of the linked lists with hashtables. Wget has got serious problems retrieving huge sites, FWget hasn't. That's about all about it. Version 1.5.3.1 contains some fixes I have not reviewed."

Optimising and Compiling WGET

By default, WGET converts the ~ (tilder) into a numeric equivalent which can create a bit of havoc when trying to do auto-mirroring like this. To have it leave the ~ as is, the following is done (thanks to Hrvoje on mentioning how to do this):

For the obsolescent WGET 1.5.3

  • Extract the files from the gz file with the command
    gzip -d < wget-version.tar.gz | tar xvof -
i.e.,  gzip -d < wget-1.5.3.tar.gz | tar xvof -
  • Before compiling a new version of WGET, edit the url.c file
  • Remove the ~ from the following i.e.,: 
    #ifndef WINDOWS
# define URL_UNSAFE " <>\"#%{}|\\^~[]`@:\033"
#else
# define URL_UNSAFE " <>\"%{}|\\^[]`\033"
#endif
    To: 
    #ifndef WINDOWS
# define URL_UNSAFE " <>\"#%{}|\\^[]`@:\033"
#else
# define URL_UNSAFE " <>\"%{}|\\^[]`\033"
#endif

For the latest WGET 1.7.1 and later

  • Extract the files from the gz file with the command
    gzip -d < wget-version.tar.gz | tar xvof -
i.e.,  gzip -d < wget-1.8.1.tar.gz | tar xvof -
  • Before compiling a new version of WGET, edit the url.c file

  • Change the following from (The U corresponding to the ~ with a 0 (a zero)): 
    const static unsigned char urlchr_table[256] =
{
  U,  U,  U,  U,   U,  U,  U,  U,   /* NUL SOH STX ETX  EOT ENQ ACK BEL */
  U,  U,  U,  U,   U,  U,  U,  U,   /* BS  HT  LF  VT   FF  CR  SO  SI  */
  U,  U,  U,  U,   U,  U,  U,  U,   /* DLE DC1 DC2 DC3  DC4 NAK SYN ETB */
  U,  U,  U,  U,   U,  U,  U,  U,   /* CAN EM  SUB ESC  FS  GS  RS  US  */
  U,  0,  U,  U,   0,  U,  R,  0,   /* SP  !   "   #    $   %   &   '   */
  0,  0,  0,  R,   0,  0,  0,  R,   /* (   )   *   +    ,   -   .   /   */
  0,  0,  0,  0,   0,  0,  0,  0,   /* 0   1   2   3    4   5   6   7   */
  0,  0,  U,  R,   U,  R,  U,  R,   /* 8   9   :   ;    <   =   >   ?   */
 RU,  0,  0,  0,   0,  0,  0,  0,   /* @   A   B   C    D   E   F   G   */
  0,  0,  0,  0,   0,  0,  0,  0,   /* H   I   J   K    L   M   N   O   */
  0,  0,  0,  0,   0,  0,  0,  0,   /* P   Q   R   S    T   U   V   W   */
  0,  0,  0,  U,   U,  U,  U,  0,   /* X   Y   Z   [    \   ]   ^   _   */
  U,  0,  0,  0,   0,  0,  0,  0,   /* `   a   b   c    d   e   f   g   */
  0,  0,  0,  0,   0,  0,  0,  0,   /* h   i   j   k    l   m   n   o   */
  0,  0,  0,  0,   0,  0,  0,  0,   /* p   q   r   s    t   u   v   w   */
  0,  0,  0,  U,   U,  U,  U,  U,   /* x   y   z   {    |   }   ~   DEL */

  • To: 
    const static unsigned char urlchr_table[256] =
{
  U,  U,  U,  U,   U,  U,  U,  U,   /* NUL SOH STX ETX  EOT ENQ ACK BEL */
  U,  U,  U,  U,   U,  U,  U,  U,   /* BS  HT  LF  VT   FF  CR  SO  SI  */
  U,  U,  U,  U,   U,  U,  U,  U,   /* DLE DC1 DC2 DC3  DC4 NAK SYN ETB */
  U,  U,  U,  U,   U,  U,  U,  U,   /* CAN EM  SUB ESC  FS  GS  RS  US  */
  U,  0,  U,  U,   0,  U,  R,  0,   /* SP  !   "   #    $   %   &   '   */
  0,  0,  0,  R,   0,  0,  0,  R,   /* (   )   *   +    ,   -   .   /   */
  0,  0,  0,  0,   0,  0,  0,  0,   /* 0   1   2   3    4   5   6   7   */
  0,  0,  U,  R,   U,  R,  U,  R,   /* 8   9   :   ;    <   =   >   ?   */
 RU,  0,  0,  0,   0,  0,  0,  0,   /* @   A   B   C    D   E   F   G   */
  0,  0,  0,  0,   0,  0,  0,  0,   /* H   I   J   K    L   M   N   O   */
  0,  0,  0,  0,   0,  0,  0,  0,   /* P   Q   R   S    T   U   V   W   */
  0,  0,  0,  U,   U,  U,  U,  0,   /* X   Y   Z   [    \   ]   ^   _   */
  U,  0,  0,  0,   0,  0,  0,  0,   /* `   a   b   c    d   e   f   g   */
  0,  0,  0,  0,   0,  0,  0,  0,   /* h   i   j   k    l   m   n   o   */
  0,  0,  0,  0,   0,  0,  0,  0,   /* p   q   r   s    t   u   v   w   */
  0,  0,  0,  U,   U,  U,  0,  U,   /* x   y   z   {    |   }   ~   DEL */
  • To: 
    







    

  • NOTE: On SGI IRIX 6.5.x, it can be dangerous using gcc to 
   compile at WGET as strange things can happen due to bugs in gcc vs IRIX.  
    (e.g., it can start mirroring the destination server on top of itself).

    

  • For SGI IRIX, config the system for installing on your home area by running the following via Bash shell
to force the compilation to occur via cc (not gcc):
    
    CC=cc ./configure

  • Then compile wget (the executable is created in the src directory) by typing
     make

  • Check that the ~ mod works by starting a grab on a user site 
    (e.g., wget http://www.chem.gla.ac.uk/~louis/).  Check that it has created
    the user directory as ~louis.



















The .wgetrc Config File


After compiling WGET, you need to install a .wgetrc file in your home area.
The WGET manual file goes into detail about options and how to possibly setup the
.wgetrc file.   Main things to note is which vary from the default.


    

  • follow_ftp = off

  • recursive = on

  • dirstruct = on

  • robots = on




(To disable robots.txt via the command line, add --execute robots=off)



Click here to view the simple CCP14 .wgetrc file.




















WGET Manual Files


(from wget-1.5.3 - September 1998)




Windows Help File format


(In Zip format - from wget-1.5 - April 25th 1998)



















Setting Up Script Files and Cron Job for Automirroring



Automirroring is still under Construction. An alpha version of a WGET that is VMS
friendly is being used to automatically mirror VMS FTP servers. 
(affects GSAS, Larry Finger FTP Site and Profil).




Automirroring run daily (or if less frequent mirroring is request by remote
web-master) using cron (~1 to ~5am UK time).  Depending on the site,
some customisation can be required.  Either to link the web to the ftp
site, or overcome "non-recommended" html in the files that WGET is not
happy with.  The reason for the above logging times is that grabbing files over the US
link (from the UK) incurrs a charge (~2pence per Meg) but there is a free time
between 1am and 5am.  Thus automirroring is rigged to occur between these
hours - though even if done outside these hours, the amount of material coming
over for incremental mirroring is quite trivial.  Just before 5am, all wget jobs
are killed and the automatically emailed summary log is quickly examined to see which
jobs did not complete.  These can be swapped around so they do complete within
the required time.


    

  • The cron file contains the list of script files that are run each 
morning.  As recommended by the local cron guru, a .crontab file exists 
in the home directory which is freely edited.  To enable cron
to use this updated file, the following command is invoked: 
  crontab .crontab.  It is possible to inadvertantly corrupt a 
crontab file (i.e., crontab -r means remove - not read!).  The default UNIX
line editor can cause some mayhem for the unwary when crontab -e is run.


    #start these auto-mirroring jobs up
01 01 * * * cp web_area/mirrorbin/logs/report-proforma.txt web_area/mirrorbin/logs/report.txt
31 01 * * * web_area/mirrorbin/wget.script.alife
32 01 * * * web_area/mirrorbin/wget.script.armel
33 01 * * * web_area/mirrorbin/wget.script.asia
34 01 * * * web_area/mirrorbin/wget.script.australia
35 01 * * * web_area/mirrorbin/wget.script.cguitools
36 01 * * * web_area/mirrorbin/wget.script.cguitools.wxwindows
37 01 * * * web_area/mirrorbin/wget.script.commercial
38 01 * * * web_area/mirrorbin/wget.script.europe.one
39 01 * * * web_area/mirrorbin/wget.script.europe.two
10 01 * * * web_area/mirrorbin/wget.script.generalcode
11 01 * * * web_area/mirrorbin/wget.script.gnu
12 01 * * * web_area/mirrorbin/wget.script.gnu-site
13 01 * * * web_area/mirrorbin/wget.script.gnu.mumit-khan
14 01 * * * web_area/mirrorbin/wget.script.netlib
15 01 * * * web_area/mirrorbin/wget.script.northamerica.one
16 01 * * * web_area/mirrorbin/wget.script.northamerica.two
17 01 * * * web_area/mirrorbin/wget.script.southamerica
18 01 * * * web_area/mirrorbin/wget.script.swarm
19 01 * * * web_area/mirrorbin/wget.script.tcltk
20 01 * * * web_area/mirrorbin/wget.script.uk
21 01 * * * web_area/mirrorbin/wget.script.utility
22 01 * * 0 web_area/mirrorbin/wget.script.weekly
23 01 * * * web_area/mirrorbin/wget.script.wget
05 05 * * * /usr/sbin/Mail -s "Mirroring_Results `date`" ccp14@dl.ac.uk < web_area/mirrorbin/logs/report.txt
    

  • Cron invokes the commands from the home directory (~/).  Thus everything
must be relative to the home directory.  

  • The columns stand for when the command is run:
 
      
 
    • First column is the minute
 
    • Second column for the hour
 
    • Third column for the day of the month (* means run on each day of the month)
 
    • Forth column for the month of the year (* means run on each month of the year)
 
    • Fifth column for the day of the week (0 is a Sunday, * means run on each day of the week)
 
    





    

  • To mirror both the FTP and Web area of a site, the following type of system
is defined in an ASCII Script file.  In this case, for grabbing the ORTEP
site.


    

To internally link the html files to point correctly to the local
ftp area, the rather kludgy cshell scripts are run after WGET.

    
After each WGET grab, it's completion time is logged to the report.txt file which
at the end of all the wget sessions, is mailed for easy and convenient viewing
to see that all jobs ran.


    #!/bin/csh
# You should CHANGE THE NEXT 3 LINES to suit your local setup
setenv  LOGDIR   ./web_area/mirrorbin/logs    # directory for storing logs
setenv  PROGDIR  ./web_area/mirrorbin         # location of executable
setenv  PUTDIR   ./web_area/web_live/ccp      # relative directory for mirroring

#FTP Ortep 3 FTP Site
#  E-mail: ortep@ornl.gov (Dr. Michael N. Burnett)
$PROGDIR/wget -nH -r -N -nr -l0 -k -np -X /cgi-bin --cache=off \
 ftp://ftp.ornl.gov/pub/ortep/ \
 -P $PUTDIR/ccp14/ftp-mirror/ornl-ortep \
 -o $LOGDIR/northamerica.one.log 

set DATE=(`date`)
sed "/ORNL_Ortep_FTP/s/NOT_finished/COMPLETED $DATE/" $LOGDIR/report.txt  > $LOGDIR/report.txt.new
mv $LOGDIR/report.txt.new $LOGDIR/report.txt


#WEB Ortep 3 Web Site
#  E-mail: ortep@ornl.gov (Dr. Michael N. Burnett)
$PROGDIR/wget -nH -r -N -nr -l0 -np -X /cgi-bin --cache=off \
 http://www.ornl.gov/ortep/ortep.html \
 http://www.ornl.gov/ortep/topology.html \
 -P $PUTDIR/web-mirrors/ornl-ortep \
 -o $LOGDIR/northamerica.one.log 


foreach f ($PUTDIR/web-mirrors/ornl-ortep/ortep/*.html)
sed -e 's+=\"http://www.ornl.gov/ortep+=\".p+g' < $f > $f.tmp
mv $f.tmp $f
end

foreach f ($PUTDIR/web-mirrors/ornl-ortep/ortep/*.html)
sed -e 's+\"ftp://ftp.ornl.gov+\"./../../../ccp14/ftp-mirror/ornl-ortep+g' < $f > $f.tmp
mv $f.tmp $f
end

foreach f ($PUTDIR/web-mirrors/ornl-ortep/ortep/topology/*.html)
sed -e 's+\"ftp://ftp.ornl.gov+\"./../../../../ccp14/ftp-mirror/ornl-ortep+g' < $f > $f.tmp
mv $f.tmp $f
end

foreach f ($PUTDIR/web-mirrors/ornl-ortep/ortep/examples/*.html)
sed -e 's+\"ftp://ftp.ornl.gov+\"./../../../../ccp14/ftp-mirror/ornl-ortep+g' < $f > $f.tmp
mv $f.tmp $f
end

set DATE=(`date`)
sed "/ORNL_Ortep_Web/s/NOT_finished/COMPLETED $DATE/" $LOGDIR/report.txt  > $LOGDIR/report.txt.new
mv $LOGDIR/report.txt.new $LOGDIR/report.txt
    











Defining what each WGET option does


(Credit goes to the IUCr Technical mirroring page at 
  
   http://www.iucr.org/iucr-top/docs/mirror/)


    

  • -nh - (This option has been deleted in wget 1.8.x) 
   Do not perform DNS lookup on the host name (speeds things up)
    
    However, for where a webpage has absolute computer names that are really the
    same computer (i.e., wserv1.dl.ac.uk and www.dl.ac.uk), disable this so that it
    can check if these pages do belong on the same computer and can grab them.
    DNS lookup is enabled on the Sirware site to in theory make sure it 
    treats www.ba.cnr.it and area.ba.cnr.it as identical webservers.
   
    

    #Sirware WebSite
#  E-mail: cryst@area.ba.cnr.it
$PROGDIR/wget -nH -r -N -nr -l0 -k -np -X /cgi-bin --cache=off  \
 http://www.ba.cnr.it/IRMEC/SirWare_main.html \
 http://www.ba.cnr.it/IRMEC/ \
 -P $PUTDIR/web-mirrors/sirware \
 -o $LOGDIR/europe.log

    


  • -nH - Disable generation of host name prefixes (stop the creation of
         host subdirectory names - i.e, ./pub/ortep instead of ./ftp.ornl.gov/pub/ortep/)

  • -r - recurse down the subdirectory tree

  • -R - reject files (e.g., -R .listing)

  • -N - turn on time stamping to help enable incremental updates of files

  • -nr - retain .listing files for FTP (enables incremental updates of ftp areas)

  • -l0 - recursively fetch files to infinite depth (-l3 would tell wget to
         recursively go 3 levels deep)

  • -np - "no parents" - only recurse down the directory tree - never fetch
    a parent of the root directory.

  • -X - reject subdirectories with this name.

  • -A - accept files with this text (i.e., -Aexe grabs all *.exe files, but nothing
                else)

  • -I - accept subdirectories with these extensions (this is an alternative to using
     -np "no parents".  Is used where icons or images may be kept in a completely
     different part of the web server.

  • -k - try and convert absolute links into "working" relative links. (not 100%
    efficient in the present version of WGET).

  • -K - (WGET 1.7 and later) --backup-converted'
     When converting a file, back up the original version with a `.orig'
     suffix.  Affects the behavior of `-N' (*note HTTP Time-Stamping
     Internals::).

  • -P - tell WGET where to put the downloaded files.  (Using relatively 
     directory paths can be helpful as WGET can do strange things when specifying
     absolute directory paths)

  • -o - tell WGET where to put the screen output to.  If you don't specify this, the
     cron job will give you this output in an email message (NB: using the append command does 
     not work with latest version of WGET as output is now considered to be stderr - thus going
     via email if invoked via cron)


  • -D - Only grab files from the defined domains.  This is required when for some
strange reason -nh does not work.  In this case, for the www.netlib.org
site which also starts grabbing the university (.edu) web-site that is 
hosting this area.  In this case use -Dnetlib.org as it hosts other org
bodies.

  • --cache=off - Will get around problems of caches reporting cached versions.
 (When set to off, disable server-side cache. In this case, Wget will send the remote server an appropriate directive (`Pragma: no-cache') to get the file
     from the remote service, rather than returning the cached version. This is especially useful for retrieving and flushing out-of-date documents on proxy servers.
     Caching is allowed by default.)

  • -debug - debug mode - for when you have to send a bug report in

   
    

    #Netlib - Code Library
#  E-mail: ehg@research.bell-labs.com (Eric Grosse)
$PROGDIR/wget -nh -Dnetlib.org -nH -r -N -nr -l0 -k -np  --cache=off \
   -X /cgi-bin,/~icl,/utk  \
 http://www.netlib.org/ \
 -P $PUTDIR/web-mirrors/programming/netlib \
 -o $LOGDIR/programming.log

    




















Efficient Web/FTP Mirroring - using WGET



Efficient HTTP Based Web Mirroring - using WGET



Generally, HTTP based mirroring is "NOT" very efficient for incremental updates 
as it requires that every web file is "pinged" to check that it has changed.  
However (presently) if you have to do some processing of these files, or use it's 
internal links to get off-domain files, it is quite effective.  On small sites, HTTP 
based mirroring is easy to do and not a problem, but on large sites, it can take most 
of the day to do a daily or weekly incremental mirror.






Efficient FTP Based Web/FTP Mirroring - using WGET


FTP mirroring can be far more efficient for mirroring sites as to get an incremental update
of a web or ftp site, a directory listing is all it requires to check whether there have been
updated files.  (compared to http mirroring where it has to ping every file to do this check)


While FTP based mirroring can be an order or magnitude or more efficient than
HTTP based mirroring; one downside with ftp mirroring is that WGET cannot presently interrogate 
the html files grabbed by FTP to convert any absolute links to relative links, etc.  This method 
also assumes all the files are available by grabbing the FTP tree.  For web areas
not accessible via public/anonymous FTP, just use password protected access.







#Armel Le Bail's WEB Site - via password protected FTP
#  E-mail: armel@fluo.univ-lemans.fr (Armel LeBail)
$PROGDIR/wget -r -nH --cut-dirs=4  -N -nr -l0 -np  --cache=off \
 ftp://username:password@199.199.199.199//home/armel/doobry/webdocs \
  -P $PUTDIR/web-mirrors/armel \
  -o $LOGDIR/armel.log



Notice how the --cut-dirs=4 is used to make 4 level deep directory that houses
the web pages look like the root area on the web at mirrored pages 
../ccp/web-mirrors/armel/

Otherwise   the mirroring would be at ../ccp/web-mirrors/armel/home/armel/doobry/webdocs



It is also possible to create an account where all that can be performed
is read-only ftp and the web-area to be mirrored also looks like an root
ftp area, as though it was an anonymous ftp area, but with password
access restriction.  On a SGI IRIX system, this can initially be a fiddly job 
to set up but once you know how to do this (preferably by having access to a local
UNIX/network admin guru) (man ftpd explains how to set this up in quite some detail).



    

  • Create a new user (e.g., ftpmirror) whose home area is the root area
    of the web-files (e.g., /web_disk/ccp14/web_area)

  • Create a /ftpusers file with the appropriate restrictions

    ftpmirrror restrict
    
 (refer to man ftpd for how this may work on your varient of UNIX).
This means that when logged in as ftpmirror, the system will do a chroot

  • Set various permissions (which I am going to have to get the info on)

  • Create an appropriate FTP set of subdirectories with ownership of root and group owner of sys.
    Locate the appropriate functionality within these directories, such as ls to be
    able to do directory lists.
   In the subdirectories, . is owned by root with group of sys.   
   .. is owned (as example) by ccp14 with group of dlccp14a.
  (again, refer to man ftpd for how this may work on your varient of UNIX)
     
  
      
  
    • Make sure to read the "man ftpd" as this may vary with versions of the
      operating system. 
  
    

  • As people may try to browse these ftp config files (i.e., http://sv1.ccp14.ac.uk/bin/), redirect them to 
    an appropriate page via the web server redirect procedure.  In the case of the Apache web server, 
    something like the following in the /usr/local/etc/httpd/conf/httpd.conf file:

    RewriteEngine on
RewriteRule ^/bin/(.*) http://www.ccp14.ac.uk/bad-link.html [R]
RewriteRule ^/dev/(.*) http://www.ccp14.ac.uk/bad-link.html [R]
RewriteRule ^/etc/(.*) http://www.ccp14.ac.uk/bad-link.html [R]
RewriteRule ^/lib/(.*) http://www.ccp14.ac.uk/bad-link.html [L,R]
RewriteRule ^/lib32/(.*) http://www.ccp14.ac.uk/bad-link.html [L,R]
    
This may not be the 100% security kosher way of doing this, but is also cited on the 
main apache newsgroups as a nifty trick to do.




















ULTRA Efficient netlib style Co-operative Mirroring - NOT Presently in WGET


The basis of this system is that the server being mirrored co-operates in providing information
on what files have been modified over time.



Refer to the following published article on this.


From ehg@research.bell-labs.com  Thu Jun 11 23:41:38 1998
Date: Thu, 11 Jun 1998 18:30:54 -0400
To: L.M.D.Cranswick@dl.ac.uk
From: "Eric Grosse" <ehg@research.bell-labs.com>
Subject: re: More Detailed Spec on your Mirroring Method?

There is the paper
 Article Grosse:1995:RM,
 author =       "Eric Grosse",
 title =        "Repository Mirroring",
 journal =      "ACM Trans. Math. Software",
 volume =       "21",
 number =       "1",
 pages =        "89--97",
 month =        mar,
 year =         "1995",
 CODEN =        "ACMSCU",
 ISSN =         "0098-3500",
 bibdate =      "Tue Apr 25 15:42:31 1995",
 URL =          "ftp://netlib.bell-labs.com/netlib/crc/mirror.ps.gz",
 keywords =     "C.2.4 [Computer-Communication Networks]: Distributed
 Systems -- distributed databases",
 subject =      "archives; checksum; distributed administration;
 electronic distribution; ftp"


The basic idea is to publish (as an ordinary file on the web/ftp site)
a simple listing of the entire collection as
   pathname unixtime bytes checksum
and then provide utilities to compare two such files, generating
commands to bring the "slave" into sync with the "master."  The
customary shell or perl tools can be used to filter these files
for complicated master/slave relationships, or you can use it
straight, for straightforward mirroring.

If I were redoing this today, I might use persistent HTTP in place of
ftp, and might merge netlib's MD5 files into these "crc" files.  But
that would be pushing farther forward than some people want to go
just yet;  ftp and crc are still the right conservative choice.

You're welcome to repost this to the wget mailing list if you like.

Eric



















rsync style Co-operative Mirroring - NOT Presently in WGET


While not present in WGET yet, another option for high effeciency co-operative mirroring is the
rsync software.  The server you wish to mirror must have an rsync server running.  These
are not difficult to install and configure.



















Logging Details of Nightly Incremental Updates





To keep control on the incremental nightly mirroring, script files log 
the time when a backup has finished into a single file which is the emailed
each morning to the operator.  This makes any problems that have occured
the evening before quite obvious.




A report-proforma.txt text file is kept up to
date with sites being mirrored.

At the start of the cron job, a new report.txt file is created.

01 01 * * * cp web_area/mirrorbin/logs/report-proforma.txt web_area/mirrorbin/logs/report.txt





Script file run after each incremental WGET script is of the following type:

#ANSTO LHPM Rietveld FTP Area
#  E-mail: bah@ansto.gov.au (Brett Hunter)
$PROGDIR/wget -nH -r -N -nr -l0 -np -X /cgi-bin \
  ftp://ftp.ansto.gov.au/pub/physics/neutron  \
  -P $PUTDIR/ccp14/ftp-mirror/ansto \
  -o $LOGDIR/australia.log 

set DATE=(`date`)
sed "/ANSTO_LHPM_FTP/s/NOT_finished/COMPLETED $DATE/" $LOGDIR/report.txt  > $LOGDIR/report.txt.new
mv $LOGDIR/report.txt.new $LOGDIR/report.txt


This substitutes the relevant "NOT_finished" with "COMPLETE and the date-time" such that
un-finished jobs due to network/domain problems are relatively obvious.


EUROPE_ONE
Fullprof_PLOTR_FTP                  has COMPLETED Thu Aug 6 01:38:12 BST 1998
PLOTR_Web                           has COMPLETED Thu Aug 6 01:38:28 BST 1998
Simref_Simpro_MEED                  has COMPLETED Thu Aug 6 01:38:47 BST 1998
FIT2D_FTP                           has COMPLETED Thu Aug 6 01:38:49 BST 1998
FIT2D_Web                           has COMPLETED Thu Aug 6 01:41:32 BST 1998
Jana_FTP                            has COMPLETED Thu Aug 6 01:41:55 BST 1998
Jana_Web                            has NOT_finished
Stefan_Krumm_FTP                    has COMPLETED Thu Aug 6 01:42:40 BST 1998
Stefan_Krumm_Web                    has COMPLETED Thu Aug 6 01:43:27 BST 1998
XND_Rietveld_FTP                    has COMPLETED Thu Aug 6 01:43:33 BST 1998
XND_Rietveld_Web                    has COMPLETED Thu Aug 6 01:44:02 BST 1998
Sirware_Web                         has COMPLETED Thu Aug 6 01:45:58 BST 1998
XPMA_Zortep_Web                     has COMPLETED Thu Aug 6 01:46:00 BST 1998
DMPLOT_FTP                          has COMPLETED Thu Aug 6 01:46:13 BST 1998
DIRDIF_FTP                          has COMPLETED Thu Aug 6 01:46:20 BST 1998
CRUNCH_Web                          has COMPLETED Thu Aug 6 01:47:17 BST 1998
DIRDIF_Web                          has COMPLETED Thu Aug 6 01:47:45 BST 1998
DRXWIN_FTP                          has COMPLETED Thu Aug 6 01:47:47 BST 1998
DRXWIN_Web                          has COMPLETED Thu Aug 6 01:48:56 BST 1998
AXES_FTP                            has COMPLETED Thu Aug 6 01:49:02 BST 1998
BGMN_Web                            has COMPLETED Thu Aug 6 01:49:53 BST 1998
ORTEX_Suite_Web                     has COMPLETED Thu Aug 6 01:50:03 BST 1998



All jobs are terminated before 5am (UK time), and the report is them mailed to
the operator using the following command in cron


05 05 * * * /usr/sbin/Mail -s "Mirroring_Results `date`" ccp14@dl.ac.uk < web_area/mirrorbin/logs/report.txt




















 
 

 [CCP14 Home: (Frames | 
     No Frames)]


 CCP14 Mirrors:

  [UK] | 
  [CA] | 
  [US] | 
  [AU]

 

 		
 
 
 
 
 
 
 
 
 
 




 
 
 
 
 
 
 
 
 
 
 
 
 










    (This Webpage Page in No Frames Mode)
    
    


    


    

    If you have any queries or comments, please feel free to 
     contact the CCP14