ISP Upload Script

As the number of my homepages has grown I have found it increasingly difficult to keep track of the changes that need loading from my system to the ISP. In order to address this problem I have written the upload script.

The script determines all the recent changes made to my web pages. It then uses ftp to send the changed files up to the ISP.

This page describes how to install and use the upload script.

Terms & Conditions

Conditions

You are free to install, modify and use upload on any system free of charge for 14 consecutive days. After the 14 days have expired you can continue to use upload free of charge provided that:
  1. You send us an email containing the following information:
    1. The name of the site.
    2. The name of the organisation or person that owns the site.
    3. For Internet sites, its URL.
    4. The Email address of the site administrator.
  2. you do not remove the original copyright notice from the script.
  3. you do not pass on this script, or any software derived from it to any third-party.
We would like you to acknowledge our contribution with a link to our site at http://www.pccl.demon.co.uk. But this is not a condition.

Disclaimer

Pentagon Computer Consultants Ltd., does not make any warranties with respect to the contents of the upload software or documentation and disclaims any implied warranties of merchantability or fitness for any purpose.

Pentagon Computer Consultants Ltd., reserves the right to revise the software or documentation at any time without obligation to notify any person or organisation.

Copyright

Upload is copyright ©1999 Pentagon Computer Consultants Ltd., Allrights reserved.

How it Works

Upload maintains 3 images of the web site. These are:
  1. The ROOT image is the local copy of the web site. It is a Unix directory structure containing copies of the data for the web site. The webmaster edits these files with any changes they require.
  2. The GHOST image is a directory structure that is a ghost image of the web site. This image is maintained by the script. The ghost image shows the web site structure but all of the files are empty.
  3. The WEB image is the web site held on the ISP, visible to anyone on the World Wide Web.
The webmaster creates and edits their web pages in the ROOT image. The web pages can be tested in situ. When the changes are complete, the webmaster runs upload.

The script compares the timestamps on the files in the ROOT image with those in the GHOST image. If any files in the ROOT image are newer than the GHOST image then there is some work to do.

The script uses ftp to connect to the ISP. It sends updated files up to the WEB image. It creates the same file structure as is in the ROOT image. It creates any new files and directories as required.

As each ROOT file is successfully transferred to the WEB, the GHOST file is touched. This prevents the file being re-sent on subsequent calls to upload.

Prerequisites

The upload script works on Linux systems. I have used the script on SuSE 5.3, 8.1 and 9.3 systems. It should also work on other variants of Linux and Unix.

Before you can use the upload script you will need to have expect(1) on your system. You may have it already. On my SuSE 5.3 system it is in /usr/bin. You may have it /usr/local/bin.

I believe that Expect can be obtained from the following ftp sites:

You will also need a copy of ftp. And, of course, you'll need a web site to update.

Installation

Downloading

Clicking on the upload link displays the script text. You can use the browser File-Save options to write the file to your disc. I keep the file in $(HOME)/bin on my system, which is on the execution PATH. You may prefer to put in /usr/local/bin.

Once you have the upload file on our system, you will have to make it executable. You do this with the chmod(1) command: 

chmod +x upload

Customising

You will need to configure FTP so that it can connect to your ISP. In your home directory, you should have a .netrc It should contain a record that describes how FTP connects to your ISP. See ftp(1) for details.

The upload script does not take any command line parameters. This may change in a future release. Currently, program parameters are "hard-wired" into the script. You will have to edit these parameter to suit your environment. These parameters are:

Around about line 38 of the script, you will see a comment: "These might be parameters one day". You can set the parameter values on the next few lines.

Setting up the filestore

Before running upload you will have to create the ROOT directory and put in it all the subdirectories and files you want on the WEB. Do not put any files that you do not want on the WEB.

You will also have to create the GHOST directory. Do not put any files or directories in it.

Use

Running

Warning:before using upload for the first time, make sure that you have a reliable backup copy of the data that you give it to deliver.

Because it take no parameters, you just execute: 

upload
It works out what to do, connects to the web site and sends up the changed files. It shows which file it is transmitting. If an error occurs then it stops after displaying an error message.

Solving Problems

When things go wrong it is useful to see what ftp messages are being exchanged. Expect provides ways of doing this. To turn these features on you will have to modify the script. If you uncomment the line that reads: 
log_file -noappend log
then logs will be written to the file called log in the current directory.

If you uncomment the line that reads: 

loguser     1
then logs will be written to the standard output.

When you use upload for the first time, you should use one of these logging options and study the output carefully. My version of ftp may format messages differently to yours. It may fool the script in to thinking that a transfer has worked when it has failed. Conversely, the script may interpret success as failure.

Before you come to rely on upload, you should be sure that it behaves correctly. You may have to modify the script to work with your FTP.

If you do solve any of these incompatibility problems then please let me know. I'll attempt to publish the information so that it can help others.

Limitations

By reading these notes, and the script itself, you have probably seen the script's limitations. These are mainly due to the fact that I wrote the script for my own use. I can live with these limitations. I may fix them one day if I have the time.
  1. The program does not take any command line parameters. This is not a problem if you have only one site to maintain.
  2. Files deleted from the ROOT filestore are not deleted from the WEB or GHOST filestores. These have to be deleted by hand. I find that I rarely delete files, so it does not bother me.
  3. If there is a problem with FTP so that a WEB directory is not successfully made then subsequent calls to upload will fail because the directory is expected to be there. The problem is solved either by using FTP to create the directory or by removing it from the GHOST filestore. This problem is so rare, it hasn't happened yet.
  4. If you add several layers of directory to a ROOT filestore and one of the directories does not contain files then the lower directories and files are not created. This is because I have not yet implemented the FTP equivalent of "mkdir -p". The problem is solved by creating the missing directories by hand, with FTP. This can happen, for example, when storing Java classes.

References

A good guide to expect programming can be found in the book by Don Libes.
Home