1 - INTRODUCTION
dsupdt is a utility for performing periodic dataset updates by calling dsarch to archive data onto the CISL Geoscience Data Exchange (GDEX) Servers.
The application supports the following capabilities:
Configure a dataset’s update controls, local files, and remote files
Download or copy data files from remote servers or other local areas
Invoke specialist-defined routines to validate and convert downloaded data
Build local files from remote files, applying tar and/or compress operations automatically as needed
Archive local files onto the GDEX Servers via dsarch
Remove local and temporary files after data files have been archived successfully
Check availability of files on remote servers
Initiate update actions through the centralized daemon dscheck, via a cron job, or manually
A dataset must already be configured for dsarch before it can be set up for periodic updates with dsupdt. See the dsarch help document for information on placing a dataset under dsarch control.
Once update information has been configured in GDEXDB, dsupdt downloads and archives data in the following stages:
Server Files on remote server or other local area --DOWNLOAD/COPY------>
Remote Files staged in the local working area ------VALIDATE/CONVERT--->
Remote Files ready to build Local Files ------------TAR/COMPRESS/BUILD->
Local Files ready for archive ----------------------DSARCH------------->
Files on the GDEX Servers
If a server file name is not specified explicitly, it defaults to the remote file name. For one-to-one matches between remote and local files, the remote file name defaults to the local file name when not otherwise specified.
A local file update record is the minimum configuration required to update data files for a given dataset. Add remote file configuration records when the remote file name differs from the local file name, or when several remote files are needed to build a single local file.
Update control records determine which data-processing computers may run update actions. Without an update control record, updates must be started manually or through cron jobs on a specific computer.
To prevent specialists from accidentally updating data files using configuration records owned by others, only the owning specialist may execute dsupdt against a given update record. Validation is also performed to guard against configuring the wrong dataset: any input file used for update configuration must begin with the dataset number in the format ‘dNNNNNN.*’, where ‘*’ matches one or more valid filename characters. In addition, dsupdt verifies that the specialist is authorized for the given dataset. Execution stops if the specialist is not listed as an owner, unless Mode option -MD (-MyDataset) is supplied, which forces dsupdt to proceed regardless.
If an error occurs while updating an individual data file, that file’s update is skipped and dsupdt continues with the remaining updates. Restarting dsupdt resumes from where it left off for any previously unfinished files. An email notice is sent to the specialist who initiated the update. By default, the email contains detailed information on successful updates along with any error messages.
The sections below first describe general usage of dsupdt, then present detailed descriptions of each option, with examples interspersed throughout.