==========================================================================
This file describe briefly different useful tools to administrate Mobyle.
==========================================================================

In this document we refer to MOBYLEHOME as the folder which contains the Mobyle installation core files.
 
1- mobdeploy : to deploy the programs and viewers interfaces
=============================================================

Mobyle allows administrators to deploy programs that are either *local* or *imported* and now viewers that are *local*. 
The `mobdeploy` script is used to *deploy* them on the server, making them available to the public. 
This script also handles their *indexing*.
Note: In the following, we will talk about Services which means the Programs & the Viewers.

Overview
:::::::::

Once a Mobyle server has been installed in a **MOBYLEHOME** directory on the system, the program definitions (i.e., the xml files that describe them to the Mobyle system) are stored in two directories:
        - the MOBYLEHOME/Services/ directory which contains program (Programs/) and viewer (Viewers/) definitions that the administrator has downloaded from distributions,
        - and the MOBYLEHOME/Local/Programs directory which contains custom program definitions that have either been created de novo by the server owner, or that he modified to fit his own needs.

In order to deploy the services, the service definitions are first tested to check that these definitions are valid. 
Then, there is a predeployment step: the valid service definitions are copied to a temporary directory (tmpservices) the time necessary to generate the index.
(In case of validation problem, the previous service definition installed is temporary kept, if it exists.)
Finally, the whole content of this directory is transferred to a web-accessible directory defined by the server configuration in the **DOCUMENT_ROOT/PROGRAMS_URL** in case of Programs 
or **DOCUMENT_ROOT/VIEWERS_URL** for Viewers.
Therefore, these files will be directly accessible from the Web on their corresponding url. This document refers to this directory as **HTDOCS**.

The imported program definitions are downloaded *via* the Web from the **HTDOCS** directory of the remote servers. 
These program definitions are stored in **HTDOCS/programs/imports** directory.

The programs and viewers index are stored in the **HTDOCS**/index directory.


Configuration
::::::::::::::

* PROGRAMS *

The configuration of published/imported services is located in the Local/Config/Config.py file.

The deployment of local services (programs, workflows, viewers ) is configured by the **LOCAL_DEPLOY_INCLUDE**, and **LOCAL_DEPLOY_EXCLUDE** directives. 
 LOCAL_DEPLOY_INCLUDE and LOCAL_DEPLOY_EXCLUDE are python dictionaries with 3 keys programs, workflows and viewers associated to 3 lists of names.
 these defined repectively the programs, workflows and viewers and can include jokers.

e.g.::
  LOCAL_DEPLOY_INCLUDE = { 'programs' : [ 'golden', 'clustalw*', 'dnadist', 'fitch' ] ,
                           'workflows': [ '*' ] ,
                           'viewers'  : [ 'jalview' , 'varna' ]
                           }
  LOCAL_DEPLOY_EXCLUDE={'programs' : [ 'clustalw-profile' ] ,
                        'workflows': [ '*' ] ,
                        'viewers'  : [ 'jalview' , 'varna' ]
                        }
* VIEWERS *

The important rule about viewer is that the viewer dependencies have to be available within a directory named like the
viewer xml description in MOBYLEHOME/Services/Viewers/.
For instance, for a viewer called 'jalview', MOBYLEHOME/Services/Viewers/ directory has to contain a 'jalview.xml' file and
a 'jalview' directory where are the corresponding dependencies: the archive 'jalviewApplet.jar'.
If the dependency directory is missing, the viewer won't be deployed.



The exportation of services to other portals is defined by the **EXPORTED_SERVICES** directive, which lists these services.

e.g.::
  EXPORTED_SERVICES = [ 'golden', 'abiview' ]

The importation of services from other portals is defined by the **PORTALS** directive, 
a dictionary that defines external portals the following way:
    - the key is the nickname of the portal, as it will appear in the programs list of the portal
    - for each portal:
      + url is the path to the directory that contains all the cgis (such as job_submit.py).
      + help is the e-mail of the remote portal maintainer, that receives help requests about his job.
      + repository is the path to the directory that contains all the published programs.
      + services is a dictionary with 2 keys
        - 'programs' associated to the list of imported programs.
        - 'workflows' associated to the list of imported workflows.

e.g.::
  PORTALS={'portal1': {
            'url': 'http://otherdomain.fr:port/cgi-bin/MobylePortal',
            'help' : 'help@otherdomain.fr',
            'repository': 'http://otherdomain.fr:port/MobyleData/',
            services: { 'programs': ['clustalw-multialign'],
                        'workflows' : ['workflow1_of_portal1']
                      }
            }
        }



        
Usage and options
:::::::::::::::::

The ``mobdeploy`` script is located in the **$MOBYLEHOME/Tools** directory. 
Before to run it, you should make sure that your current user has the permissions to read and write the files that belong to 
the Mobyle user (e.g., apache user).

Here are the available commands:
           deploy   : deploy services      available options [-s --server, -p --programs , -w --workflows , -v --viewers , -f --force , -V --verbose]
           clean   : clean the repository  available options [-s --server, -p --programs , -w --workflows , -v --viewers , -f --force , -V --verbose]
           index   : generate indexes      available options [-s --server, -p --programs , -w --workflows , -v --viewers, -V --verbose ]
And options:
           -h --help
           -p --programs : specify the names (comma separated list) of the programs on which the command is applied.
                           The keyword 'all' mean all programs as defined in Config.py
           -w --workflows: specify the names (comma separated list) of the workflows on which the command is applied.
                           The keyword 'all' mean all workflows as defined in Config.py
           -v --viewers  : specify the names (comma separated list) of the viewers on which the command is applied.
                           The keyword 'all' mean all viewers as defined in Config.py
           -s --server :   specify the names (comma separated list) of the servers on which the command is applied.
                           The keyword 'all' mean all servers as defined in Config.py
           -f --force : force to deploy a program even it is not imported. 
                           This option is for debugging purpose only and shall not be used otherwise. 
           -V --verbose : increase the verbosity (there is 3 level of verbosity: warning , info , debug . default is warning)

Below are a few examples of how you can use the mobdeploy utility:

  mobdeploy  deploy                           install all programs, workflows and viewers from all servers according to the configuration.
  mobdeploy -p blast2 deploy                  deploy the *blast2* program from all servers.
  mobdeploy -p blast2 clean                   remove the *blast2* from all servers.
  mobdeploy -s portal1 deploy                 deploy all the program(s) and workflows imported from *portal1* server according to its configuration.
  mobdeploy --server portal1 -p all deploy    deploy all the program(s) imported from *portal1* server according to its configuration.
  mobdeploy -s local -w pretty_tree deploy    deploy the *pretty_tree* workflow from the local server.
  mobdeploy -s local -p blast2 deploy         deploy the *blast2* program from the *local* server.
  mobdeploy -s mylab -p seqret,blast2 deploy  deploy the remote program *seqret* and *blast2* from the *mylab* server.
  mobdeploy -v all deploy                         deploy every viewers and its dependencies.
  mobdeploy -v jalview,jarna deploy 	      deploy the *jalview* and *jarna* viewers and its dependencies
  mobdeploy -p index                          only update the program index (in case the index file has been lost for instance)



2- mobjobw : to list the running job 
=====================================

mobjob permit to monitor quickly the jobs which are on your batch system.
the informations displayed are:
 - the key of the job
 - the batch system used ( several system could be mixed in Mobyle e.g. SGE and system )
 - the queue name if it defined
 - the status of the job : 
         - running : the job is running,  
         - pending : the job has been queued by the batch system
         - hold    : the job has been hold by the administrator
 - the jobName
 - the email of the user 
 - the submission date
 - the IP of the user 
 - the full qualified name of the user
 - the command line 

--------------------------------------------------------------------------------
jobkey -- batch system/queue -- status
jobName -- user email -- submission date -- requestor IP\full qualified name
unix command line
--------------------------------------------------------------------------------

e.g.::
--------------------------------------------------------------------------------
K07333355412998 -- SGE/None -- running
phyml -- email@domain.gov -- 01/13/09 18:59:51 -- 111.111.11.11\name.domain.ext
phyml -i infile_data.phylipi -d aa -b 1000 -m JTT -a e
--------------------------------------------------------------------------------
O31005884758996 -- SGE/short -- running
drawtree -- email2@domain.edu -- 01/15/09 12:47:30 -- 250.250.250.250\UNKNOWN
ln -s protpars.outtree intree && drawtree <drawtree.params && ln -s plotfile plotfile.ps


3- mobkill : to kill a job. 
===========================

mobkill take one or more jobKeys and kill the corresponding jobs 
e.g.::
mobkill N32454430408001 V00863529172897


4- mobclean : to clean the job or/and anonymous session tree
=============================================================

mobclean is a tool to clean your job result and anonymous session directories. By default, it cleans
your directories(job & a. session) depending on your configuration parameters (the ones of your config.py).
The tool allows you to make the cleaning separately and to run it on simulation mode (-n).

Here are the available options:
        -J          Clean Job Results.
        -S          Clean anonymous sessions.
        -d <val>    Delete directories older than <val> days ( positive integer value ).
        -h          Display this help and exit.
        -v          With error messages (verbose).
        -n          In simulation mode (dry_run), nothing is cleaned.
        -l <val>    Path to the Logfile where put the logs.
   
And some examples:
        mobclean 	--> By default, the cleaning of jobs & sessions is done
        mobclean -S -d 2 to clean the anonymous sessions older than 2 days
        mobclean -J -S that is equivalent to the default run
        mobclean -l /var/log/mobyle/cleaner_log : put the logs (error and warning) of the mobclean 
   run into this log file.
        mobclean --help
        
The cleaning of old jobs can be automated with a cron using mobclean.


5- mobtypes : analyse the xml types
====================================

mobtypes help to ensure the coherence in xml typing between the programs xml definitions, which is
 of the utmost importance for the chaining. 
 mobtypes take programs xml file list as argument. If mobtypes is used without option , it build 
a repository with all "xml types" (see doc about typing in mobyle) found and their corresponding 
python classes. -if a repository is specify ( -r option ) mobtypes display the differences between 
this repository and the xml types.

usage: mobtypes < --ref path to a datatype repository > xmlpaths
        options:
            -h or --help  ... Print this message and exit.
            -r or --ref   ... the path to reference
e.g :: 
mobtypes MOBYLEHOME/*.xml  
python class    | xml class                      | description
AbstractText    | AceAssembly                    | 
AbstractText    | AncestorsFile                  | 
AbstractText    | BambeTree                      | 
...

mobtypes -r path_to_a_repository MOBYLEHOME/*.xml
    
--- ../Programs/mobyle_xml_types.txt
+++ xml list
- Binary          | AbiTraceFile                   | binary trace file produce by abi sequencer  
+ Binary          | HmmBinProfile                  | 
  
- Binary          | AbiTraceFile                   | binary trace file produce by abi sequencer 
means that this line is in the repository and this AbiTraceFile/Binary is not in the xml programs analysed

+ Binary          | HmmBinProfile                  |
means that HmmBinProfile/Binary is found in programs xml analysed but is not present in repository



6- session0_9_updater.py : to update sessions from mobyle 0.9 to mobyle 0.95.1
==============================================================================

update the sessions located in Config  USER_SESSIONS_PATH from a Mobyle v0.9 
to be compliant with the Mobyle v0.95.1
option :
   -f ( --force ) to force the update even an xml version of the session already exists.
   -n ( --dry-run ) translate the session.dump in xml, but do not write them.
        
e.g ::
session0_9_updater 2&> update.log

7- mobvalid: validate program definition xml files
==================================================

mobvalid is a utility that automatically validates a program description, with 
respect to the Mobyle format and constraints defined in the Relax-NG and 
Schematron definitions. When creating or editing any program definition, 
remember to validate it with this tool before installing it.

Usage: mobvalid [OPTION]... FILE... 
        Validate Mobyle program description files, each FILE being the path to the file.
    
        option:
            -h or --help  ... Print this message and exit.
