Computer Cluster for Analysis of Data from SLS and SwissFEL Facilities

The official cluster name is Ra and the cluster will be referenced below by this name.

User Mailing List

All users are strongly invited to subscribe to the ra-users mailing list at https://psilists.ethz.ch/sympa/subscribe/ra-users. where important announcement that may impact users' activity, like downtimes and changes in policy will be sent.

For any problem or further information please contact us at: ra-admins@lists.psi.ch

To be able to use Ra cluster you need :

  • PSI account
  • authorisation to use the Ra cluster.

PSI Account

If you don't have PSI account, please follow this procedure

Authorization for the Ra cluster

Please contact your beamline manager and provide the following information:

  • your PSI account
  • data identifier (Proposal ID or e-account used to collect the data) for the data you need to access.

For the beamline manager:

Login in DUO, select "Beamline Manager" on the left-hand side menu and then "Access to ra-cluster".

For any problem or further information please contact: ra-admins@lists.psi.ch or PSI Helpdesk at helpdesk@psi.ch. Please do not call the admins unless urgent (e.g. beamtime is severely impacted by the issue)

The names of the login nodes are

  • ra-l-004.psi.ch
  • ra-l-005.psi.ch
  • ra-l-006.psi.ch

They are also available under the alias ra.psi.ch. From the PSI subnet, you can connect to login nodes directly via ssh or with NoMachine using ra-nx.psi.ch as target host. From outside the PSI subnet (e.g. when not onsite, or not using VPN), you have two options to connect to RA. 

Please note that for both options (ssh over hop.psi.ch and NoMachine connection through rem-acc.psi.ch) you need Multi-Factor Authentication (MFA) enabled: please follow the instructions here https://www.psi.ch/en/computing/change-to-mfa. 

Option one: connect to the hop.psi.ch system and from there go via ssh to the one of the Ra login node

Password: <password> 
# AFTER login 
$ssh ra-l-006.psi.ch 

For further details see

Option two: (recommended) NoMachine Connection via Remote Access Server

Please follow these instructions

From PSI subnet and using PSI-VPN, one need to use ra-nx as target for NoMachine connection, to reduce load on VPN and network infrastructure.

rem-acc is accessible from PSI subnet, but not from PSI-VPN.

NoMachine hints:

  • Set the display preferences of the NoMachine ("Ctrl+Alt+0") to "Resize remote screen". This will provide a better default resolution matching your monitor to the remote desktop on the Ra login node.
  • In case of heavy graphic application try to run it prepending vglrun before the application name. See yourself the difference by running "glxgears" vs. "vglrun glxgears".
  • If a NoMachine sessions is not being accessed for 5 days it gets automatically terminated.

Login nodes usage limits

Login nodes are the entry point to the Ra cluster and they are the common place for every user. For this reason please avoid to overload the login nodes with CPU or memory intensive job, since for that ones the batch system should be used instead.

 

There are several file systems for different purposes on the Ra cluster:

File systemPathDefault QuotaAccess rightsAccess modeUsed for
HOME/mnt/das-gpfs/home/$USER5 GBuser onlyread-writeuser home directory, code, private data, etc.
SLS raw data/sls/$BEAMLINE-the PI* group onlyread-onlySLS data
work data/das/work/$PGR/$PGROUP4 TBthe PI* group onlyread-writederived data

*) PI -- principal investigator or Main Proposer

Example:

  • /sls/X06SA/Data10/e15874    # contains the raw data for the PI group 15874  
  • /das/work/p15/p15874             # contains the derived data for the PI group 15874


To see the group members for a given group use the getent group command, e.g.:

  getent group p15874


To check the quota for home, use homequota command being on one of the login node:

  homequota

Work area

The p-groups inside the work area are split between the so called internal and external area. The default quota per p-group only applies to the external p-groups. The internal ones are members of one and only one unit, that provides a certain amount of space on the work area to all its p-groups. The p-group of a unit don't have a default quota so a single p-group could fill up all the space on the work area for its unit.

To see to which unit a pgroup is member of and if is internal or external:

[talamo_i@ra-l-002 ~]$ /das/support/users/space_usage/pgroup_info p17277 
Name: p17277 
Unit: tomcat 
Kind: internal 
Used: 63 GB Members: talamo_i,gsell 
Unit quota: 500 TB

Files permission and ownership

All files inside a specific p-group directory are considered to be owned by the specific pgroup, meaning that the unix group of the files should be the p-group and that the group should have read-write access, ie. all its members have read-write access. For this reason a regular process checks the files permission and ownership and fixes them.

The fix happens automatically every hour, but in case you would need to change permission/ownership sooner, you can run the following commands for a file:

chmod g+w file chgrp your-p-group file


And in case you want to do it recursively on a directory:

chmod -R g+ws dir chgrp -R your-p-group dir

The data analysis and advanced development software is available via PSI Environment Modules (Pmodules) system. Use the following commands to manage your environment:

module list                # show the loaded modules 
module avail               # show the available modules  
module search              # show all modules in hierarchy  
module help                # if you do not remember what to do    
module add module_name      # add module_name  
module rm module_name       # remove module_name                


Example:

  module load matlab/2015b   # load Matlab 2015b


There are MX-beamline specific environment configuration files in the /etc/scripts/ directory:

 

  • /etc/scripts/mx_sls.sh - the default configuration for the analysis of SLS data
  • /etc/scripts/mx_fel.sh - the default configuration for the analysis of data from FEL sources

Source the corresponding configuration file to use the predefined settings, for example:

  source /etc/scripts/mx_sls.sh


The environment settings done with the module are effective only in the current shell and all its children processes.

You may wish to edit the .bashrc file in home directory to make permanent changes of your environment: see the comments therein for more details.

Scientific Applications

The list of the available software includes: Matlab, python, intel and gcc compilers, Fiji, standard MX software like xds, shelx, hkl2map, adxv, CBFlib, ccp4, dials, mosflm, phenix, software used for serial crystallography like crystfel, cheetah .... For the complete list use the following commands:

module use unstable module avail

The Ra login node should be used mainly for development, small and quick tests, and work with the graphical applications.

For CPU intensive work, the compute nodes must be used. For further details refer to the documentation accessible from within PSI: https://hpc-sysadmins.gitpages.psi.ch/ra/compute.html

 

The documentation is also accessible from the Nomachine session

ra-help.png

More detailed documentation on use of slurm on Ra, description of NoMachine setup is available in the internal PSI page  (accessible from PSI subnet, PSI-VPN or in the NoMachine session (Ra help))