====== How to work on SAGA supercomputer ======

Connect using ``ssh <usersname>@login.saga.sigma2.no``.

Place your files in ``/cluster/projects/nn9114k/<usersname>`` (create your own folder if it doesn't exist yet).
To check available disk space, use ``dusage -p nn9114k`` command. In total we have 5 TB on SAGA /cluster partition.

To use matlab on SAGA, you need to export a LICENSE file (https://documentation.sigma2.no/apps/matlab.html).
For UiO users this can be done as follows:

<code>
ssh login.saga.sigma2.no
srun --ntasks=1 --cpus-per-task 8 --mem-per-cpu=4G --time=00:30:00 --qos=devel --account=NN9114K --pty bash -i 
module load MATLAB/2019a
export MLM_LICENSE_FILE=1719@lisens-prod01.uio.no
matlab
</code>

The first line allows you to get a machine where you can work interactively. 
If you need it for more than 30 minutes, you can replace ``--time=00:30:00 --qos=devel`` with ``--time=24:00:00`` (24 hours). Getting a machine is important because you are not supposed to run matlab on a login node on SAGA. More info about interactive nodes is here: https://documentation.sigma2.no/jobs/interactive_jobs.html .
Also, not sure why but support also gave me this info: ``The license server name is: matlab@uio``.

To compile MiXeR on Saga:

<code>
module load Boost/1.71.0-GCC-8.3.0 Python/3.7.4-GCCcore-8.3.0 CMake/3.12.1
pip install numpy scipy pandas numdifftools --only-binary numpy scipy pandas --user 

git clone --recurse-submodules https://github.com/precimed/mixer.git
mkdir mixer/src/build && cd mixer/src/build
cmake ..
make -j16 bgmg
</code>

To run MiXeR you need to run 
<code>
module load Boost/1.71.0-GCC-8.3.0 Python/3.7.4-GCCcore-8.3.0 CMake/3.12.1
</code>

====== How to work on ABEL supercomputer ======

The information below is obsolete as ABEL superformuter is not longer in production. It is replaced with SAGA.

This page provides useful links related to Abel supercomputer.
  * Login to Abel with ''ssh abel.uio.no''.
  * [[http://www.uio.no/english/services/it/research/hpc/abel/help/user-guide/|Abel's User Guide]].
  * Working with [[http://www.uio.no/english/services/it/research/hpc/abel/help/user-guide/modules.html|Modules]]. Example: ''module load plink2'' enable you to use ''plink'' tool.

<code>
-bash-4.1$ plink
-bash: plink: command not found
-bash-4.1$ module load plink2
-bash-4.1$ plink
PLINK v1.90b3.31 64-bit (3 Feb 2016)       https://www.cog-genomics.org/plink2
(C) 2005-2016 Shaun Purcell, Christopher Chang   GNU General Public License v3

  plink [input flag(s)...] {command flag(s)...} {other flag(s)...}
  plink --help {flag name(s)...}
</code>

  * [[https://www.uio.no/english/services/it/research/hpc/abel/help/software/Python%203.html|Python 3]]. Note that ''pandas'' is not available by default. You have to install it once by running

<code>
module load python3 & pip3 install --user pandas
</code>

  * [[http://www.uio.no/english/services/it/research/hpc/abel/help/user-guide/interactive-logins.html|Interactive Logins]].
  Example: ''qlogin --account=NN9114K --cpus-per-task 4 --mem-per-cpu=4G'' gives you 4-core machine where you can run process for up to ''16G'' memory limit.

<code>
-bash-4.1$ qlogin --account=NN9114K --cpus-per-task 4 --mem-per-cpu=4G
salloc: Pending job allocation 16093329
salloc: job 16093329 queued and waiting for resources
salloc: job 16093329 has been allocated resources
salloc: Granted job allocation 16093329
srun: Job step created
bash-4.1$
</code>
  
Note that after your ''qlogin'' job get allocation your shell is already logged into remote node. You may use ''squeue -u <username>'' to fine out node name, and login to the same node from another terminal:

<code>
bash-4.1$ squeue -u oleksanf
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          16093329    normal   qlogin oleksanf  R       2:05      1 c12-29
b
bash-4.1$ ssh c12-29
Warning: Permanently added 'c12-29' (RSA) to the list of known hosts.
oleksanf@c12-29's password:
Rocks Compute Node
Rocks 6.2 (SideWinder)
Profile built 17:31 30-Oct-2016

Kickstarted 17:44 30-Oct-2016
-bash-4.1$
</code>

  * [[http://www.uio.no/english/services/it/research/hpc/abel/help/user-guide/data.html|Managing Data on Abel]]. Best way to work from interactive node is to put everything under ''/work/users/<username>''. Once you are done copy important files to your home folder ''/usit/abel/u1/<username>''.
  * [[https://slurm.schedmd.com/cpu_management.html|SLURM Cpu Management]] --- read here to find out the difference between ''--cpus-per-task'' and ''--tasks-per-node''.
  * Clone code from https://github.com or https://github.uio.no. Note that Abel's nodes have very good link to access internet.
<code>
-bash-4.1$ git clone https://github.com/precimed/python_convert.git
Cloning into 'python_convert'...
remote: Counting objects: 140, done.
remote: Compressing objects: 100% (33/33), done.
remote: Total 140 (delta 17), reused 0 (delta 0), pack-reused 107
Receiving objects: 100% (140/140), 12.71 MiB | 3.86 MiB/s, done.
Resolving deltas: 100% (63/63), done.

-bash-4.1$ git clone https://github.uio.no/norment/biostats.git
Cloning into 'biostats'...
Username for 'https://github.uio.no': oleksanf
Password for 'https://oleksanf@github.uio.no':
remote: Counting objects: 289, done.
remote: Total 289 (delta 0), reused 0 (delta 0), pack-reused 288
Receiving objects: 100% (289/289), 648.30 KiB, done.
Resolving deltas: 100% (138/138), done.
</code>

Some tricks:

  * Mount our NIRD project to Abel --- works only for login node, but not for worker nodes. For example:
<code>
mkdir ~/nird && sshfs $USER@login.nird.sigma2.no:/projects/NS9114K/ ~/nird
</code>
You may need to create SSH keys on abel and install them on NIRD with ''ssh-copy-id $USER@login.nird.sigma2.no''.
Command above assumes that your user name on Abel is the same as on NIRD.

Advanced stuff

  * look at ''/etc/slurm/slurm.conf'' to see SLURM configuration on abel. It helps to understand how Abel sets priority to jobs (https://slurm.schedmd.com/priority_multifactor.html)
  * install https://github.com/ubccr/stubl - a useful collection of tools that tell more about queueing on slurm


