====== TSD software ======

Available software on TSD is described [[https://www.uio.no/english/services/it/research/sensitive-data/help/software.html|here]]

===== gio open <filename > =====

"gio open" automatically chooses appropriate program to open a given file type, including text files, .pdf and figures. You can use this on a cluster (e.g. p33-appn-norment01), but then you need to connect with x11 forwarding
(i.e. with "ssh -Y p33-appn-norment01", or configure putty with -x11 and also start xming if you work from Windows).

===== Using putty, Xming and WinSCP on TSD =====

Putty, Xming and WinSCP are standard tools allowing to connect from Windows to Linux environment.
These tools are available on TSD Windows login nodes, and they allow you to connect to Linux machines within TSD.
For example, you could connect from Windows login node to the app node.
Note that this is an alternative solution: you also have an option of connecting directly to TSD linux login nodes from https://view.tsd.usit.no/, and then use ``ssh`` to connect to the app node.

Xming allows to use applications with graphical interface via x11.
See here for more details: http://www.geo.mtu.edu/geoschem/docs/putty_install.html

===== Using licensed software on TSD(SPSS/MATLAB/STATA++) =====

Every UiO TSD-user must ask **tsd-drift@usit.uio.no** to be added to the ex: TSD **SPSS-user group**

Users with other affiliations than UiO must //bring their own license// to be confirmed by TSD(see link above)

//.sav files can also be opened in R and PSPP//

===== Using Matlab on TSD =====

Only TSD-users affiliated with UiO can use MATLAB on TSD.
We know this is big limitation - it had been discussed with TSD, but it's very hard to find a good solution due to MATLAB licensing scheme. Alternative potential solutions are to use Octave (https://www.gnu.org/software/octave/) or compile Matlab scripts into binaries outside TSD, to make a self-contained binary (executable) that is independent from Matlab licensing.

UiO-affiliated TSD users must ask **tsd-drift@usit.uio.no** to be added to the TSD **p01-matlab-user**. Within TSD, run ``groups`` command to check your groups.

It's possible to run MATLAB on a remote host (e.g. p33-appn-norment01) with GUI.

<code>
ssh -Y p33-appn-norment01   # note -Y, to enable trusted x11 forwarding
module load MATLAB/2020b (or other versions)
matlab
</code>

For non-UiO users it is possible to buy Matlab license, and then ask TSD to enable your MATLAB access using this license. This is called "Bring your own license". There is no formal procedure for this - please contact USIT TSD for support if you need this.



===== Using R on TSD =====
R and RStudio is available both on Windows VM and Linux VM

Installation of R-packages is described in [[https://www.uio.no/english/services/it/research/sensitive-data/use-tsd/software/r/index.html|TSD R help]].

Tip: if running R on appnodes or submit nodes, please use an up-to-date R module: module load R/<version>. They come with many preinstalled packages and the necessary dependencies for building most other packages. Bioconductor R package are best installed using the R-bundle-Bioconductor/<version> modules


<code>
ssh -Y p33-appn-norment01   # note -Y, to enable trusted x11 forwarding
module load R/<latest version>
module spider libGLU/<latest version> (note: Rstudio also starts without this module)
rstudio
</code>



**TMPDIR on appnodes:
**

By default, the TMPDIR environmental variable is set to '/scratch' for all users in p33-appn-norment01 and '/storage/rtmp/' in p697-appn-norment01 (so that users don't accidentally fill up the /tmp space on these nodes which cause login problems for all users)
===== Using Python on TSD =====

Using Python on TSD is briefly described
[[https://www.uio.no/english/services/it/research/sensitive-data/help/software/python.html]].

On Windows VMs, python is  available through Anaconda (but to the best of my knowledge there is no official mirror of Anaconda packages within TSD - so you have whatever is pre-installed with Anaconda, but everything beyond that you need to install yourself).

On linux VM, you  need to load python via ''module''. This only works on machines where ''/cluster''  is mounted (as of now, that's p33-submit and the RHEL7 login node).

If you are using ''python3'', it's nice to deploy everything into your custom virtual environment (''venv'', read more here https://docs.python.org/3/library/venv.html). Once you ''source activate'' your environment, install python package with ''pip3 install'' without ''--user''.

<code>
ssh -Y p33-appn-norment01   (This command is for thin link users)
module load  Python/3.7.4-GCCcore-8.3.0 (This command is for VM ware Horizon users)

# clean install jupyter in a new environment
python3 -m venv /cluster/projects/p33/users/<user>/py3       # update <user> with your TSD username
source /cluster/projects/p33/users/<user>/py3/bin/activate   # best to have this on /cluster


pip3 install numpy
pip3 install pandas
pip3 install scipy
pip3 install matplotlib
pip3 install statsmodels
pip3 install numdifftools
pip3 install matplotlib_venn
pip3 install jupyter
</code>

Note that the above commands MUST NOT include <nowiki>--index-url=file:///shared/pypi/mirror/web/simple</nowiki>; this is an old mirror on TSD that is no longer maintained - only old versions of the packages are available from this mirror, and it's going to be deprecated soon.

When everything is installer, you may use python on a new terminal  (and in your Colossus jobs) as follows
<code>
module load  Python/3.7.4-GCCcore-8.3.0
source /cluster/projects/p33/users/<user>/py3/bin/activate
</code>

The stuff below is quite advanced - most likely you don't need it. But if you are interested in using jupyter on a remote VM (e.g. p33-submit or p33-appn-norment01), remember to that you connect to it as follows.
<code>
# on p33-appn-norment01
jupyter notebook --no-browser --port=8888 (This command is for VM ware Horizon users) 
(when the given port is busy due to previous sessions, try another port and change accordingly in the below command)

# directly on the login node 
ssh -N -f -L localhost:8888:localhost:8888 p33-ofrei@p33-appn-norment01 (This command is for thin link users)

# open your browser and connect to
localhost:8888
</code>

To connect to jupyter session from Windows you need to configure port forwarding in Putty. To enable this, do as on the screenshot below. To clarify: you'll probably need one "usual" session in Putty (i.e. without port forwarding) to establish "ssh" connection to your remote server and start a jupyter notebook. The host name for establish the remote session is "p33-appn-norment01". Then, you'll need a second session in Putty with port forwarding as on the screenshot below; also note that putty config windows has to be closed for the putty session to work). Some basics are described here: https://stackoverflow.com/questions/46276612/remote-access-jupyter-notebook-from-windows

UPD: screenshot below shows different ports: 8888 and 8889. It's much less confusing to use the same port, i.e. to start jupyter on 8888, and forward this port to 8888 on Windows machine. This works, there is no conflict here because ports are on different machines. 

{{:puttyportforwarding.png?300|}} 

===== Visual Studio Code ====

You can install Visual Studio Code following instructions [[https://huhuidong.wordpress.com/2018/12/13/how-to-install-visual-studio-code-in-linux-without-root-or-sudo/|here]]

  * Download compiled executable binaries from https://code.visualstudio.com/download (choose tar.gz for Linux)
  * Unpack the tar.gz ball into a directory, e.g., ~/bin/ (assume this directory is included in $PATH)
  * Create a soft link (in ~/bin) to the binary file code in ~/bin/VSCode-linux-x64/bin/ ("ln -s ~/bin/VSCode-linux-x64/bin/code ~/bin/").  Note: do not use the binary with the same name code in ~/bin/VSCode-linux-x64/ .
  * If ~/bin/ is in your $PATH, now you may start VS code anywhere in a new terminal ("code")


===== Singularity ========

Excellent tutorial about making R singularity containers is available here:
https://wiki.norment.uiocloud.no/dokuwiki/r_singularity_containers

Great examples how to use Singularity on TSD are described here:
https://github.com/comorment/gwas

Official TSD documentation for singularity containers is here:
https://www.uio.no/english/services/it/research/sensitive-data/use-tsd/hpc/software/singularity.html

Some additional issues and workarounds are described here:
https://github.com/norment/tsd_issues/issues/22


====== Nextflow =====

Nextflow module have certain problems.
In p697 project we got a local installation of the Nextflow \\
/cluster/projects/p697/nextflow-20.10.0-all \\
To run nextflow scripts: \\
export NXF_OFFLINE='TRUE' \\
nextflow=/cluster/projects/p697/nextflow-20.10.0-all \\
$nextflow run yourscript.nf \\

In p33 project the installation location is: \\
/cluster/projects/p33/nextflow-20.10.0-all \\

Instructions to run nextflow workflow could be found at: \\
[[https://github.com/norment/moba_qc_imputation/blob/master/users/zr/readme_nextflow.txt]] \\
[[https://www.nextflow.io/docs/latest/index.html]]

===== Multiple instances of VMWare on MAC OC =====

Here is a way to run multiple VMware sessions on MAC OS:
Open a terminal and run 

<code>
open -n VMware\ Horizon\ Client.app/
</code>

===== Installing EEGLAB plugin for microstates=====

To install the EEGLAB plugin for microstates http://www.thomaskoenig.ch/index.php/software/microstates-in-eeglab

  * Download the code from the link provided above 
  * Upload to TSD via the data portal: data.tsd.usit.no
  * Then copy the zip file from data/durable/file-import/somedir/plugin.zip to /cluster/projects/p33/groups/imaging/EEG/toolboxes/eeglab14_1_2b/plugins/
  * unzip the file
 
One can use this procedure to install other plugins too according to the EEGlab documentation.