django user tips

On this page, useful info is gathered for the usage of django.ugent.be.

Server Specs

  • RAM: 64 GiB (max: 64 GiB) DDR3-1600 RDIMM
  • CPU: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
  • HDD: 6 TB (HD3), 547GB (HD1), 275G (HD2)
  • OS: openSUSE LEAP 15.0

Info on recent server usage can be found on http://www.kennybogaert.eu/servers/.

There are 3 disk partitions:
1. HD1 contains the operating system and databases
2. HD2 only databases: such as nr: /media/HD2/
3. HD3 /home directories in /media/HD3/home/

LOGIN

*For Linux users:

Just login via your commandline:

ssh username@django.ugent.be

Or when programs with a graphical user interface need to be loaded (Rstudio, CLC genomics, ..)

ssh -X username@django.ugent.be 

Programs can be run by executing the command:

eg.

For a good text editor:

scite

for ugene:

ugene -ui

or for clc genomics workbench:

clcgenomicswb10

*Windows users:

Windows users may install a free client as alternative. (https://mobaxterm.mobatek.net/)

FILETRANSFER

*For Linux users:

just via commandline

port 22 is open.

VIA SCP(commandline)Copying file to host:

scp /some/local/file.txt username@django.ugent.be:/some/remote/directory/
 Copying file from host to local directory:
scp username@django.ugent.be:/some/remote/file /some/local/directory/
Copying directory from host
scp -r user@django.ugent.be:/some/local/directory/   /some/local/directory/
2. VIA FILEZILLA(gui – sFTP)
host: django.ugent.be
user: yourusername
password: youraccountpassword
port: 22

*Windows users:

Windows users can install a free client: program WinSCP (https://winscp.net/eng/download.php), a free FTP and SCP client.

VNC

Virtual Network Computing (VNC) is a graphical desktop sharing system that to remotely control another computer. On Django a VNC session will start the KDE desktop as a graphic user interface. When you close your VNC viewer on your local desktop all programs will keep running on the server and your server session remains the same when you login the next day.

This is ideal for using program that run a graphic user interface such as UGENE,  CLC genomics workbench or Geneious.

UGENE and CLC work for all users.

Please contact if you wish to use Geneious.
Please note that if one user doesn’t close Geneious after using it, nobody can use it due to limitations of the licence! In this case, launching it from ssh -X (or mobaxterm) is a better habbit.

RealVNC:
install vncviewer from realvnc
https://www.realvnc.com/download/viewer/
1.connect via ssh to django.ugent.be
on linux:
ssh username@django.ugent.be
on windows: use putty (http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html)
2. Once logged in, start a vncserver session on the server using the following command:
vncserver :an_available_vnc_number

or 

vncserver
In the last case you will have to deduce your port number from the output:
(you might want to adapt your screen resolution)
vncserver -geometry 1600x1200 :an_available_vnc_number
3. connect using vncviewer
django.ugent.be:yourvncnumber
encryption: Let VNC choose
push “connect”
vncpassword invullen
Usual problem:
In case you are not able to launch a vnc session and you get an error message telling you there is session running, there is probably already a VNC session running on that port…
Try to kill it.
vncserver -kill :an_available_vnc_number
If it tells you, you don’t have the right to kill it or it doesn’t find the .pid file, it probably is launched by another user.
You can check all vncsessions running (and their ports) for example with the command:
ps -ef | grep "vnc"
Sometimes the graphics of kde may show problems (difficult to click icons, disappearing windows, commands in konsole not visible,… ). In this case you’ll need to kill your vncserver session and start a new one.
If this doesn’t help, you ‘ll have to contact bogaert.kenny@gmail.com. Restarting from runlevel 3 might do the trick (as root).
init 3

init 5 && exit

KDE

You can personalise you KDE desktop, for example:
-Please take notice of the handy widgets to track CPU, RAM and hard disk space. Right-click on the task bar > unlock widgets > add widgets > browse for handy widgets
-To get rid of the annoying apper message “Authentication is required to refresh the system resources”: Click on the application launcher > System settings > System administration > Startup and shutdown > Service Manager > disable “Apper monitor” and press ok. After you have relaunched your server session, this new setting will take effect.

Hard drive usage:

The main hard drive is very small. Therefore it can not be used for data storage. Please don’t leave intermediary files on the server and always keep track of the free disk space (using the kde widget, the df command, …). The server may run out of free space. This results in scripts terminating with an error, and makes the server unusable for programs demanding a lot of temporary storage.

IMPORTANT: Each months, you should receive an automatic email listing all files that are older than 3 months and got categorized as expired. In case there is no other option to clear space and ensure a working server, a script will be run that permanently deletes these expired files… Therefore don’t rely on the server for data storage!

Files in ~/bin can be older of course. The ~/bin can ONLY be used for your binaries though: your personal scripts and programs. Don’t use it for real data storage.

Update:

Django has been upgraded with an extra hard disk, currently mounted at /media/HD3/.
A second small partition has been mounted now at /media/HD2/.

This means we have an extra 6TB. This will alleviate the pressure on the primary partition and will make simultaneous analyses on large datasets more feasible again.

Please work on HD3 instead of HD1 for tasks that demand a lot of temporary space. If HD1 runs out of space the server is not functional!
For most active users a personal directory in HD3 has been made or you may do this yourself.
Please move for example your CLC_data to this partition.
6TB is a lot, but at some point it will run out in of space if we don’t keep track of our usage. This drive is not meant for data storage, but just like the main partition, this partition is temporary working space.

RAM & CPU usage

1. If you overload any server it will crash at some point.  CLC bio in particular and multithreaded processes are very good at doing this.
Please keep track of top and htop if you plan to launch a process on multiple nodes. Or install a CPU & RAM monitoring widget in your KDE environment.
It is a waste to leave nodes open if they are unused. On the other hand use it with some common sense and moderation.
2. Close your VNC server if you don’t use it for a while (Not just the screen but the server session via vncserver -kill). That can save some RAM memory.

NCBI – nr database

NCBI’s nr database is accessible in /home/nr/
blastx -query seq.fasta -db /home/nr/nr  -out output.xml -outfmt 5

List of installed programs

Abyss
BLAST+
blastall (legacy blast)
CD-HIT
CLC genomics workbench
fastq-join
flash (1.2.11)
FASTX toolkit (0.0.13)
Geneious (v8.1.7) (contact to activate the licence)
Qiime1
Qiime2
Lotus  [version 1.38]
mira v4
Mothur (1.32.1)
PEAR (v0.9.5)
Rstudio
Stacks
Trinity
UGENE (v 1.31)
usearch (UParse)

  • usearch [version 5.2.236] as usearch
  • usearch [version 6.1.544] as usearch61
  • usearch [version 8.0.1623_i86] as usearch8
  • usearch [version 8.1.1861_i86] as usearch81

Note: This list is incomplete

Extra notes for specific program and pipelines

CLC workbench Updates

Desktop shortcuts for installed updated versions of CLC are made available in /opt/all_users_updates/ . You can launch them here (as sh script) or copy them here to your ~/Desktop

QIIME (v1.9.1)

add to $HOME/.bashrc the following line

export PATH="/root/anaconda2/bin:$PATH"

Anytime you want to use QIIME after installation with Miniconda, you’ll need to reactivate your qiime1 environment using this command:

source activate qiime1

To exit the virtual environment, simply run the deactivate command:

source deactivate

QIIME2 (v2-2018.6)

add to $HOME/.bashrc the following line

export PATH="/root/anaconda2/bin:$PATH"

Anytime you want to use QIIME after installation with Miniconda, you’ll need to reactivate your qiime1 environment using this command:

source activate qiime2-2018.6

Lotus pipeline:

accessible in path /mnt/lotus_pipeline/ eg. :

perl lotus.pl -h