Differences

This shows you the differences between two versions of the page.

--- software:python [2016/03/31 12:26]
meesters [Selectively Eliminate Attribute Access]
+++ — (current)
@@ Line 1: / Line 1: @@
-====== Python ======
-===== Available versions =====
-Currently, the following versions of Python are installed and usable using the specified modulefile:
-^ Version   ^ Additional Packages ^ Modulefile ^
-| //2.6.6// | //None// | //None (System default at ''/usr/bin/python'')// |
-| 2.7.7     | pip, virtualenv, virtualenvwrapper, bsub, [[http://www.scipy.org/stackspec.html|The SciPy Stack]]((NumPy, SciPy, matplotlib, IPython, pandas, SymPy)), Cython, scikit-learn | ''Python/2.7.7'' |
-| 3.3.5     | pip, virtualenv, virtualenvwrapper, [[http://www.scipy.org/stackspec.html|The SciPy Stack]]((NumPy, SciPy, matplotlib, IPython, pandas, SymPy)), Cython, scikit-learn | ''Python/3.3.5'' |
-| 3.4.1     | pip, virtualenv, virtualenvwrapper, [[http://www.scipy.org/stackspec.html|The SciPy Stack]]((NumPy, SciPy, matplotlib, IPython, pandas, SymPy)), Cython, scikit-learn | ''Python/3.4.1'' |
-| 3.5       | pip, virtualenv, virtualenvwrapper, [[http://www.scipy.org/stackspec.html|The SciPy Stack]]((NumPy, SciPy, matplotlib, IPython, pandas, SymPy)), Cython, scikit-learn | ''Python/3.5_mkl'' |
-We recommend to **avoid Python 2.6.6** since we can provide better support for the versions that we have installed manually.
-**Note**: The within module ''Python/3.5_mkl'' numpy is linked against [[https://software.intel.com/en-us/intel-mkl|Intel's MKL]].
-If you need additional Python packages, you can easily install them yourself either [[#home_directory|"globally" in your home directory]] or [[#using_virtualenv|inside of a virtual environment]].
-===== Additional packages =====
-In general, having a personal Python environment where you can install third-party packages (without needing root priviliges) yourself is very easy. The preparation steps needed on Mogon are described below.
-While the first variant is already sufficient, using virtualenvs, we recommend using [[#using_virtualenvs|virtualenvs]] since they are a lot easier to work with.
-Virtualenvs can also be shared between users if created in your groups project directory.
-==== Home directory ====
-First, create some directories in which installed packages will be placed:
-<code bash>
-$ mkdir -p ~/.local/bin
-$ mkdir -p ~/.local/lib/python<VERSION>/site-packages
-</code>
-Then add the created ''bin'' directory to your ''PATH'' in your ''.bashrc'' file and source it:
-<code bash>
-$ echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
-$ source ~/.bashrc
-</code>
-Now create a configuration file for ''easy_install'' and ''pip'', the Python package management tools:
-<code bash>
-$ echo -e '[easy_install]\nprefix = ~/.local' > ~/.pydistutils.cfg
-$ mkdir -p ~/.pip
-$ echo -e '[install]\nuser = true' > ~/.pip/pip.conf
-</code>
-If you now use ''easy_install'' or ''pip'', it will automatically install packages to the correct paths in your home directory.
-==== Using virtualenv ====
-A so called virtualenv can be seen as an isolated, self-contained Python environment of third-party packages. \\
-Different virtualenvs do not interfere with each other nor with the system-wide installed packages.
-It is advised to make use of [[http://www.virtualenv.org/en/latest/|virtualenv]] in Python, especially if you intend to install different combinations or versions of various Python packages. Virtualenvs can also be shared between users if created in your groups project directory.
-If you are using Python 2.6.6, you need to install ''virtualenv'':
-<code bash>
-$ easy_install virtualenv
-Searching for virtualenv
-Reading http://pypi.python.org/simple/virtualenv/
-Best match: virtualenv 1.10.1
-[...]
-Processing dependencies for virtualenv
-Finished processing dependencies for virtualenv
-</code>
-We need to remove the easy_install configuration file created above, since the path set there would interfere with virtualenv:
-<code bash>
-$ rm ~/.pydistutils.cfg
-$ rm ~/.pip/pip.conf
-</code>
-Now you can simply create, activate, use, deactivate and destroy as many virtualenvs as you want:
-=== Create ===
-Creating a virtualenv will simply set up a directory structure and install some baseline packages:
-<code bash>
-$ virtualenv ENV
-New python executable in ENV/bin/python
-Installing Setuptools...done.
-Installing Pip...done.
-</code>
-With virtualenvs, you can even make each virtualenv use its own version of the Python interpreter:
-<code bash>
-$ virtualenv --python=/usr/bin/python2.6 --system-site-packages ENV2.6
-$ virtualenv --python=/cluster/Apps/Python/<VERSION>/bin/python --system-site-packages ENV<VERSION>
-</code>
-If you want to install the pre-installed third-party packages (numpy, scipy, matplotlib, etc.) yourself, just omit the ''--system-site-packages'' parameter when calling virtualenv.
-=== Activate ===
-To work in a virtualenv, you first have to activate it, which sets some environment variables for you:
-<code bash>
-$ source ENV/bin/activate
-(ENV)$ # Note the name of the virtualenv in front of your prompt - nice, heh?
-</code>
-=== Use ===
-Now you can use your virtualenv - newly installed packages will just be installed inside the virtualenv and just be visible to the python interpreter you start from within the virtualenv:
-<code bash>
-(ENV)$ easy_install requests
-Searching for requests
-Reading https://pypi.python.org/simple/requests/
-Best match: requests 1.2.3
-[...]
-Processing dependencies for requests
-Finished processing dependencies for requests
-</code>
-or
-<code bash>
-(ENV)$ pip install requests
-Downloading/unpacking requests
-  Downloading requests-1.2.3.tar.gz (348kB): 348kB downloaded
-  Running setup.py egg_info for package requests
-Installing collected packages: requests
-  Running setup.py install for requests
-Successfully installed requests
-Cleaning up...
-</code>
-And now compare what happens with the python interpreter from inside the virtualenv and with the system python interpreter:
-<code bash>
-(ENV)$ python -c 'import requests'
-(ENV)$ /usr/bin/python -c 'import requests'
-Traceback (most recent call last):
-  File "<string>", line 1, in <module>
-ImportError: No module named requests
-</code>
-=== Deactivate ===
-Deactivating a virtualenv reverts the activation step and all its changes to your environment:
-<code bash>
-(ENV)$ deactivate
-$
-</code>
-=== Destroy ===
-To destroy a virtualenv, simply delete its directory:
-<code bash>
-$ rm ENV
-</code>
-==== virtualenvwrapper ====
-Using multiple virtualenvs can be made much more user friendly using [[http://virtualenvwrapper.readthedocs.org/|virtualenvwrapper]].
-If you are using Python 2.6.5, you can install and configure it using
-<code bash>
-$ easy_install --prefix=$HOME/.local virtualenvwrapper
-$ echo 'source $HOME/.local/bin/virtualenvwrapper.sh' >> ~/.bashrc
-</code>
-If you are using any other version of Python, virtualenvwrapper is already installed and you just need to
-<code bash>
-$ echo 'source /cluster/Apps/Python/<VERSION>/bin/virtualenvwrapper.sh' >> ~/.bashrc
-</code>
-Re-login to apply the changes.
-====== Load Environment Modules (module load [mod]) ======
-To load environment modules in python:
-<code python>
-execfile('/usr/share/Modules/init/python.py')
-module('load','gcc/4.8.2')
-module('load','software/bioinf/samtools/0.1.19')
-</code>
-From Python 3.4.1 onwards we enabled on mogon a //modules// module ;-), e.g.
-<code python>
-import modules
-modules.module('load', 'Java/jdk1.8.0_25')
-import os
-os.environ['JAVA_HOME'] # will be '/cluster/Apps/Java/jdk1.8.0_25'
-</code>
-This, of course, requires an environment, where the ''--system-site-packages''-option has been employed during the set up of your (currently active) python environment.
-====== Job submission ======
-For python you can use the maybe basic but friendly bsub package from: https://github.com/brentp/bsub
-<code python>
-from bsub import bsub
-BAM2FQ = "bam2fq --input %s .."
-STAR = "star --align .."
-SAM2BAM = "samtools view .."
-for dataset in datasets:
-  bam2fq = bsub("bam2fastq", R='span[hosts=1] affinity[core(1)]', app='Reserve1G',  n=1, q='long', W='2:00' )
-  bam2fq = bam2fq( BAM2FQ % dataset )
-  star = bam2fq.then(   STAR,    job_name="STAR_%s" % dataset,    R='span[hosts=1] affinity[core(6)]', app='Reserve30G', n=1, q='long', W='8:00' )
-  sam2bam = star.then(  SAM2BAM, job_name="SAM2BAM_%s" % dataset, R='span[hosts=1] affinity[core(1)]', app='Reserve10G', n=1, q='long', W='3:00' )
-  print "First job_id:" + bam2fq.job_id
-  print "Last job_id:" + sam2bam.job_id
-  last = sam2bam.job_id
-print "still running? %s" % ( "yes" if bsub.poll(last) else "no" )
-</code>
-====== Performance Hints ======
-Many of the hints are inspired by [[http://chimera.labs.oreilly.com/books/1230000000393/ch14.html#_problem_239|O'Reilly's Python Cookbook chapter on performance]]. We only discuss very little here explicitly, it is worth reading this chapter. If you need help getting performance out of Python scripts contact us.
-===== Profiling and Timing =====
-Better than guessing is to profile, how much time a certain program or task within this program takes. Guessing bottlenecks is a hard task, profiling often worth the effort. The above mentioned Cookbook covers this chapter.
-===== Regular Expressions =====
-Avoid them as much you can. If you have to use them, compile them, prior to any looping, e.g.:
-<code python>
-import re
-myreg = re.compile('\d')
-for stringitem in list:
-   re.search(myreg, stringitem)
-   # or
-   myreg.search(stringitem)
-</code>
-===== Use Functions =====
-A little-known fact is that code defined in the global scope like this runs slower than code defined in a function. The speed difference has to do with the implementation of local versus global variables (operations involving locals are faster). So, if you want to make the program run faster, simply put the scripting statements in a function (also: see [[http://chimera.labs.oreilly.com/books/1230000000393/ch14.html#_problem_239|O'Reilly's Python Cookbook chapter on performance]]).
-The speed difference depends heavily on the processing being performed.
-===== Selectively Eliminate Attribute Access =====
-Every use of the dot (.) operator to access attributes comes with a cost. Under the covers, this triggers special methods, such as __getattribute__() and __getattr__(), which often lead to dictionary lookups.
-You can often avoid attribute lookups by using the from module import name form of import as well as making selected use of bound methods. See the illustration in [[http://chimera.labs.oreilly.com/books/1230000000393/ch14.html#_problem_239|O'Reilly's Python Cookbook chapter on performance]].
-===== Too many print statements =====
-Regardless of the change of the print statement in Python 2.x to a function in Python 3.x, output is still flushed.
-To avoid constant flushing and use buffered output instead, either use Python's ''logging'' module instead as it supports buffered output. An alternative is to write to ''sys.stdout'' and only flush in the end of a logical block.
-===== Compile Code!!! =====
-Remember that every Python Module on Mogon comes with [[http://cython.org/|Cython]]. Cython is an optimising static compiler for both the Python programming language and the extended Cython programming language.
-While we cannot give a comprehensive intro in this wiki document, we recommend using Cython whenever possible and give this little example:
-Imaging you have a (tested) script, you need to call frequently. Then create modules your main script can import and write a setup script like this:
-<code python>
-# script: setup.py
-#!/usr/bin/env python
-from distutils.core import setup
-from distutils.extension import Extension
-from Cython.Distutils import build_ext
-named_extension = Extension(
-    "name of your extension",
-    ["directory_of_your_module/<module_name1>.pyx",
-     "directory_of_your_module/<module_name2>.pyx"],
-    extra_compile_args=['-fopenmp'],
-    extra_link_args=['-fopenmp'],
-    include_path = ['/cluster/Apps/Python3.4/include/python3.4m']
-)
-setup(
-    name = "some_name",
-    cmdclass = {'build_ext': build_ext},
-    ext_modules = [named_extension]
-)
-</code>
-Replace ''named_extension'' with a name of your liking, and fill-in all place holders. You can now call the setup-skript like this:
-<code bash>
-$ python ./setup.py build_ext --inplace
-</code>
-This will create a file ''directory_of_your_module/<module_name1>.c'' and a file ''directory_of_your_module/<module_name1>.so'' will be the result of a subsequent compilation step.
-In Cython you can release the global interpreter lock (GIL), see [[http://docs.cython.org/src/userguide/external_C_code.html|this document (scroll down a bit)]], when not dealing with pure python objects.
-In particular [[http://docs.cython.org/src/userguide/numpy_tutorial.html|Cython works with ''numpy'']].
-====== Things to consider ======
-Python is an interpreted language. As such it should not be used for lengthy runs in an HPC environment. Please use the availability to compile your own modules with Cython; consult the relevant [[http://cython.org/|Cython documentation]]. If you do not know how to start, attend a local Python course or schedule a meeting at our local HPC workshop.
-====== Special packages ======
-Please note that we have already installed numpy, scipy and matplotlib in the versions of Python that we provide additionally.
-===== NumPY =====
-http://www.numpy.org/
-When installing NumPY, the first installation attempt fails at exit. Don't worry, the installation is already finished then, but to be sure, you can simply run the command again to see it exiting cleanly.
-Note that NumPY can also be linked against the [[software:mkl|Intel Math Kernel Library]] or the [[software:acml|AMD Core Math Library]]:
-  * MKL: http://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl
-  * ACML: http://mail.scipy.org/pipermail/numpy-discussion/2012-May/062309.html