Python
What is Python?
Python is an interpreted, high-level and general-purpose programming language. Python's design philosophy emphasizes code readability with its notable use of significant indentation.
What is Anaconda?
Anaconda is a freemium open source distribution of the Python and R programming languages for large-scale data processing, predictive analytics, and scientific computing, that aims to simplify package management and deployment. Its package management system is conda.
Which Python versions are available?
Version | module name | Notes |
|---|---|---|
3.8.6 | python/3.8.6 | optimized for avx, avx2 and avx512 |
3.8.6 | anaconda3/2020.07 | |
3.7.4 | anaconda3/2019.10 | same as cent7 + anaconda/python_3.6 |
List of Python modules on Sol (switch to mpich for Hawk)
[2021-08-06 07:15.02] ~
[alp514.sol](1009): module av
-------------------------------------------------------- /share/Apps/lusoft/share/spack/lmod/avx2/linux-centos8-x86_64/mvapich2/2.3.4-ksbev3r/Core ---------------------------------------------------------
adios2/2.6.0 hdf5/1.10.7 (D) kahip/2.00 netcdf-cxx/4.2 (D) openfoam/2006 petsc/3.14.1 scotch/6.0.10
boost/1.74.0 (D) horovod/0.21.3-cuda (D) mumps/5.3.3 netcdf-fortran/4.5.3 (D) paraview/5.8.1 py-mpi4py/3.0.3 superlu-dist/6.3.1
fftw/3.3.8 (D) hypre/2.20.0 netcdf-c/4.7.4 (D) opencv/4.2.0 parmetis/4.0.3 pytorch/1.7.0-cuda (D) tensorflow/2.4.1-cuda (D)
-------------------------------------------------------------------- /share/Apps/lusoft/share/spack/lmod/avx2/linux-centos8-x86_64/Core --------------------------------------------------------------------
angsd/0.933 canu/1.8 gatk/4.1.0.0 libarchive/3.4.3 ngstools/2019-06-24 picard/2.20.8 snpeff/5.0
arpack-ng/3.7.0 cuda/10.2.89 gcc/9.3.0 magma/2.5.4 nlopt/2.6.2 pilon/1.22 stacks/2.55
bamtools/2.5.1 cuda/11.0.2 gmap-gsnap/2020-06-01 masurca/3.4.2 nvhpc/20.9 py-pyparsing/2.4.2 star/2.7.0e
bartender/1.1 cuda/11.1.0 (D) gnuplot/5.2.8 metis/5.1.0 octave/5.2.0 pythia8/8303 subread/2.0.0
bayescan/2.1 cudnn/8.0.4.30-11.0-linux-x64 gsl/2.5 miniasm/2018-3-30 openbabel/3.0.0 python/3.8.6 (L) suite-sparse/5.7.2
bayestraits/3.0.2 eigen/3.3.8 hal/2.1 minimap2/2.14 openblas/0.3.10 qhull/2020.1 superlu-mt/3.1
bgc/1.03 exonerate/2.4.0 hdf5/1.10.7 mpich/3.3.2 openjdk/1.8.0_222-b10 qrupdate/1.1.2 superlu/5.2.1
blast-plus/2.9.0 express/1.5.2 hisat2/2.2.0 mvapich2/2.3.4 (L) openjdk/11.0.2 (D) r/4.0.3 swig/4.0.2 (D)
blat/35 fastjet/3.3.3 intel-mkl/2020.3.279 nccl/2.7.8-1 openmpi/4.0.5 rmcprofile/6.7.9 tabix/2013-12-16
boost/1.74.0 fastqc/0.11.7 intel/19.0.3 netcdf-c/4.7.4 paml/4.9j root/6.20.08 trimmomatic/0.39
bowtie/1.3.0 fastx-toolkit/0.0.14 intel/20.0.3 (D) netcdf-cxx/4.2 pear/0.9.11 rsem/1.3.1 trinity/2.12.0
bowtie2/2.4.1 fftw/3.3.8 jellyfish/2.2.7 netcdf-fortran/4.5.3 perl/5.32.0 salmon/0.14.1 vcftools/0.1.15
bwa/0.7.17 freebayes/1.1.0 kallisto/0.46.2 ngmlr/0.2.7 phast/1.4 samtools/1.10 vesta/3.4.6
------------------------------------------------------------------ /share/Apps/lusoft/share/spack/lmod/default/linux-centos8-x86_64/Core -------------------------------------------------------------------
anaconda3/2019.10 axel/2.16.1 graphviz/2.44.1 libxp/1.0.3 ncdu/1.13 pwgui/6.1 scons/3.1.2 tcl/8.6.10 valgrind/3.15.0
anaconda3/2020.07 (D) cmake/3.18.4 julia/1.5.2 motif/2.3.8 parallel/20200822 qcachegrind/20.08.0 screen/4.8.0 tk/8.6.10 vmd/1.9.3
aria2/1.35.0 ddd/3.3.12 krb5/1.18.2 nano/5.6.1 povray/3.7.0.10 rclone/1.53.3 swig/4.0.2 tmux/3.1b
--------------------------------------------------------------------------- /share/Apps/share/Modules/lmod/applications/licensed ---------------------------------------------------------------------------
abaqus/2019 cactus/1.2.3 comsol/5.4 guppy/3.1.5-gpu guppy/4.0.14 (D) mathematica/12.0 simmodeler/8.0 xds/feb2021 (D)
ansys/20.2 cactus/1.3.0 (D) gaussian/g09 (D) guppy/3.1.5 gurobi/8.1.1 matlab/R2020a stata/15 xds/jan2020
ansys/21.1 (D) ccp4/7.0 gaussian/g16 guppy/4.0.14-gpu knitro/10.3.0 phenix/1.16 sysweld/2020.5
---------------------------------------------------------------------------------- /share/Apps/share/Modules/lmod/default ----------------------------------------------------------------------------------
cent7 gpu hawk hawkgpu sol solgpu soltools
--------------------------------------------------------------- /share/Apps/share/Modules/lmod/linux-centos8-x86_64/py_venv/pythonapps/avx2 ----------------------------------------------------------------
all htseq/0.11.2-m5yqkbo matplotlib/3.3.3-dttvy5f (D) pandas/1.1.4-sdbbcxk pytorch/1.7.0-cuda seaborn/0.9.0-7xrppb4
biopython/1.73-aoor5vo jupyter/1.0.0-7ew3qb6 mdanalysis/1.0.0-tfngrft phonopy/1.10.0-iuaj2xc scikit-build/0.10.0-vww4qdm six/1.15.0-obed3ik
cclib/1.5.post1-wv3arnb jupyterlab/2.2.7-ln6bc44 moltemplate/2.5.8-iclz2ic pip/20.2-ig2u7ky scikit-image/0.17.2-qhu4bnf tensorflow/2.4.1-cuda
cython/0.29.21-trlkbz5 keras/2.2.4-cuda-rihsv3p multiqc/1.7-xi6ehki porechop/0.2.4-tgrse4w scikit-learn/0.23.2-nnpi22y theano/1.0.4-cuda-srh7rqj
dask/2.16.0-hcixvbx lxml/4.5.2-2d2z266 numba/0.50.1-orbwfra pyparsing/2.4.2-yktitpw scikit-optimize/0.5.2-xngjbiq virtualenv/16.7.6-yh3ujrd
horovod/0.21.3-cuda matplotlib/2.2.5-fimnl7l numpy/1.19.4-hyxetpw pyspark/3.0.1-f3qh2hw scipy/1.5.4-fbs3ffk virtualenvwrapper/4.8.2-vozt6md
Where:
L: Module is loaded
D: Default Module
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
List of Conda environments
[2021-08-06 07:18.07] ~
[alp514.sol](1012): module av
-------------------------------------------------------------------- /share/Apps/lusoft/share/spack/lmod/avx2/linux-centos8-x86_64/Core --------------------------------------------------------------------
angsd/0.933 canu/1.8 gatk/4.1.0.0 libarchive/3.4.3 ngstools/2019-06-24 picard/2.20.8 snpeff/5.0
arpack-ng/3.7.0 cuda/10.2.89 gcc/9.3.0 magma/2.5.4 nlopt/2.6.2 pilon/1.22 stacks/2.55
bamtools/2.5.1 cuda/11.0.2 gmap-gsnap/2020-06-01 masurca/3.4.2 nvhpc/20.9 py-pyparsing/2.4.2 star/2.7.0e
bartender/1.1 cuda/11.1.0 (D) gnuplot/5.2.8 metis/5.1.0 octave/5.2.0 pythia8/8303 subread/2.0.0
bayescan/2.1 cudnn/8.0.4.30-11.0-linux-x64 gsl/2.5 miniasm/2018-3-30 openbabel/3.0.0 python/3.8.6 suite-sparse/5.7.2
bayestraits/3.0.2 eigen/3.3.8 hal/2.1 minimap2/2.14 openblas/0.3.10 qhull/2020.1 superlu-mt/3.1
bgc/1.03 exonerate/2.4.0 hdf5/1.10.7 mpich/3.3.2 openjdk/1.8.0_222-b10 qrupdate/1.1.2 superlu/5.2.1
blast-plus/2.9.0 express/1.5.2 hisat2/2.2.0 mvapich2/2.3.4 openjdk/11.0.2 (D) r/4.0.3 swig/4.0.2 (D)
blat/35 fastjet/3.3.3 intel-mkl/2020.3.279 nccl/2.7.8-1 openmpi/4.0.5 rmcprofile/6.7.9 tabix/2013-12-16
boost/1.74.0 fastqc/0.11.7 intel/19.0.3 netcdf-c/4.7.4 paml/4.9j root/6.20.08 trimmomatic/0.39
bowtie/1.3.0 fastx-toolkit/0.0.14 intel/20.0.3 (D) netcdf-cxx/4.2 pear/0.9.11 rsem/1.3.1 trinity/2.12.0
bowtie2/2.4.1 fftw/3.3.8 jellyfish/2.2.7 netcdf-fortran/4.5.3 perl/5.32.0 salmon/0.14.1 vcftools/0.1.15
bwa/0.7.17 freebayes/1.1.0 kallisto/0.46.2 ngmlr/0.2.7 phast/1.4 samtools/1.10 vesta/3.4.6
------------------------------------------------------------------ /share/Apps/lusoft/share/spack/lmod/default/linux-centos8-x86_64/Core -------------------------------------------------------------------
anaconda3/2019.10 axel/2.16.1 graphviz/2.44.1 libxp/1.0.3 ncdu/1.13 pwgui/6.1 scons/3.1.2 tcl/8.6.10 valgrind/3.15.0
anaconda3/2020.07 (L,D) cmake/3.18.4 julia/1.5.2 motif/2.3.8 parallel/20200822 qcachegrind/20.08.0 screen/4.8.0 tk/8.6.10 vmd/1.9.3
aria2/1.35.0 ddd/3.3.12 krb5/1.18.2 nano/5.6.1 povray/3.7.0.10 rclone/1.53.3 swig/4.0.2 tmux/3.1b
--------------------------------------------------------------------------- /share/Apps/share/Modules/lmod/applications/licensed ---------------------------------------------------------------------------
abaqus/2019 cactus/1.2.3 comsol/5.4 guppy/3.1.5-gpu guppy/4.0.14 (D) mathematica/12.0 simmodeler/8.0 xds/feb2021 (D)
ansys/20.2 cactus/1.3.0 (D) gaussian/g09 (D) guppy/3.1.5 gurobi/8.1.1 matlab/R2020a stata/15 xds/jan2020
ansys/21.1 (D) ccp4/7.0 gaussian/g16 guppy/4.0.14-gpu knitro/10.3.0 phenix/1.16 sysweld/2020.5
---------------------------------------------------------------------------------- /share/Apps/share/Modules/lmod/default ----------------------------------------------------------------------------------
cent7 gpu hawk hawkgpu sol solgpu soltools
------------------------------------------------------------------- /share/Apps/share/Modules/lmod/linux-centos8-x86_64/py_venv/2020.07 --------------------------------------------------------------------
conda/biofluids conda/bioinformatics conda/biostats conda/chem conda/mldl conda/nlp conda/r (D)
Where:
L: Module is loaded
D: Default Module
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
Seminar Slides
Basic Programming (A. Pacheco, S. Joshi)
Visualization (A. Pacheco, S. Joshi)
What's the difference between python and anaconda3 modules?
Python can be built by compiling the source code or by installing binaries from the Anaconda Python Distribution. Compiled python i.e. python/3.8.6 is built using GNU compiler and optimized for the underlying CPU architecture. Anaconda Python is most likely unoptimized or built for generic x86_64 architecture and can be slower than compiled python.
If you use tensorflow from one of the anaconda modules, you might see the following warning "Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA". SSE4/AVX provide double/quadruple the FLOPs compared to generic x86_64 while AVX2/AVX512 double/quadruples the AVX FLOPS at the expense of cpu speed. If you use python packages from the anaconda3 modules, you are most likely using unoptimized binaries. To use MPI enabled versions of Tensorflow, PyTorch, Py4MPI and Horovod, unload the intel modules and reload mvapich2/mpich/openmpi.