El mes pasado actualizamos nuestros servidores de deep learning a Ubuntu 16.04 LTS. Dado que los principales entornos de deep learning ni los drivers de cuda y cuDNN soportan de forma directa ubuntu 16.04, en este post proporcionamos un resumen de los pasos realizados para poder configurar Theano, Caffe y TensorFlow sobre esta última versión de LTS de ubuntu. Esta guía es también válida para otras versiones de ubuntu simplemente ignorandolos pasos relacionados con la adaptación de ficheros para compilación a gcc 5.x. Incluimos también algunos ficheros de configuración/test que se describen en esta guía (link). A continuación incluimos la guía en inglés:

1. Install prerequisites:

This install prerequisites required to build the different dependencies and frameworks.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential
sudo apt-get autoremove
sudo apt-get install git
git config --global user.name $MYNAME
git config --global user.email $MYMAIL

2. Install nvidia graphics driver:

      • Download drivers Nvidia driver
      • Start a text console ctrl + alt + F1
      • Stop X Server
        sudo service lightdm stop
      • Prevent noveau driver to be used:
        sudo nano /etc/modprobe.d/blacklist.conf

        blacklist nouveau
        blacklist lbm-nouveau
        options nouveau modeset=0
        alias nouveau off
        alias lbm-nouveau off
        

        sudo update-initramfs -u

      • Reboot and:
        • Start a text console ctrl + alt + F1
        • Stop xserver
      • Run script for downloaded Nvidia driver
        sudo ./NVIDIA-Linux-x86_64-361.42.run
      • Notes:
        • In order to avoid compatibility issues with NVIDIA OpenGL driver and some cards (such as GTX920M) we recommend not to install the NVIDIA OpenGL driver. In this way, the X server will use the NVIDIA card with the generic Intel driver for visualization purposes and you can access to CUDA normally:

          • sudo ./NVIDIA-Linux-x86_64-361.42.run –Z –no-opengl-files
        • Nvidia card data info can be obtained by command: nvidia-smi.

3. Install CUDA:

  • Install cuda drivers:
    • Run CUDA SDK executable
      sudo ./cuda_7.5.18_linux.run --override

      • Notes:
        • Do not install the graphics driver as we already have installed it before.
        • Override option is used to go around (by now) the unsupported compiler error.
  • Get rid off the gcc version error at Ubuntu 16.04:
    First, we have to solve the incompatibility of default gcc version at Xenial. To do that, error at host_config.h have to be removed manually:sudo nano /usr/local/cuda/include/host_config.h
    At line: 115 comment out error://#error — unsupported GNU version! gcc versions later than 4.9 are not supported!
  • Configure paths:
    At this stage we need to modify $LD_LIBRARY_PATH variable to let the S.O. access the installed libraries.
    We will do these modifications for the user’s bash and for the Application Environment Setup (/etc/profile.d/)
    For the shake of completeness we include also modifications required for caffe and conda.
    Note that we have installed all packages at /usr/progtools.
  • Content for /etc/profile.d/add_ld_library_path.sh:
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/progtools/caffe-nv/distribute/lib
    export PYTHONPATH=/usr/progtools/caffe-nv/python:$PYTHONPATH
    
  • Content added to ~/.bashrc:
    export PATH=$PATH:/usr/local/cuda/bin
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/lib:/usr/lib/x86_64-linux-gnu
    export PYTHONPATH=/usr/progtools/caffe-nv/python:$PYTHONPATH
    export CAFFE_HOME=/usr/progtools/caffe-nv
    export PATH=/usr/progtools/anaconda2/bin:$PATH
    

4. Install cuDNN:

  • Download cuDNN linux libraries.
  • Uncompress tar
    tar -xvf cudnn-7.5-linux-x64-v5.0-ga.tgz
  • Copy files to cuda library
    sudo cp cudnn.h /usr/local/cuda-7.5/include/
    sudo cp ./libcudnn* /usr/local/cuda-7.5/lib64/
  • Run ldconfig from /lib64 to update libs cache
    sudo ldconfig /usr/local/cuda/lib64

5. Install Theano:

Now it is time to install Theano and our Conda environment. This section is partially based on this Donald Kinghorn’s post

  • Configure and build openblas:sudo apt-get install gfortran
    git clone https://github.com/xianyi/OpenBLAS
    cd OpenBLAS
    make FC=gfortran
    sudo make PREFIX=/usr/local install
  • Install Conda environment:
    Conda is a really flexible package and environment system that will ease development and dl framework version changes on our developoment systems.
    Please, note that we will install conda at /usr/progtools/anaconda2)

    • Download conda.
    • Run conda .sh and install it:
      ./conda.sh
      source activate root
    • Update conda:
      conda update conda
      conda update anaconda
      conda update --all
      conda install pydot
      conda update theano

      • If you want latest Theano version instead run:
        pip install --upgrade --no-deps git+git://github.com/Theano/Theano.git)
    • Remove mkl:
      conda install nomkl
      conda install nomkl numpy scipy scikit-learn numexpr
      conda remove mkl mkl-service
    • Solve gblas problems:
      conda install libgfortran
      conda install openblas
  • Install and set Theano parameters:
    • Create ~/.theanorc file with the following content:
          [global]
          device = gpu  
          floatX = float32
          [blas]
          ldflags = -L/usr/local/lib -lopenblas
          [nvcc]
          fastmath = True
          flags=-D_FORCE_INLINES
          [cuda]
          root = /usr/lib/nvidia-cuda-toolkit
      
    • Run python test_theano.py:
          from theano import function, config, shared, sandbox
          import theano.tensor as T
          import numpy
          import time            
          vlen = 10 * 30 * 768  # 10 x #cores x # threads per core
          iters = 1000            
          rng = numpy.random.RandomState(22)
          x = shared(numpy.asarray(rng.rand(vlen), config.floatX))
          f = function([], T.exp(x))
          print f.maker.fgraph.toposort()
          t0 = time.time()
          for i in xrange(iters):
              r = f()
          t1 = time.time()
          print 'Looping %d times took' % iters, t1 - t0, 'seconds'
          print 'Result is', r
          if numpy.any([isinstance(x.op, T.Elemwise) for x in f.maker.fgraph.toposort()]):
              print 'Used the cpu'
          else:
              print 'Used the gpu'
      

6. Install TensorFlow:

  • Install TensorFlow (No current CuDNN version support)
    $ pip install --ignore-installed --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.8.0-cp27-none-linux_x86_64.whl
  • In order to install TensorFlow from sources and to support latest CuDNN we have to build it from sources link:
    • Install Bazel:
      • Add Bazel distribution URI as a package source (one time setup)
        echo "deb http://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
        curl https://storage.googleapis.com/bazel-apt/doc/apt-key.pub.gpg | sudo apt-key add -
      • Install java 8
        sudo add-apt-repository ppa:webupd8team/java
        sudo apt-get update
        sudo apt-get install oracle-java8-installer
      • Install bazel
        sudo apt-get update && sudo apt-get install bazel
        sudo apt-get upgrade bazel
        sudo apt-get install python-numpy swig python-dev python-wheel
    • Configure and build tensorflow:
      • First we have to allow it working with gcc 5.x
        Edit file $tensorflow_sources_folder/third_party/gpus/crosstool/CROSSTOOL
        Add this line:
        cxx_flag: "-D_FORCE_INLINES"
        cxx_flag: "-D_MWAITXINTRIN_H_INCLUDED"
        below any tool_path { name: «gcc» path: «clang/bin/crosstool_wrapper_driver_is_not_gcc» }
      • and build it:
        ./configure
        bazel build -c opt --config=cuda //tensorflow/cc:tutorials_example_trainer
        bazel-bin/tensorflow/cc/tutorials_example_trainer --use_gpu
      • Build pip installation:
        bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
        bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
        pip install /tmp/tensorflow_pkg/tensorflow-0.8.0-py2-none-any.whl
    • Internal Test:
      cd tensorflow/models/image/mnist
      python convolutional.py
  • Test TensorFlow:
    Run python test_tensorflow.py.

    import tensorflow as tf
    hello = tf.constant('Hello, TensorFlow!')
    sess = tf.Session()
    print(sess.run(hello))
    a = tf.constant(10)
    b = tf.constant(32)
    print(sess.run(a + b))
    

7. Make Caffe and pyCaffe:

This section helps you to install Caffe and pyCaffe. It is based on wikidot blog’s post.

  • Install prerequisites:
  • Common dependencies:
    sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler
    sudo apt-get install --no-install-recommends libboost-all-dev
    sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
  • Glog:
    wget https://google-glog.googlecode.com/files/glog-0.3.3.tar.gz
    tar zxvf glog-0.3.3.tar.gz
    cd glog-0.3.3
    ./configure
    make && make install
  • gflags:
    wget https://github.com/schuhschuh/gflags/archive/master.zip
    unzip master.zip
    cd gflags-master
    mkdir build && cd build
    export CXXFLAGS="-fPIC" && cmake .. && make VERBOSE=1
    make && make install
  • lmdb:
    git clone https://github.com/LMDB/lmdb
    cd lmdb/libraries/liblmdb
    make && make install
  • opencv:
    • Download opencv:
      sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
      sudo apt-get install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev
      git clone https://github.com/Itseez/opencv.git
      cd ~/opencv
    • Modify CMakeLists.txt to override compiler version error:
      set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -D_FORCE_INLINES")
    • Build opencv:
      mkdir release
      cd release
      sudo cmake -DBUILD_TIFF=ON -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local /home/apicon/opencv
      make
      sudo make install
  • Build Caffe:
    • Download caffe into /usr/progtools:
      git clone https://github.com/NVIDIA/caffe.git caffe-nv
      
    • Edit Makefile:
      Change MakeFile to support inline declarations and avoid gcc 5.0 errors adding this line in line 52:

          COMMON_FLAGS += -D_FORCE_INLINES
      

    Add also opencv libraries:

            LIBRARIES += opencv_core opencv_highgui opencv_imgproc opencv_imgcodecs
    
    • Edit Makefile.config:
      ## Refer to http://caffe.berkeleyvision.org/installation.html
      # Contributions simplifying and improving our build system are welcome!
      
      # cuDNN acceleration switch (uncomment to build with cuDNN).
      USE_CUDNN := 1
      
      # CPU-only switch (uncomment to build without GPU support).
      # CPU_ONLY := 1
      
      # uncomment to disable IO dependencies and corresponding data layers
      # USE_OPENCV := 0
      # USE_LEVELDB := 0
      # USE_LMDB := 0
      
      # uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
      #   You should not set this flag if you will be reading LMDBs with any
      #   possibility of simultaneous read and write
      # ALLOW_LMDB_NOLOCK := 1
      
      # To customize your choice of compiler, uncomment and set the following.
      # N.B. the default for Linux is g++ and the default for OSX is clang++
      # CUSTOM_CXX := g++
      
      # CUDA directory contains bin/ and lib/ directories that we need.
      CUDA_DIR := /usr/local/cuda
      # On Ubuntu 14.04, if cuda tools are installed via
      # "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
      # CUDA_DIR := /usr
      
      # CUDA architecture setting: going with all of them.
      # For CUDA < 6.0, comment the *_50 lines for compatibility.
      CUDA_ARCH := -gencode arch=compute_20,code=sm_20 
              -gencode arch=compute_20,code=sm_21 
              -gencode arch=compute_30,code=sm_30 
              -gencode arch=compute_35,code=sm_35 
              -gencode arch=compute_50,code=sm_50 
              -gencode arch=compute_50,code=compute_50
      
      # BLAS choice:
      # atlas for ATLAS (default)
      # mkl for MKL
      # open for OpenBlas
      BLAS := open
      # Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
      # Leave commented to accept the defaults for your choice of BLAS
      # (which should work)!
      BLAS_INCLUDE := /usr/local/include
      BLAS_LIB := /usr/local/lib
      
      # Homebrew puts openblas in a directory that is not on the standard search path
      #BLAS_INCLUDE := $(shell brew --prefix openblas)/include
      #BLAS_LIB := $(shell brew --prefix openblas)/lib
      
      # This is required only if you will compile the matlab interface.
      # MATLAB directory should contain the mex binary in /bin.
      # MATLAB_DIR := /usr/local
      # MATLAB_DIR := /Applications/MATLAB_R2012b.app
      
      # NOTE: this is required only if you will compile the python interface.
      # We need to be able to find Python.h and numpy/arrayobject.h.
      PYTHON_INCLUDE := /usr/include/python2.7 
              /usr/lib/python2.7/dist-packages/numpy/core/include
      # Anaconda Python distribution is quite popular. Include path:
      # Verify anaconda location, sometimes it's in root.
      ANACONDA_HOME := $(HOME)/anaconda2
      PYTHON_INCLUDE := $(ANACONDA_HOME)/include 
               $(ANACONDA_HOME)/include/python2.7 
               $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include 
      
      # We need to be able to find libpythonX.X.so or .dylib.
      #PYTHON_LIB := /usr/lib
      PYTHON_LIB := $(ANACONDA_HOME)/lib
      
      # Homebrew installs numpy in a non standard path (keg only)
      PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
      PYTHON_LIB += $(shell brew --prefix numpy)/lib
      
      # Uncomment to support layers written in Python (will link against Python libs)
      #WITH_PYTHON_LAYER := 1
      
      # Whatever else you find you need goes here.
      INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
      LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib
      LIBRARY_DIRS += /usr/lib/x86_64-linux-gnu/
      LIBRARY_DIRS += $(ANACONDA_HOME)/lib
      
      # If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
      #INCLUDE_DIRS += $(shell brew --prefix)/include
      #LIBRARY_DIRS += $(shell brew --prefix)/lib
      
      # Uncomment to use `pkg-config` to specify OpenCV library paths.
      # (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
      # USE_PKG_CONFIG := 1
      
      BUILD_DIR := build
      DISTRIBUTE_DIR := distribute
      
      # Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
      # DEBUG := 1
      
      # The ID of the GPU that 'make runtest' will use to run unit tests.
      TEST_GPUID := 0
      
      # enable pretty build (comment to see full commands)
      Q ?= @ 
      
      # shared object suffix name to differentiate branches
      LIBRARY_NAME_SUFFIX := -nv
      
    • Make coffe:
      make all -j16
      make test -j16
      make runtest -j16
    • Troubleshotting:
      If we find problems with libflags:
      Uninstall libgflags:
      sudo apt-get remove -y libgflags
      Delete make install versions:
      sudo rm -f /usr/local/lib/libgflags.a /usr/local/lib/libgflags_nothreads.a
      sudo rm -rf /usr/local/include/gflags
      Clean Caffe build:
      cd //caffe
      make clean
      Re-install libgflags package:
      sudo apt-get install -y libgflags-dev
      Rebuild Caffe
  • Build pyCaffe
    • Install python dependencies:
      From $CAFFE_HOME/python directory run:
      for req in $(cat requirements.txt); do pip install $req; done
    • Export $PYTHONPATH on your /etc/profile.d/xxx.sh and reboot:
      export PYTHONPATH=/python:$PYTHONPATH
      
    • Make pycaffe:
      make pycaffe
      make distribute
  • Export generated libraries:
    • They have been generated at:
      /caffe/caffe-nv/distribute/lib/
      
    • We add the PATHS to allow it being discovered by conda:
      • This will go to /etc/profile.d/add_ld_library_path.sh:
          export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/progtools/caffe-nv/distribute/lib
          export PYTHONPATH=/usr/progtools/caffe-nv/python:$PYTHONPATH
        
      • This will go to ~/.bashrc:
          export PYTHONPATH=/usr/progtools/caffe-nv/python:$PYTHONPATH
          export CAFFE_HOME=/usr/progtools/caffe-nv
        
    • Check that caffe works running provided test_caffe.py:
      import caffe    
      caffe.set_device(0)
      caffe.set_mode_gpu()        
      #output = (input - kernel_size) / stride + 1        
      net = caffe.Net('conv.prototxt', caffe.TEST)        
      print net.inputs
      

7. Build Nvidia digits:

We followed the instructions from NVIDIA github for DIGITS

  • Install Prerequisites:
    sudo apt-get install python-dev python-pip graphviz
    sudo apt-get install python-pil python-numpy python-scipy python-protobuf python-gevent python-flask python-flaskext.wtf gunicorn python-h5py
  • Download digits to /usr/progtools:
    git clone https://github.com/NVIDIA/DIGITS.git digits
  • Install python requirements:
    pip install -r requirements.txt
  • Launch server:
    • Start Development Server:
      ./digits-devserver
    • Start Production Server:
      ./digits-server

8. Create new users on the system:

Just setting the /usr/progtools folder to the group developers we can generate new pre-configured environments to any user by following these easy steps:

  • Copy the content of provided /tools folder into /usr/progtools
  • Copy the add_ld_library_path.sh file into /etc/profile.d folder
  • To create a new user:
    • Run sudo ./create_deep_user
    • A new user will be created with:
      • Configured PATHs for all the installed frameworks
      • A project folder containing test python scripts for theano, tensorflow and caffe.
      • a soft link to the progtool folder
    • Just remember to include the proper python interpreter (conda) on PyCharm or on your favorite IDE.

The provided script:

    #!/bin/bash


    if [[ $1 == "" ]]; then
       echo "Usage: sudo ./create_deep_user.sh ";
       exit 2       
    fi

    if [ "$(whoami)" != "root" ]; then
            echo "Sorry, you are not root."
            exit 1
    fi


    USERNAME=$1
    echo "Generating user: $USERNAME"
    sudo useradd -G sudo,developers $USERNAME
    HOME_USER=/home/$USERNAME

    echo "Creating home..."
    mkdir $HOME_USER
    cd $HOME_USER

    echo "Linking programming tools..."
    #link  progtools folder
    ln -s /usr/progtools progtools
    echo  "Creating projects folder"
    #create project dir
    mkdir projects
    cp -R /usr/progtools/tools/deep_test_python $HOME_USER/projects

    echo "Creating .bashrc and .theanorc files"
    cp /usr/progtools/tools/bashrc_orig $HOME_USER/.bashrc
    cp /usr/progtools/tools/theanorc_orig $HOME_USER/.theanorc

    echo "Acquiring property of $USERNAME own home"
    #acquire property
    chown -R $USERNAME:$USERNAME *
    chown -R $USERNAME:$USERNAME $HOME_USER

    echo "setting password for $USERNAME"
    #set password
    passwd  $USERNAME

    echo "Remember to set your python interpreter to /usr/progtools/anaconda2/bin/python on your IDE"

Deep Learning development setup for ubuntu 16.04 Xenial by Artzai Picon & Aitor Alvarez-Gila @ Tecnalia Research & Innovation.  Tested by Adrian Galdran