Building a Digits Dev Machine on Ubuntu 16.04

Ubuntu 16.04 with CUDA 8.0, OpenCV 3.2.0, Caffe 0.15, and Digits 5.0

Notes: Updated: 6/22/2017

Pre-Setup

If you're like me, you like to have control over where and what gets installed onto your dev machine, which also mean that sometimes, it's worth taking the extra time to build from source. Here at KickView we set up many dev machines and thought we'd share our insight on building up the latest and greatest of Digits 4.0 on an Ubuntu 16.04 machine. Have fun.

This tutorial was written based off of a clean install of Ubuntu 16.04.1 server edition.

Basic Machine Setup

  1. Update packages

    sudo apt update && sudo apt upgrade
    
  2. Restart computer

Major Dependency Setup

  1. Install make, cmake, gcc, g++

    sudo apt install make cmake gcc g++
    

Install CUDA 8.0

  1. Install Dependencies

    sudo apt install dkms build-essential linux-headers-generic
    
  2. Disable the nouveau driver

    • Create the following file /etc/modprobe.d/blacklist-nouveau.conf and add the following lines:

      blacklist nouveau
      blacklist lbm-nouveau 
      options nouveau modeset=0 
      alias nouveau off 
      alias lbm-nouveau off 
      
    • Disable the Kernel nouveau by typing the following commands:

      echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf
      sudo update-initramfs -u
      
  3. Restart the machine

  4. If you are running Ubuntu 16.04 Server, you can skip this step

    1. Disable the desktop by running the following command

      sudo systemctl stop lightdm.service
      
  5. Download CUDA 8.0 from https://developer.nvidia.com/cuda-toolkit
    Make sure to get the Ubuntu 16.04 version!

  6. Navigate to where you downloaded the file and run CUDA 8.0 install with the following command (We've had problems with the OpenGL-libs in the past so we tell the installer to not install those)

    sudo ./cuda_8.0.61_375.26_linux.run --no-opengl-libs
    
  7. Add these lines to your ~/.profile file

    # CUDA
    PATH="/usr/local/cuda-8.0/bin:$PATH"
    LD_LIBRARY_PATH="/usr/local/cuda-8.0/lib64:$LD_LIBRARY_PATH"
    
  8. Add this line to the end of the /etc/ld.so.conf file

    /usr/local/cuda-8.0/lib64
    
  9. Run sudo ldconfig to update your configs.

  10. Log out and back into your terminal instance to pull in all updates OR run source ~/.profile
  11. Run nvidia-smi to make sure install worked as expected

Install pip, virtualenv, and virtualenvwrapper

  1. Install pip and upgrade

    sudo apt -y install python-pip
    sudo pip install --upgrade pip
    
  2. Install virtualenv

    sudo pip install virtualenv
    
  3. Install virtualenvwrapper

    sudo pip install virtualenvwrapper
    
  4. Add these lines to you ~/.profile file

    # virtualenvwrapper
    export WORKON_HOME=$HOME/.virtualenvs
    source /usr/local/bin/virtualenvwrapper.sh
    
  5. Source .profile to activate virtualenvwrapper

    source ~/.profile
    

Setup System Developer Directories

There are many times where multiple people will need to use a single development machine. Software that is going to be shared between all users on a machine should go into a directory that is accessible by all developers. So we're going to setup the /opt folder to be just that, a singular place where third party software can be installed and shared between all developers using the machine.

  1. Create a developer group

    sudo addgroup developer
    
  2. Add user to the developer group

    sudo user mod -aG developer <user>
    
  3. Logout and log back in to have the new group permissions take affect

  4. Change owner and permissions of /opt to allow developer access

    sudo chown root:developer /opt
    sudo chmod 775 /opt
    

Install OpenCV 3.2.0

We have moved to Python 3.5, I will leave the Python 2.7 for legacy purposes, pick the section that is applicable to you

Python 3.5
  1. Create a new virtualenv for opencv

    mkvirtualenv --python /usr/bin/python3 opencv
    
  2. Keep this virtualenv active for the remainder of this section
  3. Pull OpenCV v3.2.0 and OpenCV_Contrib v3.2.0 from their git repo.

    cd /opt/
    git clone -b 3.2.0 --depth 1 https://github.com/opencv/opencv.git
    git clone -b 3.2.0 --depth 1 https://github.com/opencv/opencv_contrib.git
    
  4. Navigate into opencv and create a build directory

    cd opencv && mkdir build && cd build
    
  5. Install opencv dependencies by running the following two commands

    sudo apt install libjpeg8-dev libtiff5-dev libjasper-dev libpng12-dev libgtk2.0-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libatlas-base-dev gfortran libhdf5-serial-dev libeigen3-dev python2.7-dev python3-dev ffmpeg libopencv-dev libgtk-3-dev libdc1394-22 libdc1394-22-dev libjpeg-dev libxine2-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libtbb-dev qtbase5-dev libfaac-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev x264 v4l-utils unzip
    
  6. Install numpy into your virtualenv

    pip install numpy
    
  7. Build opencv NOTE, we're disabling opencv_dnn because caffe includes this and will have problems if you try to use both
    Make sure you are in your virtualenv before running this

    cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_CUDA=ON -D ENABLE_FAST_MATH=1 -D CUDA_FAST_MATH=1 -D WITH_CUBLAS=1 -D INSTALL_PYTHON_EXAMPLES=ON -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules -D BUILD_opencv_dnn=OFF -D BUILD_EXAMPLES=ON -D PYTHON_EXECUTABLE=$(which python) ..
    make -j8
    
  8. Once build completes, install opencv

    sudo make install
    sudo ldconfig
    
  9. Inside you're virtualenv add cv2.so to the path

    cd /usr/local/lib/python3.5/site-packages/
    ln -s cv2.so cv2.cpython-35m-x86_64-linux-gnu.so
    add2virtualenv .
    
  10. Now test this out by starting up a python instance inside your virtualenv and running import cv2

  11. Deactivate your virtualenv session

    deactivate
    
Python 2.7
  1. Create a new virtualenv for opencv

    mkvirtualenv opencv
    
  2. Keep this virtualenv active for the remainder of this section
  3. Pull OpenCV v3.2.0 and OpenCV_Contrib v3.2.0 from their git repo.

    cd /opt/
    git clone -b 3.2.0 --depth 1 https://github.com/opencv/opencv.git
    git clone -b 3.2.0 --depth 1 https://github.com/opencv/opencv_contrib.git
    
  4. Navigate into opencv and create a build directory

    cd opencv && mkdir build && cd build
    
  5. Install opencv dependencies by running the following two commands

    sudo apt install libjpeg8-dev libtiff5-dev libjasper-dev libpng12-dev libgtk2.0-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libatlas-base-dev gfortran libhdf5-serial-dev libeigen3-dev python2.7-dev ffmpeg libopencv-dev libgtk-3-dev libdc1394-22 libdc1394-22-dev libjpeg-dev libxine2-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libtbb-dev qtbase5-dev libfaac-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev x264 v4l-utils unzip
    
  6. Install numpy into your virtualenv

    pip install numpy
    
  7. Build opencv NOTE, we're disabling opencv_dnn because caffe includes this and will have problems if you try to use both
    Make sure you are in your virtualenv before running this

    cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_CUDA=ON -D ENABLE_FAST_MATH=1 -D CUDA_FAST_MATH=1 -D WITH_CUBLAS=1 -D INSTALL_PYTHON_EXAMPLES=ON -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules -D BUILD_opencv_dnn=OFF -D BUILD_EXAMPLES=ON ..
    make -j8
    
  8. Once build completes, install opencv

    sudo make install
    sudo ldconfig
    
  9. Inside you're virtualenv add cv2.so to the path

    cd /usr/local/lib/python2.7/site-packages/
    add2virtualenv .
    
  10. Now test this out by starting up a python instance inside your virtualenv and running import cv2

  11. Deactivate your virtualenv session

    deactivate
    

Install Caffe 0.15

NOTE: Caffe 0.16 has been released but it requires CUDNN (if using a GPU) 6.0 to be installed. Since Tensorflow is on version 1.2 which only supports CUDNN 5.1 we will stick with Caffe 0.15 for the next couple months until Tensorflow 1.3 is released with CUDNN 6.0 compatibility.

  1. Create a virtualenv for caffe and activate it

    Python 3.5
    mkvirtualenv --python /usr/bin/python3 caffe
    add2virtualenv /usr/local/python3.5/site-packages/
    
    Python 2.7
    mkvirtualenv caffe
    add2virtualenv /usr/local/python2.7/site-packages/
    
  2. Download and install cuDNN from nvidia's developer site https://developer.nvidia.com/cudnn (Get version 5.1 compatible with CUDA 8.0)

    1. Extract the downloaded folder and run the following commands to install cuDNN

      cd cuda
      sudo cp include/* /usr/local/cuda/include/
      sudo cp lib64/* /usr/local/cuda/lib64/
      
  3. Install caffe dependencies

    sudo apt install --no-install-recommends build-essential cmake git gfortran libatlas-base-dev libboost-all-dev libgflags-dev libgoogle-glog-dev libhdf5-serial-dev libleveldb-dev liblmdb-dev libopencv-dev libprotobuf-dev libsnappy-dev protobuf-compiler python3-tk
    
  4. Setting the CAFFE_HOME environment variable will help DIGITS automatically detect your Caffe installation, but this is optional. Add this to your ~/.profile

    # Caffe - example path, can be customized
    export CAFFE_HOME=/opt/caffe
    
  5. Source your .profile

    source .profile
    
  6. Download source

    git clone --depth 1 -b caffe-0.15 https://github.com/NVIDIA/caffe.git $CAFFE_HOME
    
  7. Install requirements into virtualenv

    pip install -r $CAFFE_HOME/python/requirements.txt
    cd caffe
    
  8. Note: If you are wanting to use multiple GPU's you must install NVIDIA nccl for caffe to work. Follow these instructions to install nccl if needed.

    1. Clone and install NVIDIA nccl

      git clone https://github.com/NVIDIA/nccl.git
      cd nccl
      make CUDA_HOME=/usr/local/cuda test
      sudo make PREFIX=/usr/local install
      
  9. There are two different ways you can go about this. One, you can create a Makefile.config and configure caffe manually, OR you can save yourself a headache and just let cmake configure everything for you... the cmake works very well and is what we prefer to use.

  1. Create build directory

    mkdir build
    cd build
    
  2. Prep caffe

    Python 3.5
    pip install --upgrade python-dateutil
    cmake -Dpython_version=3 ..
    
    Python 2.7
    cmake ..
    
  3. Compile Caffe

    make all -j8
    make runtest -j8
    
  4. Add Caffe path to your venv

    cd $CAFFE_HOME/python
    add2virtualenv .
    

Install Digits 5.0

==Note Digits only works on Python 2.7, because it is only used for creating datasets we have opted to use DIGITS be using NVIDIA's DIGITS docker image and just mount our data folders into the docker for processing. Makes for a very clean install that works every time. I will Leave the build instructions for Digits but just know that if you're working in Python3, using a docker is the best option.

Python 2.7
  1. Create a clone of your caffe virtualenv and rename it digits

    virtualenv-clone ~/.virtualenvs/caffe ~/.virtualenvs/digits
    workon digits
    
  2. Download dependencies

    sudo apt install --no-install-recommends git graphviz gunicorn python-dev python-flask python-flaskext.wtf python-gevent
    
  3. Setting a DIGITS_HOME environment variable for tutorial purposed, this is completely optional. Add this to your ~/.profile

    # Digits - example path, can be customized
    export DIGITS_HOME=~/digits
    
  4. Clone the digits git repo

    git clone https://github.com/NVIDIA/DIGITS.git $DIGITS_HOME
    
  5. Install the python dependencies

    pip install -r $DIGITS_HOME/requirements.txt
    
  6. From the digits directory run the following to fire up a Digits server running on port 5000

    ./digits-devserver
    
Python 3.5
  1. Install Docker
  2. Install Nvidia-docker
  3. Run digits by using the following command

    nvidia-docker run --name digits -d -p 5000:5000 -v </path/to/data>:/data -v digits-jobs:/jobs nvidia/digits:5.0
    

Congratulations!!! You now have a running dev machine with the latest versions of OpenCV, Caffe, and Digits. Have fun!