Due to work restrictions, I have been using Tensorflow CPU version for all of my development. However, with model training taking several hours, I have been fininding the development cycle a bit slow. I was therefore on the lookout for some GPU machines that I could try and see what speedup I got. Google Colab is a great free resource for development and even model training on GPUs (and TPUs). I have been using Tensorflow 2.
I have been developing a machine learning model in Tensorflow 2.0, but because I need to have some custom functionality I am taking a more manual approach instead of full Keras API e.g. see https://www.tensorflow.org/tutorials/quickstart/advanced . One thing that was not clear to me was what algorithm does Keras use when you pass in the option metric=['acc'] to model.compile(). It turns out Keras does clever things and works out the appropriate metric based on your data.
Where I work, the compute clusters have been upgraded to RedHat Eneterprise Linux 7 (RHEL7). I therefore needed to rebuild Tensorflow. I have to build because the standard pip install binaries do not work on these systems as the glibc version is 2.17 on RHEL 7 while Tensorflow >= v1.14 need glibc 2.23. (See my RHEL6 post)
First of all create a new conda environment, using anaconda3:
conda create -n tensorflow pip python=3.
TensorFlow binaries do not work on RHEL6 because they require GLIBC 2.17. The system version is 2.12. Therefore you need to build Tensorflow. (Having another glibc on the system is just painful.) That still requires a few tricks.
First of all create a new conda environment, using anaconda2:
conda create -n tensorflow pip python=2.7 Get hold of tensorflow from:
git clone https://github.com/tensorflow/tensorflow tensorflow cd tensorflow git checkout r1.5 In order to build TF, need bazel.
In order to run TensorFlow in Jupyter notebook in the conda environment that is installed in, need to do the following:
source activate myenv python -m ipykernel install --user --name myenv --display-name "Python (TF)" The second command creates a Jupyter kernel that you can initiate from Jupyter notebook. Once that is done, you should be able to
import tensorflow You may need to install pip install ipykernel before running the above command.
I am a Senior Scientist at the UK Met Office, using machine learning to develop and improve the physics in the weather and climate model. I specialise in radiative transfer with remote sensing applications. In short, I create mathematical and physics models to make sense of data.
