What Deep Learning framework to use in 2019 and why is it PyTorch?

This blog is all about Deep Learning in a 3D Animation, VFX and Games context. From the start, the goal was to create tutorials using the kind of software and data people use in these fields.

That led me to avoid TensorFlow, even though it was the most popular platform at the time, because of Python 2.7. And that is why I chose to use mostly Keras.

In 2018, PyTorch, a deep learning framework developed by Facebook, has reached version 1.0. The research community, as well as practitioners, are adopting PyTorch at a fast pace. Although PyTorch is also not compatible with Python 2.7 either, it supports ONNX, a standard format for describing ML models which we can read from other Python 2.7 compatible libraries.

PyTorch logo
PyTorch logo

PyTorch is different from every other platform in that you don’t need to describe a computation graph and then run it. You code your model in a much more Pythonic way, thanks to something called Autograd. I find this makes it easier to implement more intricate models like GANs or Reinforcement Learning models.

In this article, I show how you can use PyTorch with Maya by refactoring what is probably the most popular tutorial in this blog, the Fast and Deep Deformation Approximations prototype implementation.

What you’ll learn

You will need these resources to follow this tutorial

  • The Maya scene and dataset
  • The IPython notebook to train the Neural Net in PyTorch and Keras
  • The custom deformer to run the Neural Net in Maya (PyTorch and Keras)

Subscribe to receive them via e-mail.

We won't send you spam. Unsubscribe at any time. Powered by ConvertKit

Installation

You can run PyTorch on Google Colab, Azure Notebooks, or in a local setup. Here I’ll outline how to install it locally.

Note that PyTorch does not support Python 2.7 on Windows. Hence, for model inference inside Maya, we’ll be installing CNTK which does support it.

PyTorch

My recommendation is to have an installation of Python 3.6 managed through Anaconda (Conda). Conda allows you to have many Python installations in your system with minimal effort. But Conda is not mandatory.

At PyTorch.org you’ll find the right command to install PyTorch irrespective of what Python version and environment/package manager you are using. You can also choose the version of CUDA compatible with your GPU.

A matrix of installation options for PyTorch
PyTorch installation matrix

I do recommend you have Jupyter Notebook and Matplotlib installed as well, for this tutorial. These come default with Conda.

CNTK

To install CNTK in Maya’s Python interpreter (Mayapy), first, you’ll need to install pip in Mayapy, and dependencies like Numpy and Scikit. All this is explained in this previous tutorial.

Then you should be able to:

mayapy -m pip install cntk

And be good to go.

Model definition and training

Training a PyTorch model is not much more complicated than training a Keras model. You can declare the model pretty much the same way, using either a Sequential API or a Model API.

The FDDA model uses a super simple fully-connected, two-layer network. Here is the network in Keras:

model = Sequential()
model.add(Dense(512, input_dim=inputs_norm.shape[1], activation='tanh'))
model.add(Dense(512, input_dim=100, activation='tanh'))
model.add(Dense(outputs_norm.shape[1], activation='linear'))

Here is the same network in PyTorch:

model = nn.Sequential(nn.Linear(inputs.shape[1], 512),
                      nn.Tanh(),
                      nn.Linear(512, 512),
                      nn.Tanh(),
                      nn.Linear(512, outputs.shape[1]))

Note that fully connected layers are called Dense in Keras and Linear in PyTorch.

After you declare the model topology in Keras you would compile the model with a proper loss function and optimizer, like this:

model.compile(loss='mse', optimizer=’adam’, metrics=['mse'])

In PyTorch, we don’t compile the model like we would in any other library. Instead, we need to create the training loop ourselves.

The training loop

In other machine learning libraries, you use a training function to feed data to a pre-compiled model. In PyTorch, you create your own training loop in a very Pythonic fashion. Every operation you perform on tensors can be automatically backpropagated by PyTorch.

That makes it easier to implement training schemes and loss functions that are more sophisticated. But it comes with a slight additional overhead for simpler models. I say slight because many pieces of this puzzle, like conventional optimizers and loss functions, come ready made. For example, this is how we get an Adam optimizer and an MSE loss function in PyTorch:

optimizer = torch.optim.Adam(model.parameters() ,lr=0.01)
loss_func = nn.MSELoss()

Note that we must declare the model.parameters() as the thing we are trying to optimize.

And this is how the actual training loop looks like:

for epoch in range(epochs):
    optimizer.zero_grad() # (1)
    pred = model(inputs_norm) # (2)
    loss = loss_func(pred, outputs_norm) # (3)
    loss.backward() # (4)
    optimizer.step() # (5)

Here is what we are doing: (1) we reset gradient calculations from previous iterations; (2) We feed-forward data into our model and get a prediction; (3) We calculate the loss based on the prediction and the actual values in the dataset; (4) We backpropagate all operations executed so far (feed-forward + loss); (5) We use the optimizer to update the weights in the model.

This code is enough to train the model, although its missing some features we’ve grown accustomed to in Keras, like (a) shuffling samples at every iteration, (b) splitting data into train and validations sets, (c) printing out and logging data about the training. In the resources for this tutorial I have included an extended version of this training loop with all those features. Plus, I’ve implemented it as a function in a way you can use with other similar models.

Exporting the trained model and loading it in Maya

Once you train the model, you can export it from PyTorch using the ONNX file format. This allows you to run your model in any library that supports ONNX out of the box [CNTK, Caffe2, ONNX runtime], or in platforms for which conversion tools have been developed [TensorFlow, Apple ML, Keras]. You can also load it from Maya’s C++ API, as I have detailed in this tutorial.

Here is how to export the model after training:

dummy_input = torch.randn(1, 16)
torch.onnx.export(model, dummy_input, "fdda.onnx")

In our case, we’ll use CNTK, because of Maya’s Python 2.7. CNTK is commonly loaded using C as the abbreviation. Once the library is imported you can load the model like this:

import cntk as C
model = C.Function.load("C:/.../fdda.onnx",
                        device=C.device.cpu(), format=C.ModelFormat.ONNX)

Model predictions can be called like this:

model.eval(input_variable)

The full code for the FDDA deformer node is available in the resources for this article. And here is the FDDA model, trained in PyTorch, running inside Maya through CNTK:

Two CG articulated models with different deformations
FDDA prototype trained on PyTorch, evaluated using CNTK

In Conclusion

PyTorch is a modern deep learning library that is getting more and more attention. It is being used by most cutting-edge papers, and also in production by Facebook and others.

While it creates a slight overhead when training very simple models it is more flexible and allows for greater customization needed for more complex ones. PyTorch code is also easier to grasp and debug due to its Pythonic nature.

In this article I’ve showed you an easy way to use PyTorch inside a CG environment, with its Python 2.7 restrictions and its infamous binary-compatibility problems. What I have covered in this article is also applicable to C++ deployments with ONNX, something I have covered before here in the blog.

By adding PyTorch to your toolbox (and to mine as well), I hope we can explore more cutting-edge deep learning applications in the future.

Resources

  • The Maya scene and dataset
  • The IPython notebook to train the Neural Net in PyTorch and Keras
  • The custom deformer to run the Neural Net in Maya (PyTorch and Keras)

Subscribe to receive them via e-mail.

We won't send you spam. Unsubscribe at any time. Powered by ConvertKit
Related Posts

One Comment

  1. Pingback: Create a Texture Generator with a GAN - 3DeepLearner.com

Leave a Reply

Your email address will not be published. Required fields are marked *