Recapping the Computer Vision Meetup – January 2023

Michelle Brinich

January 17, 2023

Last week Voxel51 hosted the January 2023 Computer Vision Meetup. In this blog post you’ll find the payback recordings, highlights from the presentations and Q&A, as well as the upcoming Meetup schedule so that you can join us at a future event. Hope to see you soon!

First, Thanks for Voting for Your Favorite Charity!

In lieu of swag, we gave Meetup attendees the opportunity to help guide our monthly donation to charitable causes. The charity that received the highest number of votes this month was Foundation Fighting Blindness. We are pleased to be making a donation of $200 to them on behalf of the computer vision community!

Computer Vision Meetup Recap at a Glance

Hyperparameter Scheduling for Computer Vision

Video Replay

Executive Summary

Cameron R. Wolfe, research scientist at Alegion and Ph.D. student at Rice University, shares some of his research on the topic of hyperparameter scheduling for computer vision. To begin, Cameron explains that hyperparameter tuning is important because deep learning can be computationally expensive. When you’re training over a dataset, you go through all of the data from multiple epochs, and at the end you check whether or not your model performs well. If it doesn’t, then you have to go back to stage one. This creates a loop where, if you don’t select your hyperparameters properly, you’re continually retraining your network trying to get one that performs well. While many people attribute the computational expense of deep learning to big datasets and large models, the problem gets worse with incorrect hyperparameters because you have to incur the expense of training your deep neural network multiple times.

In his presentation, Cameron describes how to use hyperparameter schedules, how to set them properly, and practical takeaways for hyperparameters across three main areas: learning rate, training precision, and video deep learning.

Learning rate

The learning rate part of the presentation is based on a paper Cameron wrote with Rice – REX: Revisiting Budgeted Training with an Improved Schedule. The idea behind it is that when you consider different budget amounts for training a deep neural network, certain learning rate schedules work better than others.

Why does budget matter? Maybe you have a computational and/or monetary budget and need to train within it; or maybe you have a deadline and can’t spend too much time training your network. One of the most effective ways for working within a budget is to simply reduce your number of training epochs. Instead of training a model for 200 epochs on ImageNet, for example, you can shorten it down to 90 and you can probably get pretty good performance if you set your hyperparameters correctly. Cameron’s research looks at how to properly set the learning rate when training neural networks in a budget-aware setting.

To explore different learning rate options in a budget-aware setting, Cameron decomposes learning rate schedules into two components: profile (the continuous function that models the decay of your learning rate; classic examples are – cosine, linear, exponential, and step schedules) and sampling rate (how frequently you want to update your learning rate from this profile).

To set the scene, Cameron shows two classic examples – a step schedule and a linear schedule.

step schedule vs linear schedule example

What Cameron’s research contributes is a new profile for learning rate decay: reverse exponential or REX, which is a learning rate schedule that keeps the learning rate high for a while and then decays it to a lower learning rate at the end of training.

Now the question is: how do the learning rate and sampling rate pairings perform? To conduct this experiment, Cameron conducted experiments for each profile to find the optimal sampling rate across seven different domains.

From this, the research takes the optimal profile and sampling rate pairings and sees how they compare on six different learning rate schedules.

Key takeaways:

Step schedules (commonly used in computer vision) only perform well in the high-budget regime
REX performs really well across domains and budgets
Many choices exist for learning rate schedules; choose the best one based on your settings – number of epochs, domain, or problem (classification, detection, etc.)

Training precision

Because training deep neural networks is computationally expensive, another way to reduce costs would be to look at better schedules for low precision training for neural networks. Specifically, in this presentation and research he’s working on, Cameron looks at cyclical precision training (CPT) (not fixed or static training), where the precision being used for the neural network varies cyclically throughout the entire training process.

The idea behind low precision training is that instead of training your neural network at the normal 32 bit precision, you can lower this to 16 bits, 8 bits, or maybe even lower, which would save compute costs. How this works is, when you do your forward pass, you quantize your activation and weights to a lower precision before performing the matrix’s multiplication in the forward pass, which is faster. You can do the same thing in the backward pass, which has two matrix multiplications (one to compute your weight update and one to propagate the gradient to the previous layer), which saves twice as much compute.

The question Cameron’s research addresses is: what are the best hyperparameter schedules for CPT to achieve gains (reduced cost or increased performance)? This research follows a similar approach to the REX research by decomposing the precision schedules into parts (in this case, three).

Cameron notes that the harder part is choosing repeated or triangular schedules. To better understand what is meant by repeated or triangular schedules, Cameron shows the set of 10 (repeated and reflected), which you can see below.

Cameron’s research collapses the 10 schedules into three groups (large schedules, medium schedules, and small schedules) based on the impact they have on the amount of compute savings you get. Here’s the mapping -> large = RR and RTH; medium = LR, LT, CR, CT, RTV, ETV; small = ER, ETH. (You may want to come back to these mappings later when you review the precision takeaways later in the post.)

Cameron walks us through some of the highlights from the exploration:

There tends to be a correlation between the model performance and the amount of training compute that we’re using. Cameron adds “if we use less compute or have more computational savings, we tend to get a little bit worse performance and vice versa. So even though we can use these alternative precision schedules, in a lot of cases we’re not getting these compute benefits for free. If we want our training to go by faster, we’re probably going to get a model that performs not quite as good.”
In certain cases using alternative schedules (beyond just cosine schedules) can be beneficial. For example, when training a ResNet 18 on ImageNet, the best performance actually comes with the exponential schedule.
Some settings are sensitive to lots of quantization, so you can’t always train a network with really low precision; it depends on the domain. So be careful with how low the precision is with which you’re going to be training.

Key takeaways for training precision are:

You can gain a lot by exploring alternative hyperparameter schedules for cyclic precision training.
Here’s Cameron’s guidance on how to choose one:
- Use small schedules to minimize training costs
- Use large schedules to maximize model performance
- Use medium schedules to find a balance

Now for a twist: none of these can actually be used practically right now because current hardware only supports low precision training at certain levels. Cameron recommends “the best thing to use for low precision training right now is just something called Apex,” which implements 16 bit precision training with minimal code changes required. For more information, check out Cameron’s article: Quantized Training with Deep Networks.

Video deep learning

The last part of Cameron’s presentation is about video deep learning and how cyclical batch sizes can speed up wall-clock training time. In this section, Cameron walks us through some background found in the SlowFast Networks for Video Recognition paper. The idea is that instead of having just a single 3D CNN that we convolve over the entire input video over multiple layers, we separate our network into two different modules: the Slow Pathway, which is in charge of capturing spatial or semantic features; and the Fast Pathway that’s in charge of capturing motion features.

The answer proposed in the paper, A Multigrid Method for Efficiently Training Video Models, is to vary the mini-batch size according to a hyperparameter schedule.

In the image below on the right, you can see the results for training over a Kinetics dataset on a single GPU using this multigrid training method. This leads to improved efficiency – the same performance in just two days instead of nearly an entire week!

Conclusion

Hyperparameter schedules are fundamental. You can apply them in different scenarios and they end up being useful for finding a benefit (reduced compute, increased performance, reduced training time, etc.)

Q&A Recap

Here’s a recap of the live Q&A from this presentation during the virtual Computer Vision Meetup:

Q: Any comment on REX in regards to transfer learning?

A: We didn’t test that in this case, but that would be a really interesting test to run. A lot of times what I’ve found is that when you’re fine tuning a network on a downstream dataset, if you start with too high of a learning rate, that might cause problems. So my guess would be that REX is fine for transfer learning but you have to be careful with setting the initial learning rate to make sure that it’s not too high. But again, that would be pretty cool to test.

Q: In addition to problem domain and budget, do you see that just the dataset itself (size, diversity, etc.) can cause one LR to work better or worse?

A: Yes, all of these things are factors that you have to consider when choosing hyperparameters. So we can boil this down to the question of if you’re training a neural network on one dataset and you switch it and try to train it on another dataset, are you going to use the same hyperparameters out of the box? Probably not. You’re going to have to test a few things and see what works. When you change domains or datasets, you’re going to have to do more tuning and figure out what’s best for your new dataset.

Q: When do you think we will see convex optimizers for arbitrary neural networks?

Neural networks like by nature are nonconvex. But at the same time, even though the optimization problem with neural networks is nonconvex, for a lot of optimization algorithms, all of the proofs are in convex settings or simplified settings. The literature for analyzing nonconvex convergence rates is a lot harder than the convex proof. So typically it seems like for deep learning we can take intuitions from convex optimization and see whether they’ll work. And in the case of SGD, for example, it works really well.

Q: How are we initiating REX?

With REX, you choose an initial learning rate, you have a schedule by which that learning rate is varied, and then from the beginning to the end of your training, you decay the learning rate from the initial choice to kind of 10x lower than that initial choice or maybe 100x according to that schedule. So there’s no initiation needed, it’s just a fixed function profile by which we decay our learning rate.

Q: In addition to randomizing training sets, do you insert intentional “disturbances”?

A: No, really all we do here is pick a bunch of different domains, a bunch of different publicly available datasets, and then run training across all of these to see which learning rate schedules work best. All of the setups are pretty standard I would say; we’re not doing anything extra just running training over public datasets.

Q: Low Precision training, is that also called Quantization Aware Training?

A: Quantization aware training refers to methods that make your network perform better when you make it smaller at the end so that you can deploy it onto an edge device, for example. The purpose of that is training the network in a way that when you quantize it, it will still perform well at the edge because when you convert it to a low precision representation, that uses less memory so that you can deploy it. The difference between that and what I’m doing here is that cyclical precision training (CPT) is not focused on trying to create a smaller neural network at the end, it’s focused upon trying to reduce training costs in general. We’re performing the training process in a low precision manner so that we can save compute costs during training. We’re not necessarily trying to quantize the network to make it smaller at the end. Though, there could be a relationship between low precision training and quantization aware training such that performing low precision training like this could be a good method of quantization aware training to where these networks are easier to quantize at the end.

Is CPT theoretical (e.g. implemented entirely in software)?

A: Correct, there’s no hardware support for this yet. I hope that hardware will catch up with it eventually, but right now the best thing to use is Apex because you can train it at floating point 16 precision, it’s easy to implement, and it’ll increase speed without losing any performance, though check to make sure.

Q: Are there trade offs/disadvantages for Low Training Precision?

Yes. There’s a correlation between model performance and training compute. So if you’re lowering the precision too much, you’re going to pay the cost – your network is going to deteriorate in performance. This is relative to training with a fixed, but lower precision level. But if you adopt a different strategy that doesn’t quantize quite as aggressively, you might actually improve model performance while saving compute. In the talk I shared an example of an ImageNet case compared to the baselines, where the exponential precision schedules both reduced compute compared to baselines and improved the performance. So it depends; there is a trade off if you quantize too much, but there are certain cases where you can actually get better performance and save training costs. It just depends on the scenario.

Additional Resources

Check out the additional resources on the presentation:

Talk transcript
Presentation links (to papers, articles, etc.)
All links (newsletter, Twitter, company, lab, etc.)

Thank you Cameron on behalf of the entire Computer Vision Meetup community for sharing your research and helping us better understand how to approach the tuning of hyperparameters!