A new way to get started with Deep Reinforcement Learning

Every time I read an OpenAI blog post I'm inspired.

I'm inspired by the artwork, I'm inspired by the quality of work, I'm inspired by the communication skills.

They work on hard problems. Get great results. And show them off to the world in a way easily accessible to the world.

So when their post came out about Spinning Up, a free resource for learning Deep Reinforcement Learning (RL), I wasn't surprised. Not surprised by the fact I loved the artwork, by simple and effective communication in the blog post and the world class quality of the learning materials.

Seriously. This is something I'd put on my wall (stay tuned, this might happen).

 I’d love whoever designs OpenAI’s graphics to come and paint my room. Source:  OpenAI Blog .

I’d love whoever designs OpenAI’s graphics to come and paint my room. Source: OpenAI Blog.

More to the point, if you're interested in Deep RL, you should leave here and check out Spinning Up.

If you're not interested, it's worth knowing a little about it anyway.

Deep RL involves the training of an agent in an environment to learn something about the world.

This is best explained with an example.

Deep RL was the technique DeepMind used to build Alpha Zero. The best Go player in the world.

In Alpha Zero's case, it (the agent) learned to play Go by continually playing itself. It got to a superhuman level exceptionally fast because it was able to play many simulated games of Go (the environment) against itself at once.

The deep part comes from using many versions of itself to learn. So really, there were multiple agents (different versions of Alpha Zero, converging into one) and multiple environments (many different games of Go).

How about another example?

Let's say you're a doctor who wants to try a new treatment on your patients. But you're hesitant because you're not sure how it will go.

So you decide to wait for more trials to take place.

But trials are lengthy, expensive and potentially harmful to those involved.

What if you could simulate a treatment (the agent) and test it on many different simulated patients with characteristics similar to yours (the environment) and see what the outcomes were?

Using the knowledge you gather, you could potentially find an ideal treatment for each individual patient.

This example simplifies the process dramatically but the principles of Deep RL are there.

The beautiful thing is, Deep RL could potentially be harnessed for any problem where many different scenarios need to be accounted for.

Say you wanted to improve traffic lights to find a more ideal schedule, Deep RL could be used to test many different car arrivals (the environment) and many different light configurations (the agents) with the goal of maximising efficiency.

The scenarios are endless.

I know there's a way to somehow use Deep RL for health. But I don't know how (yet).

To help figure this out, I'm incorporating the Spinning Up materials into my curriculum for 2019. I can't wait.

And of course, I'll be sharing what I learn.

If you're interested in getting started with Deep RL, there are a few resources you might want to check out:

There's also a workshop OpenAI are hosting on February 2 at their San Francisco HQ for those who have tinkered with machine learning and are wanting to learn more about Deep RL. I applied and you can too (applications close December 8).

How to Learn Deep Learning in 2 Months

‘Hi, I’m a machine learning engineer from Google.’

‘Hello everyone, I’m a software engineering at Intuit.’

‘Good morning, my name is Sandy, I’m a freelance data scientist.’

Ugh.

I was scared to post my introduction.

What was I going to say?

‘Hi, I’m Daniel, I quit the first programming course I signed up to, so now I’m here to try again.’

I probably could’ve said that. That’s actually not too bad. At least it was the truth.

It was a week before the Udacity Deep Learning Nanodegree began. Everyone was introducing themselves in the Slack channel.

I was the odd one out. It seemed everyone had already some experience with machine learning or data science and here I was writing my first line of Python 3-weeks prior.

The course went on. I graduated 4-months later.

Then I went on and did Udacity’s AI Nanodegree as part of my online AI Masters Degree.

And Andrew Ng’s deeplearning.ai course on Coursera.

Then I got hired as a machine learning engineer.

All in the space of a year.

But 2 months? Woah.

That’s some ambition. I love it.

Let’s be clear though, the field of deep learning and all the other encompassing crafts is huge. You’ve seen that photo of the iceberg right? The one where most of it is under water and only the top 10% of it sticks out of the water. Yeah, that’s like deep learning.

The art of building deep learning neural networks can be picked up in a couple of months by someone with a little programming experience.


So how would you do it?

Done some Python work before? Go ahead with these steps. If not, learn some Python first and come back.

These two books will help you go from a deep learning rookie to a deep learning practitioner.

‘But Daniel, these books cost money?’

You’re right. But the education is worth it. Especially since $100 worth of books, if used correctly, could give you provide you the skills for a $100,000+ role. Or better still, provide you with the skills to build something incredible.

If you can’t afford these books, I’d recommend Siraj Raval’s or Sentdex’s YouTube channel.


‘I’ve got a little longer than two months.’

Putting together a deep learning model is often one of the last pieces of the puzzle.

Even the best deep learning models won’t help anyone if there’s no data for them to learn on.

Deep learning is the combination of statistics, programming, math and engineering. Maybe a few more things.

If you’ve got longer than two months, I’d still recommend the two books above. As well as any of the courses I’ve mentioned.

But the key thing is to try out what you’ve learned.

Find a problem which interests you and see if you can apply deep learning to it.

‘Yeah, people always say that, find something that interests you, but I’m not interested in anything…’

Not interested in anything?

Then what pisses you off?

Maybe you’re pissed off at the amount of homeless in your city. How can you use deep learning to find more people a home?

Or perhaps you don’t like the state of the healthcare system, does deep learning have a place in building a healthier world?

A major part of being a good deep learning engineer is to seek out new ways of looking at and solving existing problems.


How long per day?

I heard an interview once with a world chess champion.

‘How much do you play per day?’

‘Never more than 3-hours per day.’

‘Really?’

‘Yeah, but during those 3-hours, all I’m doing is playing chess.’

3-hours per day? That’s it?

I had to try it. So I did.

I set a timer. One hour, 5-minute break, another hour, 15-minute break, then another hour.

It took me all three hours to get through one function.

The next day I finished the whole assignment in the same timeframe.

I still do this kind of study every day.

Sometimes you may be able to do more. Other days less. But the main thing is to focus on one thing at a time.

Especially if you want to learn deep learning in two months. Block all the other crap out and learn deep learning. I have to remind myself of this all the time.

Never half ass two things, whole ass one thing. You’ll be surprised what you’re capable of.

You might even end up running down your street after finishing your final coding project.

 Study hard. Celebrate hard. Screenshot from Learning Intelligence 34:  Finishing the Udacity Artificial Intelligence Nanodegree

Study hard. Celebrate hard. Screenshot from Learning Intelligence 34: Finishing the Udacity Artificial Intelligence Nanodegree



Source: https://qr.ae/TUhEPQ

How much math do you need to start learning machine learning or deep learning?

‘I’m studying artificial intelligence.’

‘What?’ my Uber passenger asked, ‘is that like aliens?’

‘It’s more like a combination of math, statistics and programming.’

‘You’ve lost me.’

I didn’t really want to go back to university so I made my own Artificial Intelligence Masters Degree using online courses. I drove Uber on the weekends for 9-months to pay for my studies.

Now 7-months into being a machine learning engineer, I’m still working through it, along with plenty of other online learning resources.

My first line of Python was 3 weeks before starting Udacity’s Deep Learning Nanodegree.

I’d signed up and paid the course fees but I was scared of failing so I emailed Udacity support asking if I could withdraw my payment and enrol at a later date.

Screen Shot 2018-11-23 at 9.15.03 pm.png

I didn’t end up withdrawing my payment.

For 3/4 of the projects, I needed an extension. But I still managed to graduate on time.

The main challenge for me was the programming, not the math.

But to learn the math knowledge I did need, I used Khan Academy.

Especially:

I was learning on the fly. Every time a concept came up I didn’t know about, I’d spend the next couple of hours learning whatever I could. Most of the time, I bounced between Stack Overflow, Khan Academy and various machine learning blogs.

How much math do you need?

It depends on your goals.

If you want to start implementing deep learning models and achieving some incredible results, I’d argue programming and statistics knowledge are more important than pure math.

However, if you want to pursue deep learning or machine learning research, say, in the form of a PhD, you’ll want a strong math foundation.

The math topics I listed above take years, even decades to fully comprehend. But thanks to deep learning and machine learning frameworks such as Keras and TensorFlow, you can start replicating state of the art deep learning results with as little as a few weeks of Python experience.

The thing with math is, it’s never going away. Math is the language of nature. It’s hard to go wrong brushing up on your math skills.

But if the math is holding you back from jumping in and trying machine learning or deep learning, don’t let it.

A fisherman doesn’t learn how to catch every single fish before he goes fishing. He practices one fish at a time.

The same goes for learning. Rather than trying to learning everything before you start, learn by doing, learn what you need, when you need.

Source: https://qr.ae/TUhEPQ