Using Deep Reinforcement Learning to Play Sonic the Hedgehog

In between projects over the past couple of weeks, the Max Kelsen team and I attempted to replicate the epic World Models architecture to play Sonic the Hedgehog for the OpenAI Retro Contest.

Although our algorithm wasn't fully ready by the submission date, we still learned a great deal in the process.

The full writeup of our approach is available on Medium.

Right place, right time

You missed out on the promotion and Nick got it instead. 

You missed out on the girl and Tommy got her instead.

You missed out on first place because Sarah showed up to the competition early. 

Were you in the wrong place at the wrong time? Or were they in the right place at the right time?

Where’s the right place?

Right here. Right where you’re meant to be. You chose to be here, now make the most of it.

Maybe you’re in an office you don’t like. Perhaps you’re in a city which is too cold. 

There will never be a perfect place but it doesn’t have to be perfect to be right.

When’s the right time?

Right here. Right now you’re meant to be. You chose to be here at this time, now make the most of it. 

Maybe you’re up early to study before work. Maybe you’re in the 2-3pm slump and looking for a distraction to keep you entertained. 

There will never be the perfect time but it doesn’t have to perfect to be right.

Both of these have the same arguments. Waiting to be in the perfect place at the perfect time will result in a life filled with missed opportunities. 

We’ve evolved to adapt to the unknown. Fast. If you want to be at the right place and at the right time more often, you have to create those moments. 

Show up early.

Show up often. 

Show up even when you don’t really feel like it. 

Thrust into the unknown. 

People who do these things consistently make their own luck. The beautiful thing? You’re capable of doing them too.

If it takes more than one step...

I’m not doing it. 

How many times have you gone through the sign up phase of a website and left early because it took too long? 

This is what Apple do so well. You buy a device from them and you can use it as it is. You don’t need to buy more software, it’s all there for you. And if you want to customise it, you can. 

Taxi’s weren’t broken until Uber came along. Now you could get a taxi with one tap. No need to enter your location either, it’s already in the app.

We’ve become impatient because everything online should be available now. Right now. And if it isn’t, we’ll go somewhere else to get it.

If you want to be found, be easy to find. If you want more users, remove the roadblocks.

The best messages...

Are ones people can understand. 

It takes skill to be able to communicate something in a way people can understand.

You can pair public speaking with almost anything else and find a way to make a career out of it.

Write about what interests you most and find people to read it.

Make videos of your adventures and share them with the world.

We all want to feel like we belong to something. Good communicators allow that. They bring people together. They create a community. Maybe that’s why the two words are so similar. You can’t have a community without communication.

If you want to work on improving anything. Improve your communication. Not only to others but with yourself. 

Twitch vs Facebook for livestreaming, building and launching an app from scratch and conversations with robots

Sometimes you pour your heart and soul into something and then no one cares about it. It's a shitty feeling.

It takes courage to make something. To put it out into the world. To know someone might not like it but to do it anyway.

Creation is freedom. But too often we're snowed under the amount of information we have access to. A quick browse of the internet and it seems everyone is after your time and attention.

I can't talk, I make heaps of things. I want people to read my words. I want people to watch my videos. I'm grateful for the people who do but it doesn't bother me if they don't. I'm addicted to the feeling of tying ideas in my head together and making something out of nothing.

I've been trying to instil this creative mindset into my brothers. I want them to experience the same feeling I feel when I make something. Everyone should feel this feeling.

One of my brothers recently started livestreaming himself playing Fornite. We had to explain what livestreaming was to our parents. They had no idea it was even possible, let alone understand what it meant. We told our Dad it was the same thing as watching a rugby game live but others are watching him play video games. He got it. Sort of.

Will started livestreaming on Twitch for a month before moving to Facebook. Facebook is really pushing videos so his numbers went through the roof. Facebook is still the king when it comes to content distribution.

My other brother Josh launched his first app last month. When I was his age I had hopes of making my own app. He beat me to it. I failed to execute. No excuses. I failed. I asked him, what made you start building an app? He replied, "I wanted to see how hard it was to build an app, I thought of something simple and then said let's execute."

I want to live my life like that. Thinking of something and then executing on it. Friction annoys me. If rocks fall down and block my path, I clear them out or climb over them.

It's natural to want to remove obstacles. We subconsciously seek out comfort. Our energy is precious. We're still in the mindset of needing to conserve it or else we'll die.

My brothers streaming numbers went up on Facebook because the friction of leaving the platform for Twitch was taken away. Instead of clicking a link and watching, they can watch him play right in their newsfeed.
 
If something takes more than 1-step, we avoid it. I do this.

You can have the best product in the world but if you don't have shelf space, it's useless. Facebook's newsfeed is the shelf space for my brother's stream. It's like the stack of specials you see as you walk into the supermarket. Trying to get people to look elsewhere (linking to Twitch) is like getting them to walk to the back of the store. If they need milk, they'll walk to the back. If not, they won’t.

The internet is a beautiful place. We can entertain ourselves with almost any form of content we like. And the platforms we use are getting better at continually serving us similar kinds of content. I haven’t searched for a YouTube video in a long time. But it’s not just a consumption paradise, we can also use it to learn almost anything we want.

Josh built his app after doing half an iOS course on Udemy. And I’ve been learning artificial intelligence and machine learning through only online resources.

There's no shortage of education, only a shortage of willingness to learn.

There's even more of a shortage of sharing what you learn. Everyone can be a teacher. Everyone should be a teacher. We could all learn something off each other. You learn something, you teach it to someone else, it cements your knowledge and helps them learn too. It creates a circle. Everyone wins.

We finished up the episode talking about Google's latest breakthrough. Google Duplex. It's an AI-powered personal assistant who's able to make phone calls on your behalf. It sounds like a person. It even makes very human-ish sounds during the conversation. Whilst talking to a hairdresser, it casual dropped a ‘mhmmmm’. It's the beginning of the movie Her. Perhaps I'll be able to have a conversation with my Alexa soon.

Robots holding real conversations, learning to code and launching your app after $7.50 worth of an online course and broadcasting yourself playing video games to hundreds of people live online, we live in a crazy world. I love it.

See you next week. Hopefully, we'll have half the technical difficulties.

PS good luck getting past 77 on Josh’s app. Our little brother, Sam, currently has the high score. Send me a message if you manage to beat it and I’ll give you a shoutout.

Links mentioned in the show:

Listen to the show:

Watch the show:

 

 

 

The Amazon takeover, Facebook's camera in your house and Sleep

There’s a lot going on in the world.

Sometimes it’s hard to keep up.

My brother and I live together and we still struggle to stay up to date on what each other is up to.

Usually, we go for walks after work and talk about the world. Our conversations revolve around technology, business, technology in business, health and life.

Our skill sets complement each other. I’m a tech nerd who’s still learning about the business world and he’s a business dude who’s learning about the tech world.

We’re launching this podcast with no rules. Except for the fact we’ll try and keep the time under an hour or so. No promises.

Each episode will have 3ish topics. A mixture of what our normal conversations are about.

We may get some guests on the show at some point too. So if you know anyone we should talk to or want to come on the show, give us a holla.

This episode was about Amazon, Facebook and Sleep.

Two of the biggest companies in the world and one of the most important activities we all do.

Sleep is so important. Not one bodily function isn’t affected in some way by sleep. We could all operate a little better if we paid more attention to our sleep.

Amazon is so big. They have 560,000 employees. How does someone even manage all the thoughts going through their head? I don’t know how Jeff Bezos does it. One of the reasons I wanted to talk about Amazon was so I can absorb some of Jeff Bezos’s wisdom. I love his philosophy of every day being Day 1. This is how I try to live my life.

Facebook is even bigger than Amazon. Not by employee count or market cap. By users. By the amount of time people spend on their services. They own half of the top 10 apps on the App Store. It’s likely if you’re reading this you use a Facebook product daily.

That’s enough of an introduction, here’s a more formal account of what we talked about. I’ve included all the links to anything and everything I could remember.

If you want to get on the show or give us some advice, send an email to daniel@mrdbourke.com. If you send an audio file of your question, we’ll do our best to put it on the show.

One more thing, we need a show name. I was thinking Banter with the Bourke Boys? Bourke Brothers Banter? Or something simple like 3ish things? I’d love to hear your ideas.

Show Notes

Will’s shirt is from the MIT AGI course available FREE on YouTube - https://agi.mit.edu/

Amazon

Amazon 2016 shareholder letter - bit.ly/amazon2016letter

Amazon 2017 shareholder letter - bit.ly/amazon2017letter

Amazon Prime is at 100 million members globally (~$9B RR).

Third-party sellers sold more on Amazon than Amazon itself in 2017 (the first time this has happened).

Every day is Day 1 at Amazon.

Four principles of what makes a great business (from Amazon’s 2016 shareholder letter):

  1. True customer obsession: are you delighting your customers?
  2. Resist proxies: do we own the process or does the process own us?
  3. Embrace external trends: if you fight external trends, you’re probably fighting the future. Embrace external trends and ride the tailwinds of society.
  4. High-velocity decision making: if you’re 90% sure of the decision you’re about to make, you’ve probably been waiting too long. 70% of information is enough. If someone doesn’t agree with you 100%, rather than trying to convince them and wasting time, disagree and commit.

Amazon Prime is in 2/3 US households.

Daniel shared the story of his experience with AWS. He ran up a large bill unexpectedly and contacted support to have the bill cancelled. A testament to Amazon’s true customer obsession.

AWS Sagemaker - an online platform to get machine learning models trained, running and deployed quickly. Even for non-developers.

AWS has $20B yearly revenue.

Google is trying to catch up to AWS in the cloud space - Daniel experienced this at the Google Cloud Event he attended. Both AWS and Google are offering world-class Machine Learning APIs and services but AWS still has a larger market share.

Rule of thumb: your net worth 10 x your recurring revenue (RR). E.g. if you have $100M RR, you have a $1B company. - Will

Amazon buying Whole Foods is a big plus. Making organic, quality foods more accessible is incredible. Amazon also now has 100+ different locations around the US it can use as midway points for its other services. By buying Whole Foods, Amazon saved itself 10-years trying to build the infrastructure themselves. - Will

Amazon has 560K employees by is also patenting and building technologies which automate a lot of other jobs. Eliminated jobs aren’t thought of as often as created jobs. For every 1 job Amazon creates, it may eliminate 3 middle management jobs. - Will

Amazon acquired Kiva robotics (now Amazon Robotics) in 2012 for $775M and cancelled all other existing contracts the company had. Now Amazon’s warehouses and fulfilment centres are full of robot workers.

Facebook

Facebooks 2018 F8 event - https://www.f8.com/

From the event:

Facebook wants to use their in-app camera, computer vision and your old photos to reconstruct your living room (virtually). Can you imagine ads targeted directly by what kind of fridge you’re using?

You can now clear your browsing data on Facebook. I’d love to clear my data on Facebook, that may actually get me to start using it again. I’ve got a lot of crap built up. - Daniel, This isn’t a new feature. - Will

Facebook is hiring 20,000 people to manually audit newsfeed items and ads to reduce fake news and political issues. Could this be a great way for them to keep governments and the strong grip of regulation off their heels? - Will

Facebook is also using AI to detect fake news and get them out of the feed. - Daniel

Group video chat is coming to Messenger, Instagram and WhatsApp. I saw this as a given, especially with Facebook’s massive push towards video lately. - Daniel

FAIR (Facebook Artificial Intelligence Research) replicated DeepMind’s AlphaGo and open sourced the code.

Who else wants to see OpenGo vs AlphaGo? Let’s make this happen!

FAIR also released an updated version of their open source reinforcement learning framework, ELF (Extensive, Lightweight, Flexible) along with OpenGo.

Open source is key. Imagine if you were a youngster building things and using Facebook’s open source technologies, where would you want to work when you’re older? - Daniel & Will

Sleep

Joe Rogan Podcast with Matthew Walker PhD on Sleep.

School test scores went up when the school time started later (from 7:30 am to 8:30 am). - Will

1 in 20 doctor interns who work sleep deprived will accidentally kill someone on the job, 1 in 5 will accidentally injure someone.

Challenge

1. Visit one of the resources we’ve linked above and send us your favourite takeaway.
2. What advice do you have? What worked? What didn’t work? What would you like to see more of?

Send anything you’d like to daniel@mrdbourke.com.

Listen to the show:

Watch the show:

The Key To Life

It's simple.

DO REALLY F*CKING HARD THINGS.

Life is suffering.

Regardless of how you look at it.

Why would some magical force give birth to us as creatures to go through life trying to come to terms with death?

We wake up in the middle of the night, body aching, busting to go to the toilet. Suffering.

As we get older, the one thing that makes us who we are begins to shut down. Memories fade. Thinking gets harder. Suffering.

Work is ingrained in us. Our bodies are cranes by design - muscle in our lower backs, bending knee joints - we were built to work. Work is by definition hard.

Don't even get me started on thinking. Thinking is so hard! Why the hell did I choose to keep going with that project? I knew I should've stopped but I didn't. And why isn't Mark returning my calls? He must not like me!

But when you do really f*cking hard things by your own choice, suddenly everything else seems a little easier.

There are things which still suck but the scale is vastly different. It's hard to complain about having to walk somewhere when your car breaks down if you've run a marathon.

It seems counterintuitive but choose voluntary suffering wherever possible.

Voluntary suffering helps to prepare you for the inevitable chaos of life.