Learning

---

Echo's learning mimics the way the hippocampus in our brain works. 

Basically, the hippocampus is a component of our brain that is key for memory, imagination and learning. To create abstract knowledge, the hippocampus retrieves recent experiences from the memory, and repeats them at much higher speed to which those experiences occurred, giving priority to those experiences that lead to greater reward, activating the imagination to construct scenes for the future, and comparing those imagined experiences with what really happened in reality, all of this with the sole purpose of adjusting the responses of the neural network system.

Now, in Echo, the imagination has been removed, because it is an expensive process that requires a lot of computer power to make the avatar work. In this way, Echo goes beyond the design of our brain, to take advantage of the possibilities that comes from creating intelligence artificially.

This evolution in the design of the digital brain offers improved speed, making the avatar super light in terms of requirements of computer power for learning, and also much faster response when operating.

After all, there is no need to implement the exact same design as in our human brain, that operates with carbon material, to simulate an entity that operates with silicon material.

Learning Loop 

Please have a closer look at the image at the top of the page, which represents a learning loop with two components: the actors, that generate experiences of interaction between the avatar and the environment, and the learner that improves the responses of the deep neural network.

Initially, the behavior of the avatar is random. Every certain period of time (learning steps) the learner updates the neural network that the actors use to generate experiences. With every new version of the neural network, the actors generate experiences that are slightly better than the ones generated with the previous version. This improvement from one version of the neural network to the next one is imperceptible, but over time, the avatar improves the responses of the neural network to the point of optimizing the performance.

Key points:

• To learn, the avatar interacts with the environment
• Those interactions are stored in the avatar's memory as experiences.
• This process of interacting with the environment is an autonomous one, known as self-play.
• The objective of the Self-play is to generate and store experiences in the avatar's memory, to feed the learner.
• Let's call Actor to a machine that runs a self-play agent. There will be a number of Actors running self-plays agents to generate enough experiences to properly feed the learner.
• The most recent experiences are kept in memory, and the oldest are removed.
• Then, the learner takes experiences from the memory in batches (usually between 256 to up to 5000) depending on the computing power available for learning, and processes them to adjust the responses of the deep neural network.
• Let's call this process a single learning step.

Echo, a machine learning model, learns in a similar way to humans. This is because it incorporates the learning principles of the hippocampus, a part of the brain responsible for memory, imagination and learning.

The responses of the neural network used by Echo are initially random. Every set number of learning steps, the learner updates the neural network. This updated version is then used by the Self-Play agent to generate new experiences.

With each new version of the neural network, the quality of experiences generated by the Self-Play agent improves. Although the improvement may not be noticeable with each iteration, over time, the experiences become better and better. Eventually, the avatar will have mastered the hidden dynamics of the environment and will maximize the rewards received at the end of each episode.