[Google Cloud Skills Boost(Qwiklabs)] Introduction to Generative AI Learning Path - 6. Encoder-Decoder Architecture

https://www.youtube.com/playlist?list=PLIivdWyY5sqIlLF9JHbyiqzZbib9pFt4x

Generative AI Learning Path

---

Encoder-Decoder == Sequence-to-sequence

• Encoding stage
  • It produces a vector representation of the input sentence.
• Decoding stage
  
  • It creates the sequence output. 

 

Both the encoder and the decoder can be implemented with different internal architectures.
The internal mechanism can be a recurrent neual network(RNN).

 

RNN encoder takes each token in the input sequence one at a time and produces a state representing this token as well as all the previously ingested tokens. Then the state is used in the next encoding step as input.

 

- Decoder generates at each step only the probability that each token in your vocabulary is the next one.

- Using these probabilities, you have to select a word and there are several approaches for that.

- The simplest one called Grid Search is to generate the token that has the highest probability.

- A better approach that produces better results is called Beam Search.

- In that case, you use the probabilities generated by the decoder to evaluate the probability of sentence chunks rather than individual words.

---