Hopfield Networks and Associative Memory Quiz
201 questions · AI
Score: 0 / 0
1. What computational challenge does the brain solve efficiently that would be impossible through exhaustive search?
2. How is Levinthal's Paradox of Protein Folding similar to the problem of memory association?
3. In the context of Hopfield Networks, what does the 'energy landscape' represent?
4. What two key things are needed for associative memory in Hopfield Networks?
5. What states can neurons take in a basic Hopfield Network?
6. What characterizes the connections between neurons in Hopfield's original formulation?
7. What happens when the connection weight between two neurons is positive (excitatory)?
8. How is memory stored in a Hopfield Network?
9. What is the approximate maximum number of patterns a Hopfield Network can reliably store?
10. What happens when too many patterns are stored in a Hopfield Network?
11. What famous neuroscience principle is reflected in Hopfield Network's learning rule?
12. How does a Hopfield Network perform pattern completion?
13. What is the primary focus of dynamical systems theory?
14. What are state variables in a dynamical system?
15. In a basic pendulum system, what are the two state variables needed?
16. What is a differential equation in the context of dynamical systems?
17. What is the phase space of a dynamical system?
18. What is an equilibrium point in a dynamical system?
19. What is a limit cycle in a dynamical system?
20. In the predator-prey model example, what causes oscillations in population sizes?
21. What is the significance of a phase portrait in understanding dynamical systems?
22. Why do we often use numerical methods to solve differential equations?
23. What key historical shift in AI capabilities surprised researchers regarding creativity vs. practical tasks?
24. What are the two key modifications that transform a Hopfield Network into a Boltzmann machine?
25. How does the Boltzmann distribution relate energy to probability?
26. What is the key difference between Hopfield Networks and Boltzmann machines in terms of memory?
27. What is the 'temperature' parameter's role in a Boltzmann machine?
28. What is the contrastive learning rule in Boltzmann machines?
29. What is the purpose of hidden units in a Boltzmann machine?
30. What is a Restricted Boltzmann Machine (RBM)?
31. What advantage does an RBM offer over a standard Boltzmann machine?
32. How does the learning process in Boltzmann machines differ from Hopfield Networks?
33. What fundamental challenge do both brains and AI systems face when processing sensory input?
34. What is a key advantage of using parametric distributions in machine learning?
35. Why is the Gaussian (normal) distribution particularly useful in modeling?
36. What is the 'curse of dimensionality' in the context of sampling latent spaces?
37. What is the key insight of importance sampling in variational inference?
38. What are the two components of the Evidence Lower Bound (ELBO)?
39. Why is Jensen's inequality important in deriving the ELBO?
40. What role does the recognition model (Q) play in variational inference?
41. Why is the complexity penalty term important in the ELBO?
42. What is the relationship between variational free energy and the ELBO?
43. What is a key difference between frequentist and Bayesian interpretations of probability?
44. What is the fundamental requirement for any probability distribution?
45. What does entropy measure in the context of probability distributions?
46. How is surprise mathematically defined for a probability distribution?
47. What is cross entropy measuring between two distributions P and Q?
48. Why is KL divergence useful in machine learning?
49. Why do many machine learning systems optimize cross entropy instead of KL divergence?
50. What is a key property of cross entropy compared to regular entropy?
51. What makes continuous probability distributions different from discrete ones?
52. In the context of generative AI, what does sampling from a probability distribution mean?
53. When was the modern formulation of backpropagation first published?
54. What is the primary purpose of a loss function in machine learning?
55. What is a derivative in the context of optimization?
56. What is gradient descent?
57. What is the chain rule in calculus and why is it important for backpropagation?
58. What is meant by the 'forward pass' in neural networks?
59. What happens during the 'backward pass' of backpropagation?
60. How do gradients propagate through an addition operation in a computation graph?
61. How do gradients propagate through a multiplication operation?
62. What happens to gradients when a node has multiple output branches?
63. What key innovation did OpenAI's O1 model introduce regarding model responses?
64. What is 'test time compute scaling' in the context of O1?
65. How does DeepSeek R1-0's training approach differ from traditional supervised learning?
66. What are the two main components of R1-0's reward signal?
67. Why did DeepSeek choose simple rule-based rewards instead of model-based rewards?
68. What is the purpose of the 'clipping' term in GRPO (Group Relative Policy Optimization)?
69. What key limitation did R1-0 exhibit that led to the development of R1?
70. What is the significance of AlphaGo in relation to reasoning models?
71. How does DeepSeek R1 improve upon R1-0?
72. Why has next-word prediction been such an effective training objective for language models?
73. What is the key limitation of traditional next-word prediction models?
74. What is the fundamental difference between chat models and reasoning models in terms of their scaling paradigm?
75. How should you prompt a reasoning model like O1 differently from a chat model?
76. What is the significance of the 'reasoning effort' parameter in O1?
77. What type of tasks are reasoning models particularly well-suited for?
78. How do reasoning models handle coding tasks differently from traditional chat models?
79. What role can reasoning models play in agent systems?
80. Why might reasoning models be particularly useful for medical analysis?
81. How does the interaction mode differ between chat and reasoning models?
82. How does O1's approach to post-training differ from traditional LLMs?
83. What is unique about O1's inference process compared to traditional LLMs?
84. How do O1's pricing and token usage differ from GPT-4?
85. What distinguishes O1 Mini from the full O1 model?
86. What suggests that O1 might be doing multiple passes during inference?
87. How does O1's performance compare to GPT-4 on different tasks?
88. Why does OpenAI not show the hidden chains of thought in O1?
89. What pattern appears in O1's problem-solving approach?
90. What implications does O1's success have for model scaling?
91. How might O1's approach influence future agent development?
92. Why has next-word prediction been an effective training objective for language models?
93. What are the key components needed to implement self-attention in a transformer?
94. What is the purpose of the scaling factor in scaled dot-product attention?
95. Why are residual connections important in transformer architectures?
96. What distinguishes layer normalization from batch normalization?
97. What are the two main stages in training a ChatGPT-like model?
98. Why is tokenization important in language models?
99. What is the role of positional embeddings in transformers?
100. What is the purpose of multi-head attention in transformers?
101. Why is the feed-forward network used between attention layers?
102. What makes BPE tokenization different from character-level tokenization?
103. Why do some non-English languages perform worse in LLMs?
104. What causes issues with arithmetic in LLMs regarding tokenization?
105. What is the purpose of the merges dictionary in BPE tokenization?
106. What was the issue with Python code tokenization in GPT-2?
107. How did GPT-4 improve upon GPT-2's tokenization?
108. What happens when adding special tokens to an existing model?
109. Why can't vocabulary size be infinitely large?
110. What causes the 'trailing whitespace' warning in GPT models?
111. What is an 'unstable token' in the context of tokenization?
112. What is the primary role of attention in transformers?
113. What are the three key matrices involved in a single attention head?
114. What is the purpose of the Query matrix in attention?
115. What happens in the attention pattern after computing Query-Key dot products?
116. Why is masking used in transformer attention during training?
117. What makes multi-head attention different from single-head attention?
118. How does GPT-3 factor the Value matrix to improve efficiency?
119. What is the relationship between context window size and computational complexity in attention?
120. How do position embeddings contribute to transformer attention?
121. What distinguishes cross-attention from self-attention?
122. What is the fundamental purpose of backpropagation in neural networks?
123. How does stochastic gradient descent differ from standard gradient descent?
124. Why are mini-batches used in neural network training?
125. What determines the relative importance of weight updates during backpropagation?
126. How does the phrase 'neurons that fire together, wire together' relate to backpropagation?
127. What makes labeled training data crucial for supervised learning?
128. Why does backpropagation process the network from output to input?
129. How do bias adjustments differ from weight adjustments in backpropagation?
130. What role does the cost function play in backpropagation?
131. Why is the gradient interpreted as a measure of sensitivity?
132. What is the primary purpose of a cost function in neural networks?
133. Why do neural networks use continuous activations rather than binary ones?
134. What does the negative gradient indicate in gradient descent?
135. Why might gradient descent end up in different solutions from different starting points?
136. What role do local minima play in neural network training?
137. Why is it significant that neural networks can learn digits without explicit pattern rules?
138. What insight did research reveal about networks trained on randomly labeled data?
139. What limitation of simple neural networks is revealed by their response to random noise?
140. What determines step size in gradient descent?
141. Why is the MNIST dataset particularly useful for teaching neural networks?
142. What is the significance of the chain rule in backpropagation?
143. Why do we separate the computation of z (weighted sum) and a (activation) in neural networks?
144. How does the size of the cost derivative relate to prediction error?
145. What changes mathematically when moving from single-neuron to multi-neuron layers?
146. Why must we sum over multiple paths when computing derivatives for hidden layer neurons?
147. What is the relationship between a neuron's previous layer activation and its contribution to weight updates?
148. How do bias derivatives differ from weight derivatives in backpropagation?
149. Why do we compute derivatives with respect to z instead of directly with respect to weights?
150. What role do activation function derivatives play in backpropagation?
151. Why do we need to average gradients over all training examples?
152. What is an activation in the context of neural networks?
153. How do weights function in neural networks?
154. What is the purpose of the bias term in neural networks?
155. Why might we organize neural network layers hierarchically for image recognition?
156. What is the significance of representing weights as matrices?
157. What advantage does ReLU activation offer over sigmoid?
158. What determines the number of weights in a fully connected layer?
159. Why do neural networks use continuous rather than binary activations?
160. What makes neural networks suitable for pattern recognition tasks?
161. How does matrix representation help with neural network implementation?
162. What is the main role of Multi-Layer Perceptrons (MLPs) in transformers?
163. How do MLPs process input vectors in transformers?
164. What role does the ReLU function play in MLPs?
165. What is the significance of superposition in neural networks?
166. How do facts appear to be encoded in transformer MLPs?
167. What happens to the dimensionality in an MLP's up-projection?
168. How do bias terms function in MLPs?
169. What proportion of a transformer's parameters typically reside in the MLPs?
170. How do MLPs process tokens differently from attention layers?
171. What advantage does high dimensionality provide for storing features?
172. What is the core function of a large language model?
173. How do chatbots typically generate responses?
174. Why is randomness important in language model outputs?
175. What are the two main stages in training modern chatbots?
176. What makes Transformer models different from earlier language models?
177. Why do language models convert words to number lists?
178. What is the purpose of attention in Transformers?
179. Why is it challenging to understand specific model predictions?
180. What enables the massive computation needed for language model training?
181. How does the training process modify model parameters?
182. What does GPT stand for in Large Language Models?
183. How do word embeddings encode semantic relationships?
184. What determines a transformer's context window size?
185. What is the role of the embedding matrix in transformers?
186. What does the temperature parameter control in language model outputs?
187. What is the purpose of the Softmax function in transformers?
188. Why are dot products important in word embeddings?
189. What is the unembedding matrix used for?
190. How do transformers handle text generation?
191. What happens to word embeddings as they flow through a transformer?
192. What key innovation did DeepSeek introduce with multi-head latent attention?
193. What is the purpose of KV caching in transformers?
194. How does KV cache size scale with traditional transformer architecture?
195. What distinguishes multi-head latent attention from multi-query attention?
196. What improvement did DeepSeek achieve with multi-head latent attention?
197. How does attention pattern size relate to input length?
198. What problem does grouped query attention solve?
199. How does DeepSeek's approach handle the compute vs memory trade-off?
200. Why do attention patterns only need to compute new rows during generation?
201. What determines the size of an attention pattern matrix?