Machine learning and compression have always been closely tied together. It’s trying to learn the “rules” that describe the data rather than memorizing all the data.
I remember implementing a paper older than me in our “Information Theory” course at university that treated the creation of a decision tree as compression. Their algorithm considered sending the decisions tree and all the exceptions to the decision tree and the tree itself. If a node in the tree increased the overall message size, it would simply be pruned. This way they ensured that you wouldn’t make conclusions while having very little data and would only add the big patterns in the data.
Fundamentally it is just compression, it’s just a way better method of compression than all the models that we had before.
EDIT: The paper I’m talking about is “Inferring decision trees using the minimum description length principle” - L. Ross Quinlan & Ronald L. Rivest
Thank you for this contribution! I was familiar with the idea of ML models capturing a compressed snapshot of the data, but that work on exploring its limits in DTs looks very interesting.
This piece was written by a highly-regarded scifi author a year and a half ago. I say that not to complain about the age but rather to marvel at the authors ability to describe so well something that is only becoming clear to many a year and half later.
I had an idea recently of describing these chatbots as holograms.
Complex ideas and concepts are being flattened. Depth, a dimension if you will, in the form of context and conception, is being removed.
Like how a 3D object gets flattened on to a 2D plane, a hologram.
What a fantastic piece by a really good author. Worth reading in its entireity, long as it is.
