3 Introduction to Statistical Learning Theory
Statistical Learning Theory, or SLT for short, is the flagship interpretation and framework for learning theory, further specific in diagnosis and identification of efforts utilized by artificial intelligence researcher to search for a framework of a learning problem - how do you make a non-biological substrate or construct created from such, to be adaptive, to extract and within given reasons, to replicate what the data suggests? Further onward, what is the domain and identification thereof, such that it extracts structure underneath the information - which is called in recent terms as generalization? Such question is posited by statistical learning theorist, and populated in a rather long time, prominently as Valiant (Valiant 1984), V. Vapnik (Vapnik 1999), (Hajek and Raginsky 2021), Mohri and Afshin (Mohri et al. 2012) Kearn & Umesh (Kearns and Vazirani 1994), Peter (Peters et al. 2017), and so on (like (Luxburg and Schoelkopf 2008)). As far as everyone can consider, it is a rather popular and deeply analyzed theory on its own.
The forward introduction to the subtopic and the sub-branch field of learning theory to be, can be found in many textbooks, most often for example, Foundation of Machine Learning (Mohri et al. 2012), Understanding Machine Learning: Theory and Practice (Shalev-Shwartz and Ben-David 2014), and Statistical Learning Theory lecture notes by Bruce Hajek and Maxim Raginsky (Hajek and Raginsky 2021). However, we can see that there exists many, often distinctive approach to interpreting and formalizing the statistical learning picture, such that would be dissected and analyze on first account of the following section, so that also to give you a sense of the problem setting that the theory is trying to produce. Thereby, conducting a survey on how the setting is settled and how the general broad approximating baseline is constructed, would be prominent in understanding different variations of such, because for once most of them are rather a simple re-resolution of a single core of assumption baseline on both structure availability, and analytical form, for example, the Neural Tangent Kernel (Jacot et al. 2018), which in itself is a rough boundary-based learning theory - that is, the model of learning here is constructed by redefining the boundary of what is possible. Put more plainly - NTK is the reinterpretation of the old theory, and not a lot of advancements can be said as completed in recent times toward this direction.
What, is that, and the purpose to provide an introduction and dive into the crude form of statistical learning theory is, and why would it be said that NTK is boundary-based, one might ask? Aside from reserving such spot to the further inquiry far and wide of the preceding chapters, it is also relevant and rather easy to diagnose on what is the response on such, because it is rather unique on this front.
3.1 The setting of (classical) statistical learning theory
Before such, we would like to get the reason and particular advantage on why the word ‘statistic’ is chosen here, as far as we can consider it ourselves. Furthermore, what to mean of the mathematical language sufficient to analyze it. There are indeed many way to think about the learning theory, including why we said it as a learning process, most typical to be considered in a game theoretic sense. But usually, in learning theory, we often start with the more constrained and limited case, like classification. For once, consider such as the following. For starter, we define the system of dataset as includes the input space, or the instances space, denoted by \(\mathcal{X}\), and the output space, as also termed label space, denoted by \(\mathcal{Y}\). \(\mathcal{D}=(\mathcal{X},\mathcal{Y})\) is then the dataset in general, with many pairs \[ (X_1, Y_1), \dots (X_n, Y_n) \in (\mathcal{X}, \mathcal{Y}) \] such that there exists a concept mapping \(c: \mathcal{X}\to \mathcal{Y}\) that is canonically the truth baseline of the dataset. In here, we suppose that \(c: \mathcal{X}\to \{0,1\}\) for the binary classification case of the category label. The goal, is then, given a similarly constructed hypothesis model space \(\mathcal{H}\), for \(h\) in this space, find the approximately correct mapping to \(c\in\mathcal{C}\) now is the space of all available concept of this kind. The mapping \(h:\mathcal{X}\to \mathcal{Y}\) is then called the classifier. Learning theory do this task by choosing a criterion - a fundamental evaluation standard, we can denote by \(g\). Then, this criterion is based on the following form: \[ g(h,\mathcal{D}[n]) = \bigwedge^{n}_{i=1} (h(X_{i})\land c(X_{i})) \] The function \(g\) then gauge the correctness \(\mathbb{I}_{h,c}\to \{0,1\}\), as the indicator function, of the hypothesis compare to the supposed concept \(c'\sqsubseteq c\); again, we note that this is a particular assumption thereof that should be said so that \(h\) and \(c\) architecture is the same. While we use \(\mathbb{I}_{h,c}\), it is only the numerical mapping on convenience, and can be seen purely from the logical side, for \((h(X_{i})\land c(X_{i}))\) to land fully on \(\{T,F\}\) of the classical truth value. Thus, the context of such would be evaluating between many similar resultants of truth values, being: \[ \{T,F\}_{1} \land \{T,F\}_{2} \land \{T,F\}_{3} \land \dots \land \{T,F\}_{n} \] That said, the above generalized conjunction is a rather complicated term, and it compresses from the step of \(h(X_{i})\land c(X_{i})\). So, if we want to evaluate \(g\) in a different way, for example, allow for quantification like \(90\%\) correct, then we can try, for example, \[ g(h,\mathcal{D}[n]) = \bigwedge_{i=1}^{n} {\!}^{(C)} \Delta_{[0,1]} [h(X_{i}), c(X_{i})] \] Here, \(\bigwedge_{i=1}^{n} {\!}^{(C)}\) is the extended “part of it” operator, weaker than the total generalized conjunction, and here we switched to the delta function \(\Delta\) on \([0,1]\), indicating fuzzy logic regime i.e. fractional truth. This criterion class is called observation correctness. Fundamentally speaking, we are gauging the condition of \(h\) in runtime, and its result, so much so to say, if \(h\) is fundamentally capable of being correct to specific \(c\) reflected into its proxy of the dataset, given the exact form of the input space as setting, with little variations as possible. The above notion works on quantification, we subsume the quantities and qualities that describe the interaction, in on \(\mathcal{X}\subseteq(\mathbb{N}[0,1])^{n}\), for the first wedge conjunction, while the second one, we shift to \(\mathcal{X}\subseteq(\mathbb{R}[0,1])^{n}\) for dense, real number. While elementary operations will not affect the properties that much, we note that quantification changes the configuration space and any operator acting on it, so there exists distinctively different between \(\mathbb{R}\) and any given other space, like \(\mathbb{H}\) of the hyperbolic space, or even just \(\mathbb{E}\), sometimes used as denoting for plain Euclidean space. If, one commit to the additivity aggregation action, by choosing to evaluate the criteria above as a sum of sort, thence, \[ \begin{split} g(h,\mathcal{D}[n]) &= \sum_{i=1}^{n} {\!}^{(C)} \Delta_{[0,1]} [h(X_{i}), c(X_{i})] \\ & = \sum_{i=1}^{n} \Delta_{[0,1]} [h(X_{i}), c(X_{i})]. \\ \end{split} \] Under more simplification, if we take \(\Delta_{[0,1]}[\cdot]\) as the operator on functionally real space, and use Euclidean notion, then, \[ \begin{split} g(h,\mathcal{D}[n])&= \sum_{i=1}^{n} [h(X_{i}) - c(X_{i})] &\propto \sum_{i=1}^{n} [c(X_{i}) - h(X_{i})] & \equiv \sum_{i=1}^{n} \mathbb{I}_{h,c}(h(X_{i})\neq c(X_{i})) \end{split} \] Again, the last one is the indicator function. Under this scheme, we can see that, in particular, we now have the total aggregated information about the hypothesis and the concept, based solely on how there exists of the observations, and how closely hypothesis is in its way of replicating that resultants of such, regardless of the inner working difference, of which most certainly we do not have such knowledge of the true concept. If you want to take the average, i.e. from a certain view, you want to sum them up, then divide them, we then get the averaging operator \(\mathrm{Avg}\), such that, \[ R_{\mathcal{D}}=\mathrm{Avg}(g(h,\mathcal{D}[n])) = \frac{1}{n}\sum_{i=1}^{n} \mathbb{I}_{h,c}(h(X_{i})\neq c(X_{i})) \] This, is the recovery mode of the famous empirical risk, here we call as the observational risk for an isomorphism of naming. Then, a good model, is when \(h^{*}\)
From here, the tooling is now capable to host a variety of different purposes - one of which is the probabilistic view of the statistical learning language, and we should see how that goes. Here, we also remark it briefly, that this probabilistic view is not fundamental, nor it has any justification to confirm itself to be ontological of the present time. This implies us to trek with caution, to not overclaim on the principality and commonality of probabilistic interpretation, for its ontological conundrum solution.
3.1.0.1 Probability and statistics
Statistics, as said, is a fundamental way of aggregating and so forth, of available information, and the carrier of such information. Hence, there exists particularly two types: statistically blind information, and statistically total information. For example, a statement, given knowledge under the scheme of the carrier of knowledge being English language, talks about “I am ill”. The fundamental information carried is impossible and thus is blind to be handled on the statistics side. It is only a truth value, no more, no less in the form of its functionality, and it is informing us, of the observation, the important figure, the properties of such figure, not much else. Statistically total information would then require it to be quantified - that is, we have the following observation:
Conjecture 3.1 (Statistics) We say that statistics is the fundamental analytical framework of analysis, investigation, aggregation, extraction, knowing and of obtaining reasonable type of information in regard to such object, via its container of knowledge, called statistical information of itself or system of interest. Existentially, we posit for this information depends on (numerical) quantification, i.e. numbers and quantified construct. Any information without numerical quantification is called statistically blind information (SBI), while the opposite being statistically total information (STI), and there must exist a conversion for SBI to STI. We say SBI is then useful information in the context of statistics, and not so much elevating STI, however, STI is then actionable information.
From this view, here on out, we can conjecture or in sufficient description, permits the viewing point that statistics is the framework of the general study of observations, the set of those observations, and so on. Directly on the observations, is statistics. And when observations are in numerical presentation, it functions as the actionable form for us to derive any form of knowledge, hidden in those observations. In frequency analysis of observations, for example, often time whether discrete or continuous, and is available en masse of a single characteristic of a large number of individuals, often it is necessary to condense the data as far as possible, without losing much information of interest. Which, breeds to the art of data presentation - which is part of statistics. The term distribution here is also interesting. By far, this word here only means counting, but a bit more over the top - for example, if there exists 500 in 1000 cows you see in that particular space with red colour, you say the distribution there gives you 50% of them as red. Frequency, same thing, happening in a space, a place, and a setting. Not much probability there, is it?
Insofar, we also have interesting observation, since for example, in most explanation thereof of statistics, the most mundane important part of it, is referring to “collection of procedures and principles for gaining and processing information” (see in Here, for a quick example on the very first page of the content). However, the other side, is most often, probability, i.e. for example as to make decisions when faced with ‘uncertainty’ - a concept only known in probability or rather being widely popularized of sort. So, where does probability fit into this place?
3.1.0.1.1 Probability
While statistics down the same line can be said to concern with data analysis - on how to make sense of the data, to extract patterns presented into data. However, this process of data analysis, provides questions - questions about the general idea, framework, concept, truth, ‘what is going on’, and so on, of the statistics. For example, in the task of, say, analyzing on ‘how many cows are there in America, and what types of them are?’, the usual typical observation and such gives you a plenty bunch of grounds to ask further questions, like ‘why so’, ‘what is the relatable factor that determines this result we see that red cows are not so much?’, why it is so, what is the possible explanation? What can support more, what can we extract more about the information? In doing so, you are shifting from just looking and analyzing the data to interpreting it. Logic can do this part, for example, when you can state of a given fact - the fact that we aggregated that we are 50 people wet in a suit, is because, by a fact that it just rained. However, when you consider yourselves in the position of something more unverifiable, uncertain, for example, a man with hallucination, such logical line is then cannot be trusted. Because you do not know, thus consider the action forward is that - how much can I trust this? Similar in gambling where the odds and the thrill of such are in the unknown, where you are hoping for something, because you are withheld of such, what you do not know, you have to forgo a bit of the fact presented of the logical system. The recovery apparatus, partially or so, and many in one, is what we called, probability.
Probability enters statistics as the interpretation of what it means. It covers, the logical line, under the notion of uncertainty, and the loss of logical consistency, as said of the above. In doing so, we can basically say so as of the following logical necessity. Suppose something, called \(A\) happens to be able to get two absolute values. Either \(T\), or either \(F\). Non-probabilistic logic dictates that such can be separated to two statements, and the world is traceable and determinate enough, such that it can be concluded of “\(A\) is \(T\)”, or “\(A\) is \(F\)”. However under such circumstances of what we call that the system is uncertain, traceability and determinability are not sufficient, leading to us unable to make sure if the fact can be reflected and concluded in a singular form, or any given form. This is where probability comes in as the interpretation to interpret this particular non-collapsible phenomena as a system under uncertainty, as a system with probable nature.
A probable nature of a system depends on two things. First, the determination of the probable system and what it constitutes. Insofar, we can consider the definition of probability, as now the noun of this particular property of any given system, as the following:
Theorem 3.1 (Probability) Probability, as for the notion of the property of being probable of a given system. Is an interpretation of such system as for its nature, or properties, or structures, and of its mechanisms under observation and resolution, for both terms mention and collude with the existence of said system, its states, and the different properties of itself independent of a given reference point, or an observing system.
The two main paradigms, frequentism and Bayesian probability, can be explained as the same thing, but under different reference point of the above definition philosophically speaking. Frequentist believes in their interpretation, from the focal point of the observing system, as undisturbed of the “parameters” and properties, structures underneath - governed by truly just the observation itself and the problem for them, as said, is the purity of the observation. Bayesian, on the other hand, thinks that such is mutable and dependent on the non-convergence of observing system. That is, frequentist believes in such implicitly defines all observing system and convergent to a same reference point, because such is irrelevant in the face of the observed system that they are inspecting. Bayesian, at that, allows for pluralism of reference points, such that there exists subjectivity, there exists prior, and there exists posterior. Many more interpretations of probability, like likelihoodism, or epistemic probability, works the same way of considering this particular system on what under observations (existence) and the structural properties (system structure) question of all, which is quite reflected in the above definition of the philosophical insight on what probability is.
We do not start with any amount of mathematics, because such definition defines the philosophy, the architecture of the mathematics no less, and such will only follow, if we know and commit to a particular version of the world, the system, the notion of probability as considered, how we think of the precursor form that such space of all consideration works, and the properties of a given system and environment of interest for its interpretation, then comes the utilization of mathematics to denote, quantify, and abstract it away, with also the same length of concern as for the actionability of such terms. In fact, if we do not start with this, we forgo the invariant conjecture of mathematical language, that is, for a given object, no matter what language, only reflects part of what it is, under the limitation of a formal language, and under the limitation of the semantic of that language itself, and is then stuck with variations of calculus-probability, topological-probability and whatever you can say about it, different and separated.
3.1.0.2 The probability of statistical learning
Then, let us starts with the general lining of how, for now, the probability setting of the theory of statistical learning theory, and what is the often mainstream version of such. Kolmogorov interpretation, and so on, defines for you the second question, about the constitutions and language of function in mathematics, for the probability notion itself. We can then start, with the term probability space. A probability space is presented under Kolmogorov taxonomy as a triple \((\Omega,\mathcal{F},P)\), where \(\Omega\)1 is the set of outcomes, \(\mathcal{F}\)2 is the set of events, and \(P\) is a relational space that maps \(P:\Omega \times \mathcal{F}\to [0,1]\) from probabilities to events3. If \(\mathcal{F}\) is designed to encapsulate also \(\Omega\), then \(P: \mathcal{F}\to [0,1]\). The space \((\mathcal{F},[0,1])\) that the function maps to, is then denoted as \(\mathbb{P}^{*}\), the assignment space4.
1 Under such view, it is easy to say what each of them means. First, \(\Omega\) is presented here purely as the logical value that can be taken. If, it is a logical system, then the outcome would be \(\{0,1\}\). If it is height, it shall be something like a range \([a,b]\), or a finite amount of height value taken. Think is, \(\Omega\) and its existence posit a truly presentable presentation and enumeration of all given outcomes that one can have, i.e. the number of reality that the system might chart. The limit, to this, is that it is entirely subjective - for those outcomes to be determined, you either subsume the infiniteness assumption of the universe with every outcome possible, or you return to a fixed \(n\) observed outcome, and distinguish them for \(m\) distinct observations of occurrence. Only then, you will have \(\Omega\), and you also have no idea if the space is actually big enough. Usually, \(\Omega\) would even be quantified or so under \(\mathbb{R}\), in such case, again, there is nothing much about infinity, but ‘we are too lazy to count’. This is the first openness problem of the probability space.
2 Similarly, here, \(\mathcal{F}\) is the list of events. There exists two assumptions here. First, is that the event must happen, and its observational resultant always lands inside \(\Omega\). This is a very strong assumption and it may or may not happen, can be deceived, can be corrupted, and so on. Second, is that it introduces the second openness problem - by looking into the determinancy of how such can happen. Simply speak, \(\mathcal{F}\) is subjective - it constraints the set of all selected events rather than the list of all events in consideration. The path to \(\Omega\) can be enumerately many more than \(\mathcal{F}\), however, one can constraint itself and the environment description enough, so that \(\mathcal{F}\) can be computed and considered finite. But in doing so, one then must see that there exists other \(\mathcal{F}'>\mathcal{F}\), and thus their system is not obviously applicable elsewhere. Furthermore, since it is such system of said nature, under noise in typical probabilistic setting, like \(\mathcal{N}(0,1)\), noise can mask system function, thus the existence of \(\mathcal{F}\) will always be considered a flawed proxy instead.
3 Lastly, \(P\) assigns value on range \([0,1]\) to the system. This action gives possibility interpretation to the system and restrict itself to that the outcome ‘will never happen’, and the outcome ‘will happen’ for a specific path of the event, in its extremal form. I think this is enough to say what restricts what, and how it constitutes the third openness problem.
4 Lastly, and we also see that the encapsulation of pairing \((\mathcal{F},[0,1])\) is a real problem, and might introduce the fourth openness problem if tred carefully. However, this is a statistical learning theory coursebook. I would not bother you with that much details of probability.
Under this system, we assume typically, that \(\mathcal{F}\) is a \(\sigma\)-field (or called \(\sigma\)-algebra), as a non-empty collection of a subset of \(\Omega\) that satisfy its axiom. Without \(P\), \((\Omega, \mathcal{F})\) is then called a measurable space. Why the name? Essentially, it is a space, went through a quantification system, and it is concretized as such, and is given a good enough space that is not pathological or so. The term measurable here is a problem. Because of the language, which is related to measure theory, its fundamental purpose is to gauge the size or volume of any given set it is taken on. Because the notion depends on the notion of geometry - whether you like it or abstract it away enough, the quantification exists in a singular form, yet because you are gauging size and volume, instances of such would be required to be able to add up consistently, and this is a particular problem, because measurability means you need to take into account of the many-parts, with a measurability of something, to be taken as references, or of a given ‘geometry’ or shape. This, is the measure that we choose to give to the measurable space, and in doing this, we have the assumption that this space, before a measure is colluded, is indeed capable of handling a measure set function, that is, \(\mu:\mathcal{F}\to \mathbb{R}\), with, for example,
- \(\mu(\Omega)=1\),
- \(\mu(A)\geq \mu(\varnothing)=0\) for all \(A\in \mathcal{F}\),
- if \(A_{i}\in\mathcal{F}\) is a countable sequence of disjoint sets, then
\[ \mu(\cup_{i}A_{i}) = \sum_{i}\mu (A_i) \] Using this space, we can now add interpretation to the above scheme of empirical risk and so on. The criterion of the statistical learning problem is mostly \(R(h,c)=R(h)\), that is such that, \[ R(h) = \mathbb{P}[h(X)\neq Y] \equiv \underset{h\in \mathcal{H}}{\mathbb{P}} [h(X)\sim c(X)] , \quad (X,Y) \in \mathcal{S} \] Under simple classification on binary value, the outcome space, is \(\Omega=\{0,1\}\). The events, then, are those under \(X\). Thus, \[ R(h) = \underset{X\in \mathcal{X}}{\mathbb{E}} [\Delta(h(X),c(X))] = \mathbb{E} [\mathbb{I}_{h,c}] = \mathbb{E} [\mathbb{I}_{h(X)\neq Y}] \] We can decompose the probability \(\mathbb{P}\) in here into \(\mathbb{P}(X)\times \mathbb{P}(Y|X)\), that is, the probability of \(X\) existing, but also the one probability of the concept fitting there, given what is available of \(X\). We can also decompose this so much that, under the probabilistic view by then, \[ \mathbb{P}(g(X),Y) = \mathbb{P}(X) \times \mathbb{P}(Y) \times \mathbb{P}(Y|X) \times \mathbb{P}(X,Y) \times \mathbb{P}(X|g(X)) \times \mathbb{P} (\mathcal{H}|\mathcal{S}). \] This is the probability settlement reached, under the hypothesis that the hardness of the hypothesis configuration, the probability that the event of non-fitting happens (that is, \(g(X)\neq Y\) as stated to be an event), the abstract probability that the input and the model is even making sense,, the sensibility of the dataset itself, and the probability of the existence and density of the dataset are correlated.
3.1.1 Observations and the statistics of data
As stated in the previous chapter about the classical theory of learning, a particular way of learning can be considered as to learn from observation. That is, we presuppose the existence of the object set \(\mathrm{Obj}[i][j]\) for \([i]\) components and \([j]\) given exposure - the available ports that you can extract its status and what it presents at any particular moment of inspection, or so call the act of observing to exist and can be captured. Such then any result given from said act is called an observation. Such information is the reflection of the object at will, in which we will call the collection as \(\mathcal{S}\), or dataset which contains information about said interaction or state of the system including the object and its context of operation, and the optional ‘timestamp’ of the system and object’s state. This can be given by in the precondition that you have the exposure guaranteed as said and be measurable, and you observe the object in a surrounding environment conditioning of such, assuming you can separate this discretely, of which we would name it as the domain separability hypothesis, which is often used plenty in statistical distribution or so.
These observations are important, because they define, or at least give us information, here means any given description or inquisition of such, about the object, so that for us to know about the object and its surrounding interest. Then, information here is to say of the criteria to know - that is, the necessity to capture something. Thence, we defined information per context by three factors:
- The knowing of interest.
- The scope of the knowing.
- The presentation of the knowledge.
Thus, such is information, and observable provides information about a system’s state in a singular form, and in plural form comes ‘connected information’, that is, aggregated information coming out from this knowing. We can say it is the separation from first-level knowing, and the second-level knowing. For example, first-level knowing is the raw information given of its context, and its behaviours, while higher-level knowing is the informed picture if you get multiple information in together, thus creating a dynamic picture where you now can know ‘how it moves’, ‘what it becomes’, and ‘what it does’. Such is the importance of information and given such, observations, and the many-collection thereof, provides us with a canvas to extract its principles and so on. If a canvas of observations can now be called as observation space, then if the observation apparatus, we can say as \(\mathrm{ObA}(\cdot,\dots)[f]\) by a framework or methodological system \(f\) (also can be called utilization and so on), then if \(\mathrm{ObA}\) capture everything possible of all information, all components and all congruent ‘being’ that the object posit of its existence, then we say the observation is complete. This is the apparatus completeness of the observable theory, and if a system is with observations \(w\in \mathcal{S}\), then if properties from domain \(f\) can capture everything such that there exists a mutation process \(\mathrm{Mu}(w,f,\dots)\) that mutates the system using information extracted from an observation extractor \(\mathrm{Extr(\mathcal{S},\dots)}=\mathrm{Prop}_{f,\mathcal{S}}(\mathrm{Obj})\) that defines the components of \(\mathrm{Obj}\), and the next observable after \(\mathrm{Mu}\), i.e. \(w'\), can be fully explained by what extracted i.e. \(\mathrm{Prop}\), then we say the apparatus and the underlying analytical engine is given domain completeness. Given such, we can say per perspective, observable is the silver lining of any given process or system sought to either analyze it, or at least capture what it is. Including logic, including facts, including situations or any form of actions toward certain setting.
What is the observation of its properties, then? The observation contains many properties, however, again, most of the time under specification and of concrete understanding, observation can be described loosely, and imperfectly congruently minimal as the following substrate. We note that this is not primitive nor it is fundamentally irreducible, it is merely subjective inference and totality of such, being a flawed by-this-point observations, ironically, of the notion of observations thereof.
3.1.1.1 Language and its actionability
We do have to put a cautionary tale sign on top of this, for distinction and de-complication or decoupling of sort about the importance of language. Specifically, it is called, in particular, as the problem of groundlessness of the language. Arguably speaking, language in itself is a form of information carriage. Under such view, the meaning of the language comes from the reality it is subjected to encode, by and from the writer whom wrote it down. In such sense, the origin of the language’s meaning is the one using it, not the language. Without the reader, the language bears no meaning. Without the writer, the language does not exist. Actionable language, in such taxonomy, is the reader on its own, so much to execute its own command given - and thus if the writer dies, the system will still retain its control and meaning under existence.
Language has many forms, and in a sense it is used to provide information, but also action and procedure per se, to a given engine or interpretability framework, to actionalize the system toward a given state or configuration. We then can separate analytical information, information that gives rise to description and knowledge about a system, and actionable information, those that functions as also instruction or informant to, say, the course of action or guideline thereof. Those are two fundamental discrete types of information in the respective viewline of what they contain, and how active they are. Most information given in dataset are to be said, analytical information, while the language specification of machine learning model, can be considered as actionable information - they contain the function and describe the working of the system, of the model, and else.
That said, the view can be also reverted. For some, actionable information can also function and be interpreted as analytical, because of the apparatus of observation. For example, one can see the function getThingsdone() and be informed of what is the shape of the model, why it is designed to be this way, or such. On the flip side, analytical information interestingly, cannot be transferred or analyzed, or in any given sense, actionalized smoothly into actionable information, because their mode of being is informative, and the transition of such requires an additional layer of interpretation and extraction to know what to do with said information. Such is the asymmetry of information archetype, and within such, we can see that embedding information with action is often sometimes more informative than the inverse of sort, if given plainly of ‘fastest way to transform’ information. That said, it is also more so of the ‘ease of operationalize information’, and this comes at a place we can say as the no-free-lunch of information archetype. Loosely speaking,
For any given form of analytical information, description comes at the cost of more baggage for extensions and action-forming thereby, hence theoretical infinite actionable course. Actionable information, on the other side, provides ease of transformation and thereof, while provide exponentially and theoretical infinite many interpretations, by virtue of infinite meaning ‘does not bother to count’.
Such is the tradeoff that would be at least symmetric to the given archetypical separation there is. However, the operational cost is immensely asymmetric toward the analytical side, which is often lower and provide more straight-forward semantic. That is, when you run the system, this comes as the Eulerian asymmetry on operational cost, that information with execution semantic, somehow, provide adequate execution capacity, while maintaining infinite interpretation. Meanwhile, data of plain descriptions and no action, contains infinite course of action on nondeterministic (by this we mean, per instruction of what to do) ground, and contain the further indeterminacy of the system component in runtime - knowing a car has four wheels, and many things, does not render it useful to see it as a car, essentially. The line between these two archetypes are very blurry, and often we would see many to be said and to be seen as a mix of both. However, languages and so on that opt for the extreme, will present these tradeoffs. The question, then, is that whether we can find a language that is sufficient to describe the observable, given in the previous section, that can provide us with both? And can we find a language, that can construct a given being, that retains both semantic on operations, and its inherent meaning? Those are two open problems, and the prospect of the second one failing, is much higher than the first. And then, the third problem to be - can one information type full describe the others and functions as the other?
3.1.2 Learning theoretic
The goal of learning theory, or the baseline of what to be called as machine learning for the object as the computational machine, and the action of learning on top of it, can be defined rather by the need and necessity that given the usage of machine learning. In general, it can be considered useful to use machine learning in the specific tangent line of a setting where:
- It is too complex to program, because of either observability impossibility that is, the collapse of the domain separability hypothesis, and the collapse of observation discreteness.
- The domain completeness is not satisfied, which is the majority of all cases.
- Inexplainable phenomena or process, for example something that fails to be secularized and generalized using the information format of human language to be reduced or specified to finite steps, or any given descriptions that is actionable.