Theory evaluation

Theory evaluation can be approached from both empirical and analytical directions. Within empirical evaluation, experience with the theory is used to discover the strength of the theory. Analytic evaluation analyses the formal structure of the theory and tries to draw from this an assessment of the theory.

The first step in theory evaluation is the identification of theories that explain or predict the type of behaviour of a natural system we are interested in. If no theories exist then one must be developed. Once a set of theories has been identified for the problem then the criteria for prediction, assumptions, consilience and simplicity can be used to select the most suitable one. This is a complicated evaluation, because most of the features are incompatible; a theory can never score best in all categories.

  1. Unambiguous encoding vs. consilience: concrete entities are easier to measure than abstract entities, but the theory is more specific for that reason.

  2. Preciseness of prediction vs. measurement effort: Precise predictions require a faithful characterisation of the natural process and precise input, but precise measurements are more difficult to make.

  3. Strength vs. consilience: A theory that is able to make predictions using fewer observables than another theory often needs more knowledge about a specific natural process, as a result it will be less consilient.

  4. Simplicity vs. consilience: A theory that makes more predictions based on less empirical information often has to make more assumptions about the natural process.
The most consilient theory explains the `most' explananda. In terms of information-sets this is the theory that can predict observables for the greatest number of natural processes. The strongest theory predicts the the observables using the fewest observations. This is often related to the assumptions of the theory. If a theory introduces an axiom in order to make a prediction, then this reduces its simplicity. Then there is also the issue of the measurability of the empirical concepts postulated by the theory. If these are difficult, costly or dangerous to measure at required precision, then the theory cannot be applied.

Analytical evaluation involves checking the a priori structural correctness of syntactical operations on statements of the theory. A priori because they are true or false as a matter of logic alone. The more that is known about the structure of the theory, the more can be inserted as logical statements and checked analytically. Consistency and completeness are examples of structural criteria.

Reasoning with universal laws represented by implications in predicate logic results in `tautologies' that are valid if the axioms are valid. In this case the evaluation process can concentrate on empirical validation of the applied laws and axioms.

If the applied inference over implications is not strictly deductive, but applies inductive, abductive or analogical reasoning steps, then the inferences of valid propositions will not necessarily lead to valid conclusions; the theory could be making incorrect predictions. In this situation, in addition to the validation steps required for the previous case, every abductive, inductive or analogical reasoning step needs to be checked a postiori for its empirical validity.

Statistical laws come with a probability for a defined type of problem set. Deductive inference over statistical laws will preserve this probability. Abductive, inductive an analogical reasoning with statistical laws leads to conclusions whose probability is not preserved. In this case the validity of the analogy should be checked empirically for every analogical reasoning step.

Tue Jan 25 10:37:49 MET 1994