Tools to Test Assumptions

Thinking about testing assumption

The post today goes into some details into thinking about how can we think in clear terms about the assumptions and indicators that we want to implement. This post is not going to be as conceptual  as the previous terms, although the concepts here remain very related to the ones that have been previously explored.

The main question this post addresses is this: it’s great that we are following through with all of these things. It’s great that we are exploring assumptions and cognizant of how indicators can petuntse the assumptions that we have laid down. We might even be doing this without consciously realizing the implications. So now the question is: how do we begin exploring our assumptions and indicators in robust ways? How do we know if the systems we want to establish with turn out in the way we truly desire?

There are entire fields of studies for this. In the post reviewing the literature of the field, I introduced a whole bunch of disciplines, the founding figures, and the seminal works in there. I’m interested in system dynamics and complexity science here, and the kinds of tools that they use to study the systems they are interested in. I’m only beginning to learn about the tools themselves and exploring how they can be used to address questions we are interested in.

There are tools for systems dynamics and there are tools to begin exploring complexity sciences. I’m still learning about the to tools themselves especially for those who are trying to learn about the disciplines themselves. After all, the only way to truly learn about these philosophies is to practice them, and the tools allow one to get into the language of thinking about these things in a real way.

For system dynamics, there’s STELLA and Vensim, and I have only used Vensim. For approaches to complexity, there’s Netlogo for agent based modeling – which allows for the easy creation of agent based models without a lot of knowledge of coding. I know that R does a few things as well, even social network analysis, but I have not used that myself, but it’s something I will be exploring. Santa Fe Institute has launched an online course, and there’s a module on learning Netlogo. Coursera has a course by Scott Page on Models Thinking; Michael Kearns’s Networked Life is worth a look too!

Having used Vensim to do system dynamics, one of the most immediate things that I realise is that it’s not easy to think in clear terms just what is going on in the entire system that’s being studied. Thinking about a predator-prey relationship already assumes knowledge about a bunch of things, such as the lifespans of the preys and predators, their death rates, the predation rates, and so on.

Knowing the numbers is often insufficient to construct a model. In fact the numbers that are needed might never be quantifiable – in constructing various models, one has to inject various assumptions especially in describing the relationships between one variable and another.

With modelling and simulations, one has to think about how assumptions translates into quantitative terms. This isn’t as obvious as it sounds. I often find that our assumptions are sometimes entire systems packaged in a few words. Think about a statement such as, “excessive gambling destroys families.” At face value, this statement seems true, and few people would find difficulty rejecting that. But what does “excessive gambling” mean? Are there studies about the prevalent rate of gambling? And what does “destroys families” mean? We are usually talking about a diverse of adverse consequences, ranging from loss of incomes, divorces, incurring debts and bankruptcies. We are also often referring to the psychological costs that come with the conflicts that often lead to the results mentioned. If even for simple cases the causal mechanisms are difficult to establish, imagine the work undertaken to understand larger systems. Often, there is little choice but to introduce factors on some assumptions, and to refine them as the model is developed. “All models are wrong, but some are useful.” – statistician George Box

These models should be thought of as sandboxes – a container in which to let our assumptions play out. Without the ability to run controlled experiments in social affairs, models and simulations offer a decent way to play with our own mental models of reality.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s