Category Archives: Models

Are all models wrong?

Artem Kaznatcheev considers whether Box’s slogan, “all models are wrong,” should be framed as an empirical question.

Building on the theme of no unnecessary assumptions about the world, @BlackBrane suggested … a position I had not considered before … for entertaining the possibility of a mathematical universe:

[Box's slogan is] an affirmative statement about Nature that might in fact not be true. Who’s to say that at the end of the day, Nature might not correspond exactly to some mathematical structure? I think the claim is sometimes guilty of exactly what it tries to oppose, namely unjustifiable claims to absolute truth.

I suspect that we won’t learn the answer, at least in my lifetime.

In a sense, the appropriate answer is “who cares?” Whether or not there can in principle be perfect models, the real problem is finding ones that are useful in practice. The slogan isn’t helpful for this. (NIPCC authors seem utterly clueless as well.)

In a related post, AK identifies a 3-part typology of models that suggests a basis for guidance:

  • “Insilications – In physics, we are used to mathematical models that correspond closely to reality. All of the unknown or system dependent parameters are related to things we can measure, and the model is then used to compute dynamics, and predict the future value of these parameters. …
  • Heuristics – … When George Box wrote that “all models are wrong, but some are useful”, I think this is the type of models he was talking about. It is standard to lie, cheat, and steal when you build these sort of models. The assumptions need not be empirically testable (or even remotely true, at times), and statistics and calculations can be used to varying degree of accuracy or rigor. … A theorist builds up a collection of such models (or fables) that they can use as theoretical case studies, and a way to express their ideas. It also allows for a way to turn verbal theories into more formal ones that can be tested for basic consistency. …
  • Abstractions – … These are the models that are most common in mathematics and theoretical computer science. They have some overlap with analytic heuristics, except are done more rigorously and not with the goal of collecting a bouquet of useful analogies or case studies, but of general statements. An abstraction is a model that is set up so that given any valid instantiation of its premises, the conclusions necessarily follow. …”

The social sciences are solidly in the heuristics realm, while a lot of science is in the insilication category. The difficulty is knowing where the boundary lies. Actually, I think it’s a continuum, not a categorical. One can get some hint by looking at the problem context for models. For example:

Known state variables? Reality Checks (conservation laws, etc.)? Data per concept? Structural information from more granular observations or models? Experiments? Computation?
Physics yes lots lots yes yes often easy
Climate yes some some for many things not at scale limited
Economics no some some – flaky microfoundations often lacking or unused not at scale limited

(Ironically, I’m implying a model here, which is probably wrong, but hopefully useful.)

A lot of our most interesting problems are currently at the heuristics end of the spectrum. Some may migrate toward better model performance, and others probably won’t – particularly models of decision processes that willfully ignore models.

Facebook Reloaded 2013

Facebook has climbed out of its 2012 doldrums to a market cap of $115 billion today. So, I’ve updated my user tracking and valuation model, just for kicks.

As in my last update, user growth continues to modestly exceed the original estimates. The user “carrying capacity” now is about 1.35 billion users, vs. .95 originally (K950 on graph) and 1.07 in 2012 – within the range of scenarios I originally ran, but well above the “best guess”. My guess is that the model will continue to underpredict for a while, because this is an inevitable pitfall of using a single diffusion process to represent what is surely the aggregate of several processes – stationary vs. mobile, different regions and demographics, etc. Of course, in the long run, users could also go down, which the basic logistic model can’t represent.

You can see what’s going on if you plot growth against users -the right tail doesn’t go to 0 as fast as the logistic assumes:

User growth probably isn’t a huge component of valuation, because these are modest differences on a percentage basis. Marginal users may be less valuable as well.

With revenue per user at a constant $7/user/year, and 30% margins, and the current best-guess model, FB is now worth $35 billion. What does it take to get to the ballpark of current market capitalization? Here’s one way:

  • The carrying capacity ceiling for users continues to grow to 2 billion, and
  • revenue per user rises to $25/user/year

This preserves some optimistic base case assumptions,

  • The risk-free interest rate takes 5 more years to rise substantially above 0 to a (still low) long term rate of 3%
  • Margins stay at 30% as in 2009-2011 (vs. 18% y.t.d.)

Think it’ll happen?

facebook 3 update 2.vpm

Random rein control

An interesting article in PLOS one explores the consequences of a system of random feedbacks:

The Emergence of Environmental Homeostasis in Complex Ecosystems

The Earth, with its core-driven magnetic field, convective mantle, mobile lid tectonics, oceans of liquid water, dynamic climate and abundant life is arguably the most complex system in the known universe. This system has exhibited stability in the sense of, bar a number of notable exceptions, surface temperature remaining within the bounds required for liquid water and so a significant biosphere. Explanations for this range from anthropic principles in which the Earth was essentially lucky, to homeostatic Gaia in which the abiotic and biotic components of the Earth system self-organise into homeostatic states that are robust to a wide range of external perturbations. Here we present results from a conceptual model that demonstrates the emergence of homeostasis as a consequence of the feedback loop operating between life and its environment. Formulating the model in terms of Gaussian processes allows the development of novel computational methods in order to provide solutions. We find that the stability of this system will typically increase then remain constant with an increase in biological diversity and that the number of attractors within the phase space exponentially increases with the number of environmental variables while the probability of the system being in an attractor that lies within prescribed boundaries decreases approximately linearly. We argue that the cybernetic concept of rein control provides insights into how this model system, and potentially any system that is comprised of biological to environmental feedback loops, self-organises into homeostatic states.

To get a handle on how this works, I replicated the model (see my library).

The basic mechanism of the model is rein control, in which multiple unidirectional forces on a system act together to yield bidirectional feedback control. By analogy, the reins on a horse can only pull in one direction, but with a pair of reins, it’s possible to turn both left and right.

In the model, there’s a large random array of reins, consisting of biotic feedbacks that occur near a particular system state. In the simple one-dimensional case, when you add a bunch of these up, you get a 1D vector field that looks like this:

If this looks familiar, there’s a reason. What’s happening along the E dimension is a lot like what happens along the time dimension in pink noise: at any given point, the sum of a lot of random impulses yield a wiggly net response, with a characteristic scale yielded by the time constant (pink noise) or niche width of biotic components (rein control).

What this yields is an alternating series of unstable (tipping) points and stable equilibria. When the system is perturbed by some external force, the disturbance shifts the aggregate response, as below. Generally, a few stable points may disappear, but the large features of the landscape are preserved, so the system resists the disturbance.

With a higher-dimensional environmental state, this creates convoluted basins of attraction:

This leads to a variety of conclusions about ecological stability, for which I encourage you to have a look at the full paper. It’s interesting to ponder the applicability and implications of this conceptual model for social systems.

Positive feedback drives email list meltdown

I’m on an obscure email list for a statistical downscaling model. I think I’ve gotten about 10 messages in the last two years. But today, that changed.

List traffic (data in red).

Around 7 am, there were a couple of innocuous, topical messages. That prompted someone who’d evidently long forgotten about the list to send an “unsubscribe me” message to the whole list. (Why people can’t figure out that such missives are both ineffective and poor list etiquette is beyond me.) That unleashed a latent vicious cycle: monkey-see, monkey-do produced a few more “unsub” messages. Soon the traffic level became obnoxious, spawning more and more ineffectual unsubs. Then, the brakes kicked in, as more sensible users appealed to people to quit replying to the whole list. Those messages were largely lost in the sea of useless unsubs, and contributed to the overall impression that things were out of control.

People got testy:

I will reply to all to make my point.

Has it occurred to any of you idiots to just reply to Xxxx Xxxx rather than hitting reply to all. Come on already, this is not rocket science here. One person made the mistake and then you all continue to repeat it.

By about 11, the fire was slowing, evidently having run out of fuel (list ignoramuses), and someone probably shut it down by noon – but not before at least a hundred unsubs had flown by.

Just for kicks, I counted the messages and put together a rough-cut Vensim model of this little boom-bust cycle:

unsub.mdl unsub.vpm

This is essentially the same structure as the Bass Diffusion model, with a few refinements. I think I didn’t quite capture the unsubscriber behavior. Here, I assume that would-be unsubscribers, who think they’ve left the list but haven’t, at least quit sending messages. In reality, they didn’t – in blissful ignorance of what was going on, several sent multiple requests to be unsubscribed. I didn’t explicitly represent the braking effect (if any) of corrective comments. Also, the time constants for corrections and unsubscriptions could probably be separated. But it has the basics – a positive feedback loop driving growth in messages, and a negative feedback loop putting an end to the growth. Anyway, have fun with it.

Computing and networks have solved a lot of problems, like making logistics pipelines visible, but they’ve created as many new ones. The need for models to improve intuition and manage new problems is as great as ever.

Spot the health care smokescreen

A Tea Party presentation on health care making the rounds in Montana claims that life expectancy is a smoke screen, and it’s death rates we should be looking at. The implication is that we shouldn’t envy Japan’s longer life expectancy, because the US has lower death rates, indicating superior performance of our health care system.

Which metric really makes the most sense from a systems perspective?

Here’s a simple, 2nd order model of life and death:

From the structure, you can immediately observe something important: life expectancy is a function only of parameters, while the death rate also includes the system states. In other words, life expectancy reflects the expected life trajectory of a person, given structure and parameters, while the aggregate death rate weights parameters (cohort death rates) by the system state (the distribution of population between old and young).

In the long run, the two metrics tell you the same thing, because the system comes into equilibrium such that the death rate is the inverse of the life expectancy. But people live a long time, so it might take decades or even centuries to achieve that equilibrium. In the meantime, the death rate can take on any value between the death rates of the young and old cohorts, which is not really helpful for understanding what a new person can expect out of life.

So, to the extent that health care performance is visible in the system trajectory at all, and not confounded by lifestyle choices, life expectancy is the metric that tells you about performance, and the aggregate death rate is the smokescreen.

Here’s the model: LifeExpectancyDeathRate.mdl or LifeExpectancyDeathRate.vpm

It’s initialized in equilibrium. You can explore disequilbrium situations by varying the initial population distribution (Init Young People & Init Old People), or testing step changes in the death rates.

What a real breakthrough might look like

It’s possible that a techno fix will stave off global limits indefinitely, in a Star Trek future scenario. I think it’s a bad idea to rely on it, because there’s no backup plan.

But it’s equally naive to think that we can return to some kind of low-tech golden age. There are too many people to feed and house, and those bygone eras look pretty ugly when you peer under the mask.

But this is a false dichotomy.

Some techno/growth enthusiasts talk about sustainability as if it consisted entirely of atavistic agrarian aspirations. But what a lot of sustainability advocates are after, myself included, is a high-tech future that operates within certain material limits (planetary boundaries, if you will) before those limits enforce themselves in nastier ways. That’s not really too hard to imagine; we already have a high tech economy that operates within limits like the laws of motion and gravity. Gravity takes care of itself, because it’s instantaneous. Stock pollutants and resources don’t, because consequences are remote in time and space from actions; hence the need for coordination. Continue reading

Et tu, Groupon?

Is Groupon overvalued too? Modeling Groupon actually proved a bit more challenging than my last post on Facebook.

Again, I followed in the footsteps of Cauwels & Sornette, starting with the SEC filing data they used, with an update via google. C&S fit a logistic to Groupon’s cumulative repeat sales. That’s actually the end of a cascade of participation metrics, all of which show logistic growth:

The variable of greatest interest with respect to revenue is Groupons sold. But the others also play a role in determining costs – it takes money to acquire and retain customers. Also, there are actually two populations growing logistically – users and merchants. Growth is presumably a function of the interaction between these two populations. The attractiveness of Groupon to customers depends on having good deals on offer, and the attractiveness to merchants depends on having a large customer pool.

I decided to start with the customer side. The customer supply chain looks something like this:

Subscribers data includes all three stocks, cumulative customers is the right two, and cumulative repeat customers is just the rightmost.

Continue reading

Time to short some social network stocks?

I don’t want to wallow too long in metaphors, so here’s something with a few equations.

A recent arXiv paper by Peter Cauwels and Didier Sornette examines market projections for Facebook and Groupon, and concludes that they’re wildly overvalued.

We present a novel methodology to determine the fundamental value of firms in the social-networking sector based on two ingredients: (i) revenues and profits are inherently linked to its user basis through a direct channel that has no equivalent in other sectors; (ii) the growth of the number of users can be calibrated with standard logistic growth models and allows for reliable extrapolations of the size of the business at long time horizons. We illustrate the methodology with a detailed analysis of facebook, one of the biggest of the social-media giants. There is a clear signature of a change of regime that occurred in 2010 on the growth of the number of users, from a pure exponential behavior (a paradigm for unlimited growth) to a logistic function with asymptotic plateau (a paradigm for growth in competition). [...] According to our methodology, this would imply that facebook would need to increase its profit per user before the IPO by a factor of 3 to 6 in the base case scenario, 2.5 to 5 in the high growth scenario and 1.5 to 3 in the extreme growth scenario in order to meet the current, widespread, high expectations. [...]

I’d argue that the basic approach, fitting a logistic to the customer base growth trajectory and multiplying by expected revenue per customer, is actually pretty ancient by modeling standards. (Most system dynamicists will be familiar with corporate growth models based on the mathematically-equivalent Bass diffusion model, for example.) So the surprise for me here is not the method, but that forecasters aren’t using it.

Looking around at some forecasts, it’s hard to say what forecasters are actually doing. There’s lots of handwaving and blather about multipliers, and little revelation of actual assumptions (unlike the paper). It appears to me that a lot of forecasters are counting on big growth in revenue per user, and not really thinking deeply about the user population at all.

To satisfy my curiosity, I grabbed the data out of Cauwels & Sornette, updated it with the latest user count and revenue projection, and repeated the logistic model analysis. A few observations:

I used a generalized logistic, which has one more parameter, capturing possible nonlinearity in the decline of the growth rate of users with increasing saturation of the market. Here’s the core model:

Continue reading

Models and metaphors

My last post about metaphors ruffled a few feathers. I was a bit surprised, because I thought it was pretty obvious that metaphors, like models, have their limits.

The title was just a riff on the old George Box quote, “all models are wrong, some are useful.” People LOVE to throw that around. I once attended an annoying meeting where one person said it at least half a dozen times in the space of two hours. I heard it in three separate sessions at STIA (which was fine).

I get nervous when I hear, in close succession, about the limits of formal mathematical models and the glorious attributes of metaphors. Sure, a metaphor (using the term loosely, to include similes and analogies) can be an efficient vehicle for conveying meaning, and might lend itself to serving as an icon in some kind of visualization. But there are several possible failure modes:

  • The mapping of the metaphor from its literal domain to the concept of interest may be faulty (a bathtub vs. a true exponential decay process).
  • The point of the mapping may be missed. (If I compare my organization to the Three Little Pigs, does that mean I’ve built a house of brick, or that there are a lot of wolves out there, or we’re pigs, or … ?)
  • Listeners may get the point, but draw unintended policy conclusions. (Do black swans mean I’m not responsible for disasters, or that I should have been more prepared for outliers?)

These are not all that different from problems with models, which shouldn’t really come as a surprise, because a model is just a special kind of metaphor – a mapping from an abstract domain (a set of equations) to a situation of interest – and neither a model nor a metaphor is the real system.

Models and other metaphors have distinct strengths and weaknesses though. Metaphors are efficient, cheap, and speak to people in natural language. They can nicely combine system structure and behavior. But that comes at a price of ambiguity. A formal model is unambiguous, and therefore easy to test, but potentially expensive to build and difficult to share with people who don’t speak math. The specificity of a model is powerful, but also opens up opportunities for completely missing the point (e.g., building a great model of the physics of a situation when the crux of the problem is actually emotional).

I’m particularly interested in models for their unique ability to generate reliable predictions about behavior from structure and to facilitate comparison with data (using the term broadly, to include more than just the tiny subset of reality that’s available in time series). For example, if I argue that the number of facebook accounts grows logistically, according to dx/dt=r*x*(k-x) for a certain r, k and x(0), we can agree on exactly what that means. Even better, we can estimate r and k from data, and then check later to verify that the model was correct. Try that with “all the world’s a stage.”

If you only have metaphors, you have to be content with not solving a certain class of problems. Consider climate change. I say it’s a bathtub, you say it’s a Random Walk Down Wall Street. To some extent, each is true, and each is false. But there’s simply no way to establish which processes dominate accumulation of heat and endogenous variability, or to predict the outcome of an experiment like doubling CO2, by verbal or visual analogy. It’s essential to introduce some math and data.

Models alone won’t solve our problems either, because they don’t speak to enough people, and we don’t have models for the full range of human concerns. However, I’d argue that we’re already drowning in metaphors, including useless ones (like “the war on [insert favorite topic]“), and in dire need of models and model literacy to tackle our thornier problems.