Entries (RSS)  |  Comments (RSS)

Predicting the End of the World with Mathematica

Frequently you want to predict when things are going to happen, and if it’s not the end of the world, it might be something occurring a bit sooner, such as your disk filling up.

First capture some data, with cron. We’re going to capture the free space in our database, once a day, so we’ll put something like this in cron, and set it for every night at midnight:

0 0 * * * ls -l /var/log/somefile | tail -1 >> /tmp/somefile_log

Wait a few days. We were looking at daily file growth, so we waited a full week to collect data. We had the luxury at the time.

Now you’ll have a file with a series of ls entries in it. Run those through awk, and capture the sizes of the files.

cat /tmp/somefile_log | awk '{ print $5 }'

At this point, it’s time to fire up Mathematica. Mathematica is a stunning piece of software by Wolfram Research, used for data visualization, scientific work, and in a number of industries.

First, let’s load the data into Mathematica.

What we’re going to do now is to copy the 1st data point into the Fit, and create a function that will allow us to predict the future.

(* Fit data to a curve using a polynominal model, make sure you insert the 1st data point or the curve fit will be bad *)
result =  Fit[data, {336660004864, x, x^2}, x]

(* current free space on our partition (312334824), refunding the 1st data point as that \
space is already alloc'd *)
diskfree = (312334824 * 1024) + Take[data, 1]

(* use the fit function to find doomsday *)
diskfreefunc =  diskfree - result

(* when x = 0 , we are dead *)
deathday =  NSolve[diskfreefunc == 0, x]
deathday = Take[x /. %, -1]
deathday = deathday[[1]]
DatePlus[datastartdate, N[deathday]]

So now we know when this data set will hit zero, we have the date of that failure, and an ability to graph when that will happen.

For this example, my data set turns into:

We now know that in ~64 days, we’ll run out of disk space. Prediction is pretty nice, huh?

For you stats types, you’ll want to know how good the fit is for this curve, and for that, we look at R2.

This entry was posted on Thursday, June 25th, 2009 at 12:39 am and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

  • http://www.perfdynamics.com/ Neil Gunther

    An op-eng who slings MMA. Too totally cool. :D

    If you like that, you should love this (eventually). “WolframAlpha Performance Degradation” <http://perfdynamics.blogspot.com/2009/05/wolfra…> The model I used is briefly outlined here <http://www.perfdynamics.com/Manifesto/gcaprules…>

    Hi John, We chatted briefly @ Velocity. I did an Ignite talk <http://perfdynamics.blogspot.com/2009/06/ignite…>. Your Velocity talk was great and pretty much worth the price of admission for me. We'll chat again soon, hopefully.
    –njg

  • http://kitchensoap.com/ john allspaw

    Nice work, John. Can haz automation with the Mathematica, a la cfityk?

  • John Adams

    John- been trying to figure out how to automate cfityk; I'lll take a look at that this evening.

  • http://kitchensoap.com/ john allspaw

    Oh, I know how to automate cfityk, it's page 78 of my book :)
    http://bit.ly/4YWX6

    I meant, can you automate Mathematica? :)

  • http://www.perfdynamics.com/ Neil Gunther

    An op-eng who slings MMA. Too totally cool. :D

    If you like that, you should love this (eventually). “WolframAlpha Performance Degradation” <http://perfdynamics.blogspot.com/2009/05/wolfra…> The model I used is briefly outlined here <http://www.perfdynamics.com/Manifesto/gcaprules…>

    Hi John, We chatted briefly @ Velocity. I did an Ignite talk <http://perfdynamics.blogspot.com/2009/06/ignite…>. Your Velocity talk was great and pretty much worth the price of admission for me. We'll chat again soon, hopefully.
    –njg

  • http://kitchensoap.com/ john allspaw

    Nice work, John. Can haz automation with the Mathematica, a la cfityk?

  • John Adams

    John- been trying to figure out how to automate cfityk; I'lll take a look at that this evening.

  • http://kitchensoap.com/ john allspaw

    Oh, I know how to automate cfityk, it's page 78 of my book :)
    http://bit.ly/4YWX6

    I meant, can you automate Mathematica? :)

blog comments powered by Disqus