gSkinner - Home

PerformanceTest v2 Beta: Benchmark AS3 / Graphics

Posted on February 3, 2010 by Grant Skinner

When I built the original PerformanceTest, I hacked it together quickly to meet a specific need I had at the time. As I used it, I found I needed more capability, and I became increasingly frustrated with both the messy code, and the poorly thought out API.

In response to feedback, and my own frustrations, I’ve completely rebuilt PerformanceTest to be more robust, extensible, feature-rich, *and* easier to use.

Core features:


  • Track time, memory usage and retained (unreleased) memory for functions.
  • Test rendering time for any display object.
  • Write simple one line tests, or build formal test suites.
  • Test suites can be built using inheritance or aggregation/composition.
  • Trace simple text results, or access robust XML reports.
  • Save XML and compare results between different versions of your code, or on different Flash players.
  • Easily build logging classes to analyze or visualize results.
  • Run multiple iterations of tests to get min, max, and deviation values.
  • Run tests synchronously, or queue them to run them asynchronously.
  • Listen for events from tests, test suites, and PerformanceTest.
  • Integrates easily with TDD by facilitating the creation of tests for time, memory, and retained memory limits.

Here’s a very simple demo:


In it’s simplest form, you can use PerformanceTest in one line:

trace(PerformanceTest.run(myMethod));
trace(PerformanceTest.run(mySprite, "spriteRenderTest", iterations, loops));

You can go even simpler when testing methods, by using the ptest function shortcut:

trace(ptest(myMethod));
trace(ptest(myMethod, ["param"], "name", iterations, loops));

Want something a bit more robust? You can set up formal collections of tests using TestSuite. You can use these to set up suites of benchmarks to test different player platforms, or you can maintain them over the life of a project to track its performance profile by saving the XML Logs.

This is a beta. The docs are complete, but a little sparse. There are very few sample / demo files. It almost definitely has bugs. I also plan to add at least one more class/tool to it, for comparing XML logs.

Given that, I’d appreciate anyone who’s interested checking it out and giving some feedback. Does it do what you need? Find any bugs? Is the API straightforward? How much more documentation do you need?

You can download PerformanceTest v2 here. As with pretty much all of the source code I release, it is licensed under the MIT license.


UPDATE: I just uploaded a new version with a working FLA. My apologies for the trouble with the original one. It also includes a *very* simple benchmark comparison tool.

Follow @gskinner on Twitter for more news and views on interactive media.
26 Comments

Thanks, Grant, this looks awesome. Especially if you don't use the FlexBuilder.

Posted by: bigfish on Feb 3, 2010 1:00pm URL: http://dafishinsea.com

Hmm. Made me curious. So you use something like BitmapData.draw to test sprite drawing performance? Or you do it differently?

Posted by: wonderwhy-er on Feb 3, 2010 1:30pm URL: http://wonderwhy-er.deviantart.com

Right, it uses BitmapData.draw(). This gives a pretty good idea of what the render time for a display object is on a single core. Not an exact value, but great for comparative metrics.

Posted by: Grant Skinner on Feb 3, 2010 1:34pm URL: http://gskinner.com/blog/

I saw your tweet, and if you decide not to publish your thoughts on the future of Flash you will be my hero for about a week.

I value your opinion above most others that I've heard, but it would be a pretty pimp move not to beat this topic any further.

Posted by: cgortz on Feb 3, 2010 1:38pm URL: http://www.chrisgortz.com

cgortz - sorry to disappoint. :)

Posted by: Grant Skinner on Feb 3, 2010 5:11pm URL: http://gskinner.com/blog/

This is just in time for a project I'm working on. I used v1 quite a lot and found it very useful, so this is going to be even better.

For some reason I'm finding the FLA in the zip is failing to open in Flash CS4. Everything else unzipped fine.

Posted by: Ben Beaumont on Feb 4, 2010 7:15am

So Powerful

Posted by: xiehuc on Feb 7, 2010 9:08pm

Same as Ben, the FLA is not opening in Flash CS4.

Posted by: Tronster on Feb 12, 2010 9:11am URL: http://tronster.com

Thankyou Grant! I was about to write my own really simple one when I found this, which is far better and really easy to use.

For those who can't open the flash file, it contains a dynamic text field with the instance name of 'outFld' and the document root class of PerformanceTestDemo

Posted by: Carly on Feb 13, 2010 12:07am URL: http://koali.com.au

Just wanted to say that I just found PerformanceTest tool set (20 minutes ago) and it is both very simple to use and invaluable. Thank you for releasing it!

Posted by: Alex on Feb 16, 2010 10:01pm URL: http://blog.nodicerequired.com

I tried doing some memory usage checks for my unit tests they way you have, but it seemed like I kept getting false positive. Something as simple as:

var sprite : Sprite;

var(i : int = 0; i

Posted by: Brian on Feb 22, 2010 10:00am

Im sorry to inform you that the Fla fails to open i both CS3 and CS4 on PC ??

Posted by: Henrik on Feb 24, 2010 10:26am

In the PerformanceTest class there are a static method and protected member both named "queue", maybe we need different names?

Posted by: eidiot on Feb 28, 2010 8:09pm URL: http://eidiot.net

Hey Grant,

nice classes. But I have a question to you. How can I calculate the minimum system requirements(cpu,ram etc.) for his own flash movie. Could be the perfomancetest class a help for that?? I have already googled the whole day - no solution found!!

Thanks

Posted by: radykal on Mar 15, 2010 5:21pm URL: http://www.radykal.de

Great Work Here... I Finally Able to optimize my development and also my FYP research... LOL... Thanks a lot... ^ ^

Posted by: Wilson on Mar 28, 2010 12:30pm

I was linked to this page by my friend who was struggling to make use of this tool, and after some time of me trying to get some sense of the results it produces I just came to the conclusion, that I don't know what it does. Sorry to say this, but it's based on a whole bunch of misconceptions about how the code is translated to the machine codes, or, sometimes, the testing methods themselves are so poorly written that the mistake that they generate is bigger than the actual time you want to measure...

Well, just for the record, I've tested some function, of which I was certain that it had to spend about 120-160ms on processing ~600Kb of data, and using this tool I was getting 400+-10ms... This is off by 150%... what's the point?

Posted by: wvxvw on May 28, 2010 11:07am

Hello,

PerformaceTest v2 looks awesome, thanks for sharing :) But I just found something that possibly invalidates all speed tests.

If you "play" a series of identical loops, the later ones are always fairly quicker than the previous ones. This happens both with while than with for loops.

I tried it by myself (with custom fors and whiles, targeting FlashPlayer 10 and also FlashPlayer 10.1 in both Debug and Release versions), and also using your TestPerformance.

With TestPerformance, I tried to run a simple method which contained only a "var n:int = 2" operation. I ran PerformanceTest.run multiple times (with a 1000000 loops value) with this same method, and it's evident that the first results are quite slower than the others.

Any idea why this happens?

Posted by: Daniele on Oct 4, 2010 4:42am URL: http://www.holoville.com

Hi Daniele,

Yes, this is due to a couple of factors.

The first is that the player has a ramp up period when it first initializes. This slows everything down for the first few seconds.

The second is that some code may not be JITed the first time through (especially if it is run immediately), but subsequent runs may be.

For this reason I recommend having tests run 3s or more after your test SWF initializes (easy to do with a Timer), and running each test multiple times (which PerformanceTest supports directly). This is also why PerformanceTest provides metrics like min, max and deviation values, so you can evaluate the consistency of your results.

Posted by: Grant Skinner on Oct 4, 2010 4:53am URL: http://gskinner.com/blog/

Hi Grant :)

Yup, I now tried using an iteration value (in PerformanceTest.run) bigger than 0, and the deviation values are very helpful :)

Though, I have a button on my stage which calls the performance test (to avoid the player ramp up period), and I ran it many times. I'm quite puzzled by the fact that, each time I ran the same test, the first loops are always slower. This surely has a striking impact when using loops in actual code, and means that actually there's also a frame ramp up period.

Posted by: Daniele on Oct 4, 2010 6:25am URL: http://www.holoville.com

Using this to try and create unit tests for a client.

And quite honesty I don't really understand the output, or how to use the iterations / loops parameters and what effect they really have on the test.

I was asked to run the RenderTest on some particles, and I just find your documentation incredibly lacking. Like is the displayobject supposed to be added to the stage or not?

And an explanation of how the render test, all of the tests really, actually works, and what is it testing for, would greatly help to understand its output.

Right now I feel like I wasted a few hours setting up a .fla, writting classes, and getting everything to work, for just a bunch of junk output.

-Robert Baindourov

Posted by: Robert Baindourov on Feb 10, 2011 6:34pm URL: http://www.winwinhost.com

And I just double checked the example I downloaded, and the output you are generating here on this page, and you don't even run a render test in either. A functional example would have been nice.

Posted by: Robert Baindourov on Feb 10, 2011 6:50pm URL: http://www.winwinhost.com

it is so great .i am excited to find this useful tool.
and i used it in my code .
But now i find a problem, The tool just can test the function without parameters. So the tool can not be used anywhere.
can you fixed it ?
thank you .

Posted by: newtamato on May 5, 2011 1:49am

Hey Grant, I just felt like commenting based on some of the previous responses (newbs). I just really want to thank you for building this test harness, I love it when someone out there saves me some programming time, and this certainly did, and will continue to save me tons of time when there's a doubt or an argument about which direction to take with regards to optimization.

Posted by: Sam Batista on Jul 10, 2011 2:33am URL: http://sambatista.com

Thanks so much! this will aid great learnings

Posted by: Sumeet Basak on Jan 9, 2013 5:13am

Thanks for great lib!
I find myself using and using it!

Posted by: Deril on Jun 24, 2014 1:39pm URL: http://mvcexpress.org/

Any possibility of it put on gitHub?

Posted by: Deril on Jun 24, 2014 1:54pm URL: http://mvcexpress.org/

Leave a Reply

Your email is never published nor shared.




You may use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>