Statically Typed

because Hindley-Milner rocks

On Bottlenecks and Performance

It strikes me as odd just how many people love to prematurely optimize or optimize without benchmarking.  Moreover, it’s unequivocally disastrous when the choice to optimize is made by non-programmers who have no concept of how to improve performance.  This is true even when engineers present detailed research giving guidance on said bottlenecks.

I have previously battled large performance issues.  In one particular case the bottleneck was I/O.  It took roughly 18 minutes just to read 1-5 million rows of information (third party API) but only 2 minutes to process them.  Writing them out, on the other hand took hours (days even.)  Despite this, the only solutions proposed, without consulting me, the guy writing the software, revolved around either increasing the amount of meta information or “reducing” all information through additional I/O operations.   Both types of solutions optimized the 2 min. calculation times and not the real performance bottleneck.

This is an issue of communication between customers and providers, management and developers, scientists and engineers.  People will tend to focus on those areas where they are most comfortable even if that area isn’t the underlying problem.  It’s only natural.  The trick is to get them to think of the impacts of their proposed solution on the larger goal.  If I/O is the cause, more of it won’t fix things.

It gets me thinking if sometimes I focus on the wrong parts first too.  I look at the solutions for the Wide Finder challenge as a guide.  I knew that a “close to the metal” type solution would be king, C or C++ taking center stage.  I had also assumed that the next in line would be OCaml which for 2 years occupied the other slots.  What I wasn’t expecting was that Scala could take #3 in the rankings.  It’s not a commanding margin between 3rd and 4th place (4 seconds) but it does demonstrate that finding the bottlenecks sometimes isn’t as easy as it first appears.

I wonder just how much the leading Python 2.4 solution could be improved using the same type of solution or even how close I could come to that time with Scala.  Perhaps I could use Akka‘s transactor Actor/STM combo to test out things.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


This entry was posted on February 27, 2010 by in C++, Pet Project, Scala.
%d bloggers like this: