Allan Packer on performance tuning: Scaling MySQL

A year ago this past week (on January 16, 2008), Sun Microsystems announced a historic deal to acquire MySQL AB for a billion dollars. Since then, MySQL-ers (aka Sun Dolphins) have settled in with Sun folks. While Sun has had an influence on MySQL, MySQL has also had an influence on several projects within "Sun Classic". Over this week, we hope to explore some of the projects around Sun which use MySQL and how the acquisition has helped bring us all closer together.

Today we talk to Allan Packer, Principal Engineer in the Performance Technologies group at Sun Microsystems and team lead for the MySQL Performance & Scalability Project.

Hi Allan, thank you for answering our questions, it's appreciated! Could you please tell us a bit about yourself? What is your personal background, where are you located?

Allan: I'm married with 4 grown-up kids (two still in college), and I've been working in the IT industry for over 30 years. Almost 20 of those years have been with Sun, and I guess that says better than anything how much I've enjoyed working here. I've lived and worked on three continents, but right now I'm working from Australia, which is where I grew up and where my family is. That means most of my work is with virtual teams; fortunately Sun offers excellent support for people who work remotely.

For the bulk of my time at Sun I've worked in performance engineering, mostly with a focus on databases. So I've worked with Oracle, Sybase, and DB2, including being responsible for publishing a bunch of competitive benchmarks on different platforms.The proprietary databases are quite mature now and they've pretty much been commodities for years, so it's been fun to get involved with MySQL, which has quite a different history and profile.

Yes, we try hard to be different in many ways :) Please tell us, what project you are currently involved in and what is your role there?

Allan: Right now I'm the team lead for the MySQL Performance and Scalability project. Sun kick-started the project almost as soon as the MySQL acquisition was underway. Our goal is to get MySQL to scale to 64 hardware threads, which is actually only one Niagara chip these days, and to boost raw performance along the way. The project is staffed by both MySQL and Sun performance engineers.

Yes, unfortunately even though MySQL makes heavy use of threads, there are some scalability problems, especially on systems with multiple cores. It's great to hear that these problems are being addressed. I assume that most of your work is focused on Sun systems like the Niagara CPU — can you tell us how other architectures will benefit from this work?

Allan: While we are using Niagara-based test systems, the majority of our performance rigs are Intel- or AMD-based. That reflects MySQL's main user base, of course. And new chips from Intel and AMD are increasing in core count, so it isn't only SPARC that presents us with fairly large SMP systems.

What were your initial thoughts when you first heard of the MySQL acquisition by Sun last year?

Allan: I was taken by surprise, but delighted. The big challenge for Sun was to retain MySQL's employees and preserve its essential culture, while complementing MySQL in areas where it was weak. I felt that if Sun was able to pull off the integration successfully (by no means an easy task!), there was a good chance the acquisition could turn out to be a substantial win-win for both companies. And I was especially relieved that MySQL hadn't been snapped up by Oracle, Microsoft, or IBM, any one of which could easily have afforded to buy it.

How is MySQL involved in or related to your project?

Allan: There's been great synergy between our group and MySQL engineering. MySQL was always a bit challenged when it came to assembling large amounts of hardware for performance work, so we were able to make a big difference there. And it turned out they didn't have a lot of people focussed solely on performance, so they really appreciated having some extra resources on the job. For our part, we've never had such ready access to both the source and the development groups, so that's made it a lot easier to get answers when we're trying to figure out what's going on under the covers.

How has the MySQL acquisition influenced your project? What has changed?

Allan: The project didn't exist before the MySQL acquisition, so that has meant a change of focus for some of us.

Where do you see the use/support of MySQL in your project going?

Allan: There's been a tremendous amount of interest in MySQL within Sun's engineering organization. A lot of people are looking for ways of leveraging it, so I expect in time we'll see it put to use in new and creative ways. I hope Sun will be able to add some value back to MySQL as well.

I noticed that you and my colleague Mikael Ronström will give a talk about How Sun and MySQL are Driving MySQL Performance and Scalability at our upcoming MySQL Conference & Expo. Can you tell us some more about this session? What will you talk about?

Allan: Our talk will explain the steps Sun and MySQL have been taking to improve the performance and scalability of the MySQL Server on Intel, AMD, and SPARC systems running Linux and OpenSolaris. We examined both the scalability and performance on each of these platforms and we will provide examples of performance fixes and results. In addition to that, the presentation will talk about the MySQL Performance Release, a special prerelease build of the MySQL Server that includes a number of performance improvements.

This sounds like a very interesting session — we're excited about it and look forward to your talk! Please tell us: what do you look forward to most with the future of your project?

Allan: Seeing MySQL scaling better on systems with large CPU counts (which means just about any system these days, given the trends in multi-core CPU chip technology). And maybe eventually seeing MySQL starting to challenge the performance of proprietary databases across a broad range of platforms.

It certainly won't be an easy task, but it's excellent to hear that we're making progress in this area! Thank you very much for sharing your thoughts, Allan. See you in Santa Clara soon!

This interview was performed in January, 2009.