Interview with Paul McCullagh, developer of the PrimeBase XT Storage Engine

Interview with Paul McCullagh, developer of the PrimeBase XT Storage Engine

With MySQL 5.1, MySQL AB has introduced a new pluggable storage engine architecture that allows storage engines to be loaded into and unloaded from a running MySQL server. The PrimeBase XT Storage Engine (PBXT) is a new transactional database engine for MySQL. It has been designed for modern, web-based, high concurrency environments. In May 2006, I had the opportunity to meet with Paul McCullagh, the head developer of PBXT for lunch at the Hamburg offices of Snap Innovation GmbH, the company behind PrimeBase. It was co-founded by Paul in February 1996 and is focussed on the development and sales of client/server database technology, specializing in inter- and intranet systems.

Q: Paul, thanks for taking the time to meet with me and answering my questions! You mentioned you attended the MySQL Users Conference in Santa Clara in April. How did you like it and what was your most favourite moment?

Well, my favourite moment was definitely when Mårten Mickos mentioned PBXT in his keynote. And if anybody read my blog then they would know that this wasn't planned at all. We just met by chance a few moments before we went into the hall and he quoted from our discussion. Considering there were over 1600 attendants at the conference, I consider myself very fortunate.

But besides this, there was a lot at the conference of great interest to me. I was kept very busy because of the many sessions that were relevant to the development of storage engines. In particular, the sessions on optimizing MySQL have given me a bunch of ideas.

And not least of all, I have benefited very much from the personal discussions. I had great fun meeting all the storage engine gurus, including Monty, Brian, Heikki and others. In fact, I have already built into PBXT an idea that Jim Starkey gave me over a beer one evening! I'll bet Jim will wonder what idea that could be. I gave him full credit in my code so he'll have to read it to find out. Anyway, thanks Jim!

Q: I am glad to hear that you had a good time over there. This really sounded like an excellent opportunity and good timing! Would you mind telling us a bit more about your personal background?

Well, after living in Germany for so long now, many people are surprised to hear that I am a South African. I was born in Cape Town and grew up in a little town called George in the middle of an east coast area called the Garden Route. I studied Mathematics and Computer Science at the University of Cape Town and qualified with a Master of Science.

During this time, I met a wonderful german girl and, after we both finished studying, we married and moved to Germany. Initially, we planned to only stay for a few years but that was 16 years ago now. And now we will be here for quite a bit longer with our church, the business and 3 children to consider.

Q: Well, I personally would have had a hard time to decide between George and Hamburg - both are very nice places! But please tell us more about the PBXT storage engine. What is the history of it and what makes it unique in comparison to other storage engines?

This is something that few people are aware of, but probably I have not made it all that clear. PBXT differs from other transactional storage engines in that it was designed and built specifically for MySQL. So PBXT has no real history to speak of, other than the ideas that have gone into it, which I have been working on for a number of years.

This can be seen as a disadvantage, but also as an advantage. A disadvantage because it is not battle tested like the other storage engines. An advantage because it will be far easier for others in the community to understand and contribute to. I think this will really make the difference for PBXT.

Otherwise, PBXT is a multi-versioning transactional storage engine, much like Falcon and to a lesser extent InnoDB. What makes PBXT different is its design. I have outlined my ideas in a White Paper, but to put it in a nutshell: PBXT is a "write-once" transactional system. While standard transactional systems write all data twice, PBXT writes data only once.

As a result, I believe PBXT has the potential of being a very fast transactional engine.

Q: Having read through the White Paper, I must agree that it really sounds promising! How long did it take you to create the initial release? Did you encounter any difficulties, and what was your motivation to do it in the first place?

Well, Mårten in his keynote mentioned that PBXT took 6 months to develop. He was calculating from the beginning of the main development phase of PBXT, which started last November, when I downloaded the 4.1.16 build of MySQL.

But this would suggest the implementation of PBXT was motivated by Oracle buying Innobase. This is not the case. In fact, I started working on the design of PBXT and a "proof of concept" implementation at the beginning of last year, basically in my spare time. I based this development on my experience of working as a database systems programmer for many years. In this time, I have learnt a lot about how to do things, and, maybe even more importantly, how not to do things.

At some point, my partners and I had to make a decision as to what to do with these ideas. There were a number of factors to consider because it is quite a big investment for a small company. And, of course, if you do something like this you have to commit to it properly because sooner or later people are going to rely on that.

The deciding factor was something which I call "the principle of the community". We believe that what we invest will come back to us in some form or another. It's a biblical principle actually of "reaping and sowing", and that's why it works.

Concerning the difficulties of building an engine for MySQL, I would say things have gotten a lot easier in 5.1. I used 4.1 and there you have to do quite a bit of code reading to figure out what needs to be done. It's not really difficult, but it takes some time, and the transaction stuff really has to be hacked in.

5.1 has a clean API which is pretty much finalized and documented. Which reminds me, I promised Mike I would review his documentation when I get around to moving PBXT to 5.1.

Q: He would probably appreciate that - your experiences will be valuable feedback for other people that want to explore building their own storage engine. How is the community feedback about PBXT so far? Did you get good responses?

The response from MySQL and the community has been overwhelming. Maybe I have been in a closed source environment to long, but I really didn't expect this. I have made a lot of new friends. The list would be too long, but I would like in particular to mention Ronald Bradford, who was the first to offer me help and introduced me to people at MySQL.

I like to take marathon running as an example. Think of the difference between training for a marathon and running a race. The closed source industry is like training for a marathon. You are basically on your own. The open source community is like running a race. Not because you want to win. Most people don't run a marathon to win, they run to complete. But during the race you experience a comradeship and sense of doing something together that makes running much easier then in training.

Going by this comparison, I have been in training too long.

Q: A nice comparison, I assume many people at MySQL would agree with you here - we have a number of marathon and long-distance runners at our company. And I know that Mårten likes analogies like this one, too! What are your future plans with PBXT and MySQL? Can you tell us what's going to happen next?

This is a good question because the landscape has changed quite a bit since the Users Conference. The other engines are planning to go Beta in the 3rd quarter. I would really like to meet this deadline myself, so this means a time of very intense work for me. In fact, if things are suddenly a bit quiet from my end, it is not because I am not working on PBXT, but in fact quite the contrary.

Ultimately I would like PBXT to become known as the "community engine". This means that when I am happy with the basic implementation, a sort of foundation, I would like to actively search for people in the community to fill out the functionality.

For example, it is quite easy for someone to add a full-text index to PBXT. This is because the indexing system does not have to be transaction aware. The design makes a non-transactional indexing system behave in a transactional manner in the context of PBXT.

This inherent simplicity, together with good documentation, will be a great advantage in the search for other contributors.

Paul and Lenz at the Hamburg office of Snap Innovation GmbH

Q: PBXT has already been added to the Project directory on the MySQL Forge. Do you plan to add more information about it to the MySQL Forge Wiki? Is there anything you would like to see in MySQL Forge that would make it easier for you to continue working on the PBXT storage engine?

I believe that I share the vision with MySQL for the MySQL forge. It should be a place where I can host my entire project, source code and all. In order to realize the goal of creating a community engine, it is necessary to communicate and to exchange details of the implementation. The forge Wiki is the perfect place for this. All aspects of the implementation must eventually be documented to make it possible for a large number of people to contribute to the code.

But we also need to coordinate development resources and manage bug tracking, fixing, testing and release cycles. I know I am thinking way ahead here. These things will not happen overnight. I'm guessing but I think this is the direction in which MySQL is setting its sights. Personally, I think it could revolutionize the way software is developed, and I would like to be a part of it.

Paul, thank you very much for your time and answering my questions. We look forward to how PBXT will evolve and wish you all the best!

This interview was performed in May, 2006.