Interview with Jan Kneschke, Developer at MySQL AB

Jan Kneschke works for the MySQL Development Team and is the author of the very popular lighttpd web server. I recently had the opportunity to ask him a few questions about himself and his work. Q: Hi Jan, thanks for taking the time to answer my questions. Please give us a little background about yourself and what you did before you joined MySQL - where else did you work and what is your educational background? I currently live in Kiel, in the far north of Germany. Before I joined MySQL I worked for NetUSE, a local Sun/iPlanet/Roxen/... consultancy company. I joined a team which was deeply involved in the early days of the web. After I finished my studies in Electrical Engineering I became a free-lancer as consultant and trainer for PHP and Web in general.

Q: When and how did you join the company and what is your current role at MySQL?

I joined MySQL in 2004 when they were looking for a German speaking trainer and consultant. As I was active in the PHP community I knew Georg Richter, the developer of the php-mysqli extension, who was already working for MySQL. He brought in many PHP-community members at that time. After a short weekend-trip to Uppsala meeting my boss I got the job. I think this was one of the few job-interviews that was actually done in a normal office, which is quite unusual for MySQL.

Q: Can you elaborate on what you are currently working on?

I'm a senior developer in the MySQL Enterprise Tools team and am driving the development of the agent for the MySQL Network and Advisory Service (aka MaAS). Part of this is looking into new developments around managing MySQL instances remotely and how to get better performance data out of the MySQL server.

Q: You recently attended the MySQL Conference & Expo in Santa Clara. What presentations did you give and how was the feedback you received?

As every year I gave a tutorial, this year about MaAS and how to enhance it. MaAS has a rule-engine which allows the users to write their own rules and the agent can be enhanced to collect all kinds of data (MySQL, CPU, Disk, ...).

Q: Which talks that you attended there impressed you the most?

Most impressive were the key-notes by Guy Kawasaki and the sessions by YouTube, Flickr and Yahoo. Sadly I couldn't attend too many other sessions.

Q: Tell us some more about lighty. When did you start working on the project and what motivated you to start writing a web server from scratch?

I started it as distraction from writing my thesis. First you start cleaning the house, the windows, help the grandma over the street, ... and then when nothing else is left to do, you write a web-server.

Writing a final thesis is a long, creative process and I needed way to concentrate on something completely different. I stumbled over the c10k-problem or "how to write a server which can handle 10.000 parallel connections". After writing a proof of concept it was several times faster as Apache and still faster than thttpd, the dominant alternative web-server of that time. Adding fastcgi support gave me a full replacement for apache for my own needs.

The ChangeLog still has the comments on the early days of lighttpd and the benchmarks for each small release.

Q: What do you consider the most attractive features of lighty?

We solve several Web 2.0 problems by default:

• keep-alive requests are "cheap" (AJAX prefers it)
• mod-sec-download allows temporary, authenticated URLs to prevent hot-linking
• X-Sendfile moves the burden of sending authenticated files from the web-app to the web-server.
• FastCGI (PHP), SCGI (Python), HTTP (rails) and AJP13 (Java) are supported with load-balancing and fail-over, making scaling simple.

Q: Why should users consider it over Apache to power their web site?

We use a lot less resources to get our job done.

Currently we are working on getting lighttpd 1.5.0 out to the public. It changes several internals and adds features like supporting accurate upload progress-bars and gives more hooks to manipulate the response. mod-magnet is providing similar features as mod-perl does for Apache as a generic request-manipulation language.

Q: What distinguishes lighty from the other "lightweight" web servers that are available?

We have several generations of web-servers now:

• the big ones (Apache, IIS, iPlanet, ...) from the early days.
• the small ones (thttpd, boa, mathopd, ...) as early alternatives against the big ones which don't provide FastCGI or similar support
• the new, small ones like lighttpd and nginx which provide FastCGI, Proxy, ... support

All the small web-servers are using non-blocking I/O and an event-scheduler in a single process to scale to many parallel connections. lighttpd (1.5.0 and later) adds threads for the File-I/O to scale with the number of disks and uses a lua as internal scripting language to manipulate the way requests are processed.

Q: Lighty seems to have a strong user base in the Ruby on Rails developer community. Are there other areas where it is popular?

As we are providing SCGI support for the Python community and FastCGI for the PHP, Perl and C world we are getting more and more users from those communities.

Q: Can you name some well-known web sites that utilize lighty? What purpose does it usually serve there?

We are used by Wikipedia for the upload.wikimedia.org server which is handling all image requests on wikipedia and scales the images on demand.

YouTube.com is using a modified version of lighttpd to send out thumbnails and videos from the server-farm in San Jose.

Sourceforge.net is handling the main-site with lighttpd. My last information is that they are using lighttpd as front-end and several PHP servers load-balanced via mod-fastcgi as backends.

Most of the major torrent-sites are using lighttpd as they have a large user-base and problems to handle the load with Apache. Resource usage was a major problem for them.

Q: What are your future plans with lighty? Any other OSS projects you are currently involved with?

The major rework of the internals of lighty 1.5.0 are done and we are working on stabilizing it to have a big major release again. It will bring several improvements for the future of lighttpd:

• better scaling for static file transfers
• better fail-over, load-balancing for all protocols
• AJP13 support
• pluggable auth-backends and SQL-vhosting

For the far future there are ideas around implementing the Bayeux protocol used by cometd. It will open a whole new class of possibilities for future web-applications.

Rumors say that I also work on a mysql proxy ;)

Thank you very much for answering my questions, Jan! All the best and keep up the good work.

This interview was performed in May 2007