Interview with Ian Wilkes from Linden Lab

Interview with Ian Wilkes, Director of Operations at Linden Lab

Linden Lab is the company behind the virtual world simulation Second Life (SL) in which users (the residents) can freely roam around and actively participate in the creation and development of the virtual world and their own appearance (their avatars). To facilitate this, Second Life provides powerful in-world 3D construction tools as well as a special scripting language, LSL. Second Life also provides its own economy and currency (Linden Dollars), that residents can use to purchase goods and objects that other residents created. Interestingly, it’s even possible to exchange Linden Dollars into US dollars and vice versa. Ian Wilkes (aka “Ian Linden” in Second Life) is the Director of Operations at Linden Lab. I met with him at the virtual meeting room of Linden Lab inside of Second Life to have a chat with him about the technology of Second Life and how Linden Lab utilizes MySQL and other Open Source components to maintain the simulation infrastructure. Ian will speak about this topic in more detail in his session My Second Life Runs on MySQL: War Stories from the Metaverse at the MySQL Users Conference 2006 in Santa Clara, CA.

Lenz: Hi Ian, good to meet you. Thanks for taking the time to answer my questions.

Ian: No problem! I haven’t been to this meeting room before. The guys on the pictures behind us are “Primitars”, the first humanoid avatars we did. Strange to see them now.

Lenz: So that’s from the very early days?

Ian: Yes, around 2001. I think we replaced them with actual people at the beginning of 2002 – that was before the alpha release of SL. It’s been a long road.

Second Life Primitar

Lenz: First of all, could you tell us a bit more about yourself and your background?

Ian: Well, I’m the Director of Operations here now, of course. But my background is really in software development – I started here in 2001 as an engineer with the primary goal of fixing our infrastructure. I knew how to use databases, few other Lindens did at the time. The company was a little over 10 people then. Before this, I was a developer in the .com world – web stuff, e-commerce. I started at the ground floor of a company, we went public and went back down to zero. It was fun!

Lenz: So this must be something totally different from web – what motivated you to join Linden Lab at this point?

Ian: Not really that different, but more interesting. We still have usernames, passwords, and account balances. But I have a lot of contact with the game industry as well – I’ve done game work before, and a lot of my friends are in the business. So coming to Linden, which looked like a game company, seemed like a natural fit. Now of course we say SL isn’t a game, but it wasn’t always so clear and of course we still have a lot in common with that industry.

Lenz: Yes, I hear that some residents feel quite offended if you call SL a “game”. How would you briefly describe Second Life from your perspective to new users? Why should they consider it over the other MMOGs (Massively multiplayer online games) currently available?

Ian: Second Life is really a big city. MMOGs aren’t cities, they’re Disneyland: they have a defined experience.

Lenz: So this is more “real” in a sense?

Ian: I think so, yes. Which also, probably, makes it harder to digest – if I go to New York, I need to take my travel guides to know where to go and what to do. SL is the same in that respect.

Lenz: I personally was fascinated by the endless possibilities of creating your own world. Building and scripting got me hooked.

Ian: Yes, that stuff is fun! Creating things in an environment with other people was always a key goal for us. All this new economy, land, organization stuff came later – although one could argue that it’s inevitable.

Lenz: And I think the collaboration and building possibilites in SL are very nice.

Ian: Yes, we’ve noticed that recently some professional 3-d modeling applications have added collaboration functionality now :)

Lenz: How many people are currently actively involved in maintaining and developing SL?

Ian: We have almost 70 full time employees and around twenty part-time liaisons.

Lenz: That’s quite an impressive team. How many of these do infrastructure work vs. client development?

Ian: It varies – our engineers are never tied to one part of the system. Of the 70 FTEs, about 22 are programmers working on SL itself. At any one time probably 1/3 of the team is on infrastructure, 1/3 is on new features and 1/3 is on various maintenance tasks (bug fixes, general stability and speed improvements) or improvements to existing features. But it varies a lot.

Lenz: Speaking of features: are there any future plans about SL that you can reveal? What’s the biggest new thing you are working on? I hear that people are eagerly waiting for the move to SL using the Havok 2 engine – is there any news on that you would like to share?

Ian: I think we’re going to jump straight to Havok 3 and simply skip Havok 2 – we’re a bit behind on that one… But I don’t think that’s the most exciting thing we’re working on. We’ve been working for almost a year on integrating Mono for our script execution, it’s hard to predict how that will affect things, but it’ll be big, of that I’m sure. Also, the graphics guys have some amazing plans.

Lenz: You mean Mono as a replacement of the current scripting language, or as an alternative? That’s surely sounds like a big step.

Ian: As a replacement. But, we’ll keep the current engine around to run old content – I just don’t think we’ll allow new scripts to be compiled for it. But, I mean, from a scripter’s perspective it shouldn’t be that big a deal as we’ll still support LSL - we’ll just compile it to the CLI (Common Language Infrastructure). So, essentially, you’ll be able to compile scripts that run a lot faster and have more memory available. And we’ll have more flexibility for adding new scripting features.

Lenz: I personally am a Linux enthusiast. One the few remaining cases I ever have to reboot into Windows is playing SL - can you comment on the progress of the Linux client port?

Ian: We’ve actually just recently released an alpha version of the Linux client! There are still a few rough edges in that version, but once these have been ironed out we will do a public beta that everyone can try, just like we did with the Mac version. The people who’re using the alpha seem pretty happy with it already.

Lenz: I am very glad to hear that! What is the typical number of residents that are logged in in parallel at the same time? Is there a noticeable peak time?

Ian: You can see that on our website, we typically peak at about 5000-5500 every day. It drops into the 2000 range in the early morning (pacific time). Sunday afternoon is the biggest sustained peak, but it gets just as high during weeknights, just not for as long. Saturday is actually sort of a low day.

Lenz: How many PCs/Sims does SL currently run on? Do you follow a Google-like approach when it comes to hardware, or is each sim an entity of its own? What about distributed facilities?

Ian: We try to be like Google, but, you know, we don’t have their resources and they haven’t released their toolset. So SL is about, let’s see… 1100 servers right now, with about 2000 CPUs.

Lenz: Wow, an impressive number already!

Ian: Well, we use a lot of CPU time. So, most of that is sims, the CPUs running the regions. They’re basically interchangeable, with certain limits.

Lenz: Speaking of hardware: do you make use of 64bit CPU systems yet?

Ian: Yes at this point we actually have more 64-bit than 32-bit systems. We moved our databases to 16GB machines several months ago, with great results.

Lenz: So the large address space is a benefit for you?

Ian: Yes, huge! It’s definitely the most cost-effective upgrade available, at least if you don’t want to go above 8GB/CPU. Also, for unrelated reasons, we moved to 64-bit on many of the sims which was mostly about getting NUMA support. We’ve been buying 100% AMD for a while now as we’re constrained by power utilization – Intel’s server chips just suck down too much juice… But clearly they’re working on that.

Lenz: Yes, all these systems must consume huge amounts of energy :)

Ian: I think we’re at about 250kW right now. WoW (World of Warcraft) can host a similar number of users with a much smaller amount of hardware. In that analysis, SL looks very inefficient.

Lenz: Any idea why this is the case?

Ian: Yes, we’re running LSL scripts for people. And, more than that, when you buy an entire sim or land in one, you expect that there’s a CPU backing that sim, not half a CPU. And not, only when someone is actually there or only when there’s many people there. So, for the sake of efficiency, we may start to do that sort of thing, but we have to be careful how we do it. So SL currently seems to waste a lot of cycles but it’s not generally possible to determine WHICH cycles we wasted and which we needed ;)

Ian and Lenz chatting in Second Life

Lenz: I see, an interesting challenge. Does each sim run it’s own MySQL instance? Or do you have central MySQL database servers?

Ian: Central – the sims have very little state outside of their object data set, which they maintain internally and then copy back to a central data store.

Lenz: Can you elaborate on what kind of data is stored in MySQL?

Ian: In MySQL, we have all the user accounts: who you are, what you’re wearing, what you have in your inventory, etc – your transaction history. Also, presence information goes in there, which includes what sims are running on what hosts, various things we need to know to keep the grid running.

Lenz: What version of MySQL do you run on? Is there any feature that made MySQL especially suitable for the task?

Ian: Well, you probably won’t like this, but we’re still on 4.0, although our production upgrade to 4.1 is scheduled for March 1st. The fact is, we didn’t choose MySQL for advanced features, we want to store as much data as possible, as quickly as possible, with a minimum of fuss. This is what drew us to MySQL to begin with, and what has kept us there.

Lenz: Which is of course an area where MySQL shines.

Ian: Right, it’s really that simple.

Lenz: Do you consider using MySQL Cluster or do you plan on making use of some MySQL 5 features like views, triggers or stored procedures?

Ian: We looked at MySQL Cluster a while ago, but ultimately we don’t need it. We can distribute our big data loads (like inventory) perfectly well using logical partitioning and that really cuts down on the maintenance overhead. As for MySQL 5 – there’s some cool stuff in there. We’re trying to get our data warehouse upgraded ASAP.

Lenz: The new partitioning support in MySQL 5.1 might suit you well then! What amounts of data are we talking about here? Could you give a rough figure of how much data you store in MySQL?

Ian: The central DB has about 200M rows in it, the logging DB has about 300M and each of the user databases, of which there are currently 3, has about 50M rows.

Lenz: What other key OSS components do you utilize for providing the infrastructure SL is based on?

Ian: Linux of course: Debian. And let’s not forget GCC - nobody talks about GCC but everyone needs it! But we use LAMP for a lot of stuff, plus CVS, Bonsai, SystemImager, cvsup, nagios, ganglia, subversion, perl, python, ruby/rails, cyrus, postfix, mozilla – the list goes on.

Lenz: Is there anything about MySQL you would like to comment on? Anything we can improve?

Ian: Sure, I have some grievances, although I am still using 4.0! The fact is, in our operation we say the Databases Are Not Special – this is an unusual view. Most organizations coddle their DBs, spend a lot of time and attention on them, and so they should. But, we’re looking at a future with 50, 100 instances of MySQL, all of which need to be at least somewhat reliable. So, deploying new ones needs to be really easy and fixing broken ones needs to be really easy. And it’s the second point which is problematic for MySQL, in our experience.
Consider this: we have two replicant slaves for each master database. If the master fails, fine, we switch over to one of the slaves – that all works pretty well. But when we repair the master, now what has to happen? Re-snapshot the DB, change around the my.cnf, restart replication – it’s not that hard, but it takes a long time and is hard to automate.
I don’t know how many other people think this is a problem, but I think it will get more common as people move to the many-database solution, which MySQL itself advocates.

Lenz: I see what you mean – this is good feedback. I am not a seasoned MySQL DBA, so I might be missing an easy way to do this, but I will ask around for feedback or plans on how to improve this. MySQL Cluster might actually be a solution here!

Ian: It could be. The last time we examined it, there were limitations that made it problematic for us, I’d have to look at it again.

Lenz: Right. Cluster has come a long way.

Ian: The other suggestion is smaller: having to edit my.cnf and restart MySQL in order to change most of the configuration parameters is not cool. But again, I’m talking about 4.0 here.

Lenz: Yes, we try to change more and more options to be runtime-configurable, and 4.1 has already improved greatly in that respect. Wrapping up this interview with a less technical question: do you have a favourite place in SL that you would like to mention?

Ian: I don’t spend a lot of time logged into SL, but I think my favorite place is a sim called Welsh – it probably doesn’t look that special, but it hasn’t changed in… years. It’s one of the first sims we brought up in Beta.

Lenz: So you can’t do your daily work in here yet? That should be fixed :)

Ian: Yes, we’re working on it, but that sort of thing is very speculative – it’s going to take a long time for SL to be that flexible. And, with all the growth we’re seeing, we do have to put a lot of time into scalability work, paying for past sins, as it were.

Lenz: Well, if we look at how far SL has come since the early days you mentioned before… I am very excited to see what’s coming up from you guys in the future.

Ian: Well, it’s not all us, right? That’s the whole point – SL looks a lot better now largely because of what the residents have done and how they’ve learned to use the system in increasingly powerful ways. Our job at Linden is simply to remove limitations on what they can do.

Lenz: Are you sometimes amazed about what people manage to do with what you provide to them?

Ian: I think I used to be, yes. These days, I expect to be surprised, if that makes any sense.

Lenz: Yes, it does :) Ian, it has been a pleasure! I appreciate the time, thanks a lot for this interview.

Ian: Alright, thanks!

This interview was performed on February 22nd, 2006.