Choosing a future MySQL career path

In the past few months I’ve not had to look for a job, they look for me. If your a well qualified MySQL DBA there are many opportunities, I’ve had so many of these offers in the past 12 months I’ve lost count. However a DBA job would not offer the job satisfaction I’m seeking in my career goals now in 2008.

I’ve been able to consider 4 genuine positions in just the past few weeks, and indeed my new employer Primebase Technologies was someone I’d never considered working for, and indeed they had never considered me until a chance lunch meeting less then one month ago.

I did seriously consider working with Paul Vallee and the team at Pythian. Pythian looks for the best people from their respective levels of technology expertise, whether a college graduate or a 2 decade RDBMS veteran. This company has a great work ethic and a strong commitment to a 40 hour work week. With offices in multiple countries and a growing business, and a prominent client list as seen with the Pythian Web site even reviewing this company before talking to anybody provides a lot of good information.

I’ve followed Pythian closely since good friend MySQL She-BA Sheeri Kritzer Cabral was considering job opportunities last year. I was very impressed during this process. During recent vacation I took the time to visit Pythian, to speak personally with a number of people, from the CEO to potential fellow colleagues, to see the work environment, to have lunch and after work drinks. I invested more time in interviewing this potential company then they did in vis-versa, a principle I’d recommend to anybody for any future positions anywhere.

Today I read in How to become CEO, “Hire the best people. attract, motivate, train and reward the best people“. A good principle and one I know is at the heart of this company. Pythian is a company and a partnership I’ll be ensuring in the future remains strong. With a team of 60+ DBA’s across MySQL, Oracle and SQL Server they provide a dedicated service that MySQL (prior to acquisition) simply could not provide for market demand.

Comments on this site reflect the personal opinion of the author and may not reflect the opinion of any present or past employers.

Goals, desires and pursuit of the ideal working environment

I’ve had a lot of feedback from my resignation at MySQL. A lot positive, a lot of shock, even some avid discussion on why, and trying for me to reconsider my decision. Roland my evil twin (it’s a funny story), really challenging me which I very much appreciated, ensuring I had considered the multiple paths before me. I am a man of my own convictions, so reversing my decision was simply not an option on principle.

So what do I want in an ideal working environment. What was lacking, that I could not see and that I’m seeking. Here are two key points.

  • “Continual improvement, 1% a day, review in 3 months.” This includes a commitment from the entire team, and a system to contribute, discuss, plan and see results. Something I’m a most firm believer of and will be ensuring is in place in my next position where I will have the responsibility.
  • “Your employees are your most valuable customers”. This goes without saying. If you don’t treat your employees as well as your best customers they leave, often people however lower there expectations and time lesses people’s goals and desires.

One book I’m reading now on the opening introduction also sums my goals and desires well.

“Having ambition to better yourself, to be a contributor, to make a difference, to grow professionally, to be more successful.”How to Become CEO – Jeffrey Fox

This sums we up well right now in 2008.

It’s often fun to review what you said previously.

Pride – August 11, 2006.
Your Ideal Job Requirements? – August 27, 2006.
If you can’t beat them, join them! – October 7, 2006.

My passion for Open Source

I am a very strong proponent of Open Source (excluding my Macbook). Joining MySQL Inc was a wonderful achievement, being part of the team behind the most popular open source database. Leaving MySQL was not an easy decision due to the people, but the Sun transition and requirements did help. However it is no surprise I am joining another open source company – Primebase Technologies in Hamburg, Germany. My association with the MySQL Community will only be strengthened with my full work and support behind the PBXT and Blob Streaming pluggable storage engines for MySQL.

It is actually poetic that I am joing Primebase for I have the auspicious recognition while an active part of the MySQL community of introducing Paul McCullagh to MySQL some 2 years ago. Only weeks later, PBXT was MySQL news in the opening CEO keynote my Marten Mickos at the 2006 MySQL Users Conference.

I expect this opportunity will increase my contributions, in particular in the lacking areas of instrumentation and memory management. As well I will now be focusing my efforts on MySQL 5.1 and ensuring this is of the highest quality for the MySQL Community. While my open source code contributions have been small to date, my first some years ago adding in JMeter support for MySQL Stored procedures and Transactions, the feeling of seeding instead of leaching is a wonderful thing. I look forward to a greater opportunity with the team at Primebase Technologies.

Pursuing new ventures with MySQL

The acquisition of MySQL by Sun Microsystems has caused a certain amount of news in 2008 from the Initial announcement at the MySQL ACM, to the completion in just 6 weeks. It has been a very quick transition and while there is the potential for further opportunities with Sun as an employee within the MySQL product line and possibly other areas, I have elected to pursue my professional goals elsewhere.

My leaving MySQL Inc has been met with comments from “Woot!”, “Congratulations”, “Good Luck”, ‘That’s terrible”, “Are you sure”, “I’m shocked, but I don’t blame you” and attempts from multiple people to reconsider and reverse my decision. I was pleased to know that it took no time from my first contacting colleagues and friends within MySQL before the community knew via the grape vine, even before my attempts to contact people.

The most interesting comment was “That’s terrible news”, to which my response was, “Well, not for me”. My best compliment has been “You are a such a great team player and MySQL aficionado.”. I actually had to look aficionado up, which means “an ardent devotee; fan, enthusiast.” I am very proud that my commitment to the MySQL Product, and MySQL Community and while at MySQL Inc/Sun has been noted, if only a small footnote in the success of the company.

There are a number of reasons why I’m moving on, which I’ll not be detailing here. In summary however, first, I have already set my career goals for Ronald Bradford 2008 and this includes the type of work I’m wanting to pursue, the money, and an increase in writing, presenting and speaking on topics in MySQL and other technologies and interests. In particular, I’m working (slowly til now) on consolidating my online presence into an online identity at The second and deciding factor was as a result of the acquisition and transition period. The issues of, the offer and requirements to transfer to Sun played an important part in my decision, being disappointed and unhappy across various things in this process.

At this time I can say that I’ll be leaving the US by the end of March as part of my US visa requirement. I will however not be leaving the MySQL Community, in fact one of my goals is to be able to contribute to the MySQL Community more, as I did prior to joining MySQL Inc in 2006 and I’m finalizing details with a prominent European Open Source company, and I expect to have some news here soon when details can become confirmed. For my friends and colleagues attending the MySQL Conference in just 4 weeks time, I’ll be there, so this will a perfect chance to catch me while I’m in the US.

I am sorry to leave MySQL. I have made many close friends here, and indeed it’s the people that helps make a company and among my peers I have great respect. There is a unique spirit working with an open source company and as I had ties to MySQL prior to joining the company, I’m confident these will continue and grow as my involvement with MySQL continues, just differently and more in line with what I want to do.

Those of you that know me personally, any recommendations via my Linked In Profile at this time is most appreciated.

A day with Jan Kneschke – Architect of MySQL Proxy

Recently I got to spend a day with Jan Kneschke at his home in Kiel, Germany. This was an excellent opportunity to interact directly with a fellow MySQL employee and resident expert in several fields. Doing this on my own vacation time enabled me to control what I wanted to achieve.

Jan uses Linux, Windows and Mac OS/X all actively and with Synergy enables a single keyboard and mouse to work seamlessly across his three screens. I knew you could do this, just last year I tried to find out the software that could do it without obvious success.

Of interest was our discussion in English, while he was also communicating on IRC in German during the day. Being multi-lingual is something we don’t often consider and talking in one breath in English, then having to type in German is a little more complex then as I’m writing it here.

One of the discussions was the impact of the Sun acquisition, but I can’t say anything here as it’s all confidential. I don’t want the MySQL legal police after me. It’s business as usual here are MySQL. However we both discussed several concerns we have, a common topic among many MySQL Employees here at MySQL.

I spent some time browsing Jan’s library. Books browsed included “MySQL Database Design and Tuning” (Amazon link). (Why as a MySQL employee I haven’t see this, or been given a copy to review/read/access I don’t know. I had heard of it). “SQL Performance Tuning” (Amazon Link) by Peter Gulutzan and Trudy Pelzer who are now both MySQL employees, and “SQL for Smarties – Advanced SQL Programming” (Amazon Link) by Joe Celko.

Within the MySQL Database Design and Tuning book I came across SQL_MAX_JOIN_SIZE, a handy per connection session variable to limit the size of poorly joined SQL statements. Problem is it’s not part of any resource control restrictions so if you can’t control connections to the database by reporting end users, then what’s the real point. A nice idea that’s only partially implemented for real use.

Part of my time here, is to get answers. Some of my specific questions were around Lua (The scripting language used with MySQL Proxy). One particular question was the ability to process a SQL request and response programmatically at startup (to manage some global data), but the way to achieve this is to piggy-back off the first actual SQL request for the connection.

Another question was in regards to the testing of newer Lua scripts. The solution is to see the test cases which manage the Proxy via Mock up functions.

On Saturday, there was another change of aspect. Jan over the weekend was needing to do some security updates with lighttpd. “You need to still respond to requests from the community or they have the impression the product is dead. It’s a burden, you can’t really step out.” I completely understand as for almost 10 years now I’ve provided support and service to a number of not-for-profit organizations. My usage requirements are now very low, I don’t make any more software or website changes, just the occasional web server issue. (My present primary server uptime is up 560 days, and this was only to do an OS level upgrade from the past x100 days.)

The lighttpd project has up to 6 other committers that add work from time to time. For any successful Open Source project you need to ensure this occurs, otherwise it’s something you can never leave. Interestingly, Lighttpd is still a popular web server as per the Netcraft Survey Results.

Jan wakes up to the sound of the coffee machine in the morning, literally. It’s rather a loud and predicable process.

Not all work, we attended the Nubit 2008 event by NetUSE AG in the evening for networking, dinner and drinks. In his spare time on Saturday I caught him playing Hellgate.

Part of interacting with other professionals is you come across tools and sites they use to benefit their productivity or functionality, and this type of viral exposure helps greatly in your experiences and knowledge transfer.

And just in closing, two bizarre technology occurrences here (just before the full moon). The first, my computer become intelligent. I must have found a magic key combination in vim because it was doing interactive spelling correction while typing. It was even doing correcting of variables names. It was a little unnerving and even Jan was surprised (so it’s probably not just a German thing). The second, my Timex watch of at least 15 years (with many band and battery replacements) reset it’s self for the first time ever. I looked and it had 12:05am, Mon 1.1 – Weird.

Jan will also be speaking at the upcoming 2008 MySQL Conference, you can read his profile.

Stay tuned for Parts 2 and 3 with Michael Zinner – Architect of MySQL Workbench and Paul McCullaugh – Architect of the PBXT Storage Engine .

Today's interesting MySQL Error Message

You have to love error messages some times. Today in MySQL is was “ERROR 1289 (HY000): The ‘UNKNOWN’ feature is disabled;”

What was even more interesting is the error message and indeed the error number changed across different versions. I took the time to also review the error in the current versions of 5.0, 5.1 and 6.0.

5.0.37/5.0.51a – ERROR 1289 (HY000): The ‘InnoDB’ feature is disabled; you need MySQL built with ‘InnoDB’ to have it working
5.1.19/5.1.23/6.0.4 – ERROR 1286 (42000): Unknown table engine ‘InnoDB’
6.0.0 – ERROR 1289 (HY000): The ‘UNKNOWN’ feature is disabled; you need MySQL built with ‘UNKNOWN’ to have it working

You can read more at Bug #29373

Noise canceling headphones

Some 9 months ago I left my noise canceling headphones on a plane. Since then I’ve though I’d just self suffer for my carelessness, and I’d decided the next set I would buy would be Bose. The problem is, Bose make two brands, The over ear 2 model and the on ear 3 model. I could never decide which one to order without first trying. They do offer a return policy but that’s just more hassle.

So eventually I get to the Bose store in the Time Warner building at Columbus Circle, try them both, can’t really decide either way and settle for the over ear 2. Some facts in my decision is basically the case is the same size, the 2 model has a recharger, while the 3 uses AAA batteries.

I then head of the the NY apple store for some smaller purchases, and I notice that all the sample headphones for the iPods are Bose. You can even buy them there. So treking across NY in the rain could have been easier?

In other Apple news, the MacBook Air is not as light as I would have thought, being 3 pounds, still it’s a cool geek thing and I was tempted even though I’d just just purchased my own Macbook recently. And of course they released a new macbook yesterday as well. Go figure.

MySQL graphics and words in use

Not sure during my vacation which part of the creative brain has taken over, but I’ve become rather obsessed with marketing graphics and associated words being used across the MySQL and Sun MySQL websites, (See previous examples here and here).

Here are a number of more interesting references from the front page of the site.

  • The MySQL website now has a MySQL/Sun logo, noting it starts with the MySQL logo first.
  • MySQL & SUN Come together. Freedom & Innovation Fast, innovative, open database solution now with world class service and support (dolphin jumping at sunset)
  • Unlimited Possibilities. Deploy an unlimited number of MySQL Enterprise Servers for the cost of a single CPU of Oracle Enterprise Edition (two dolphin jumping across a moonlight night scape)

The haven’t got to the MySQL Conference website yet, I guess that’s controlled by O’Reilly still for now.

Words and images remain the property of Sun Microsystems, Inc. Copyright 1994-2008.
Comments on this site reflect the personal opinion of the author and may not reflect the opinion of any present or past employers.

Where is the Sun MySQL Reference Manual?

Yesterday I mentioned the new The official Sun-MySQL WebSite. It interested me with the navigation, graphics and content used to describe MySQL.

Greg of One Free Voice in a comment raised a very valid question, he could not find the MySQL Reference Manual, see comments. (I should also point out Greg it is no shame to reference the MySQL manual even daily, I’m an expert in the field and I easily reference the manual multiple times a week, and for reference the single most important page for me is Option and Variable Reference. I’ve also forgotten when using multiple languages in MySQL what is OFF/ON, simple solution is in the mysql client go SELECT ON; and see it it’s 1 or 0.)

Well, I didn’t find a link to the MySQL reference manual on the Sun Website. I’m sure it’s there somewhere but this leads to the question of design. The MySQL got an overhaul several months ago, and the navigation was clearly improved down to two clear menus, see below. Now the Sun MySQL page has 4 separate menus (even the forth I missed yesterday), see below. This new menu even discovered another interesting graphic and comment to add to my list yesterday.

  • Community – 11 Million and growing (Picture the bottom have of marathon runners)

Words and images remain the property of Sun Microsystems, Inc. Copyright 1994-2008.
Comments on this site reflect the personal opinion of the author and may not reflect the opinion of any present or past employers.

The official Sun-MySQL WebSite

As already reported the deal is done (see Sun Press Release, Kaj – Ambassador to Sun comments). I’ve had a look at the Official Sun-MySQL website.

Here is a review of the interesting words and images on the new site.

  • Overview – Ultimate Scalability for the Web Economy (dolphin & sunset)
  • Features – It Just works (windsurfer with island view)
  • Tech Specs – Celebrate the Possibilities (skydiver on snowboard)
  • Perspectives – Open, Fast, and Free Just Got Better (snowboarders viewing the mountain scape)
  • Support – Unbeatable Duo – Open Source and Global Support (couple showing the V symbol)
  • Training – Get Trained. Get Ahead. (Cyclists riding into the sunset)
  • Get It – 60,000 downloads a day (surfers on a wave)

Well, they have very colorful images!

Words and images remain the property of Sun Microsystems, Inc. Copyright 1994-2008.
Comments on this site reflect the personal opinion of the author and may not reflect the opinion of any present or past employers.

MySQL Monitoring 101 – Graph your results

The problem

Hands up those that don’t monitor their production MySQL web server. I’m a little surprised by this, but I’ve visited several clients that have absolutely no monitoring other then “the customers will tell us when something is wrong”. The lack of system monitoring is one of the topics in my book “101 ways to screw up a successful startup”.

Why is monitoring important? First it can tell you when something is wrong, most monitoring systems introduce some level of constraints that trigger notifications via email, SMS or red flashing screens. Second, and I consider more important, is it allows you to analyze change and compare results over time. Let’s say you added more memory to your server, and then remembered to also increase the MySQL buffers appropriately. How much improvement did it make? Rather then “it seems faster”, you can have hard and fast numbers to back it up.

The Monitoring

If you have zero monitoring, you need to implement at least the following. Create the following script and run daily at midnight via cron. I’ve made the script as simple as can be.

DATETIME=`date +%Y%m%d.%H%M`
HOSTNAME=`hostname -s`

vmstat 5 17280 > $LOG_DIR/os.vmstat.5.$HOSTNAME.$DATETIME.log &
iostat -x 5 17280 > $LOG_DIR/os.iostat.5.$HOSTNAME.$DATETIME.log &
exit 0

If you don’t have iostat installed, you will find iostat/sar as part of the sysstat package that most distributions will have generally available.

The results

vmstat output of relative idle system.

$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0      0 304332 289484 435052    0    0     3     4  206  177  1  0 98  0
 0  0      0 304108 289496 435072    0    0     0    45  411  736  5  0 95  0
 0  0      0 304116 289500 435072    0    0     0    13  395  716  3  0 97  0

vmstat of system under disk load

$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0      0 234784 289844 516800    0    0     3     4  206  177  1  0 98  0
 1  1      0 101636 289988 645552    0    0     1 33190  448  626 35  5 31 29
 1  2      0  33092 280888 723276    0    0     4 17233  469  616 24  4  6 66
 1  0      0  62876 177444 797056    0    0     2 14846  837  938 33  5 31 30
 1  1      0  33252 168440 834064    0    0     1 30376  969  904 31  6 22 41

iostat of system under load

 iostat -x 5
Linux 2.6.22-14-generic (newyork)       02/16/2008

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.31    0.00    0.29    0.15    0.00   98.26

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.83     5.82    0.71    0.94    13.04    53.56    40.53     0.09   52.60   2.08   0.34
sr0               0.00     0.00    0.00    0.00     0.00     0.00    10.40     0.00  237.00 235.00   0.01

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          41.30    0.00    5.70   29.00    0.00   24.00

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00  5153.60    0.60   75.80     4.80 42774.40   559.94    81.57 1354.93  13.09 100.00
sr0               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          30.63    0.00    4.10   61.36    0.00    3.90

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00  3850.00    0.20   60.00     1.60 39763.20   660.54    83.68 1450.90  13.86  83.44
sr0               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          22.10    0.00    3.80   42.30    0.00   31.80

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00  7645.80    0.40   66.80     3.20 44009.60   654.95   100.34 1192.33  14.14  95.04
sr0               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

The analysis

The below graph shows CPU usage over a few hours. What was only observed via the graph was a spike of CPU that occurred every 5 minutes. In fact it occurred just on one server of several servers, and this enabled analysis to find a rouge cron job running every 5 minutes.

The below graph shows CPU idle usage over a 24 hour period. The colors represent different usage patterns such as during the day, batch processing overnight etc.

The below graph shows IO utilization over the same 24 hour period.

NOTE: Color’s added manually to highlight points.
These past two graphs highlight a number of things that require further analysis, but as in this example, it only shows part of a bigger picture. For example only CPU idle is graphed, not Wait I/O for example which as you can see from the I/O is excessive at times.

Still, these graphs were created from grep’ing the appropriate data into a CSV file, loading into Open Office, and click just the graph button, about a 60 second process for each type of data. This alone gave a better picture of what was happening then no picture.

What’s next

This level of monitoring is impractical for production systems, especially if you are monitoring more then one system. You need to monitoring more things, compare different results at the same time, and get better results at hand instantly. In my next topic “Monitoring 201″ I’ll talk about a number of easy and practical options.

Microsoft, Yahoo and Open Source

There has been plenty of press this week regarding Microsoft making a bid for Yahoo. This week the Wall Street Journal Article From Uncertain Future To Leading Yahoo Bid has prompted me to the following observations. I quote several points:

The bid, he said on the call, is “the next major milestone in Microsoft’s companywide transformation” to incorporate online services.

as Microsoft pushes the bid and, if successful, tries to meld Yahoo with Microsoft.

Microsoft had been negotiating to buy online ad company DoubleClick Inc. but lost that deal to Google, which paid $3.1 billion. Microsoft in May countered, spending $6 billion on online ad company aQuantive Inc.

While Microsoft should continue investing in its own online services, it needed to speed things up through acquisitions.

Once a company had a critical mass of buyers and sellers on its online-ad system, it could hold sway over much of the industry. In computers, Microsoft achieved that position with its Windows operating system. But on the Internet, Google was quickly taking on that role.

The Alexa Ratings has Yahoo as the number 1 real-estate property, outstripping Google. What’s important to realize that Yahoo along with many top traffic websites not only use Open Source, but their business is run on Open Source. At the database, there is MySQL powering Yahoo, Facebook, Wikipedia, YouTube, Fotolog and Flickr for example. Google also uses MySQL within critical components (not the search engine).

One can only hope that if such a bid is successful, much like the Sun acquisition of MySQL , that strong components of the Open Source ideal infects the much larger host.

I was thinking of taking this popular Tux & Microsoft Office image and badging Tux with a Yahoo logo, or perhaps he needs to be planting a big neon sign in the center.

Solid-State Drives Press

I read yesterday in The Wall Street Journal an interesting article in Personal Technology, “Solid-state drives challenge hard drives in speed, but not value”. While the title does sum up nothing new, the barrier to entry to embracing new technology is always cost, and early adopters may not have deeper pockets, but it’s also about being a alpha geek.

As the article states “Solid-state drives have some key advantages. Because that lack moving parts, they are faster, draw less power, and harder to damage and are quieter”. It also backed this up with some results.

What I did not realize, is the new MacBook Air has a 80GB HHD version at $1,799 and as 64G SSD version at $2,798. Testing showed that there was little difference in battery life between these models, however with the Toshiba Portege R500 and the right battery there was a 36% battery life increase. That’s interesting news for those that spend a lot of time on planes,trains and airports. Tests clearly showed faster cold startup and reboots, up to 40% faster.

Kevin Burton has had a lot to say recently about MySQL and SSD, and Matt Yonkovit has also just published some results with Mtron SSD Sysbench/MySQL results, DBT2 MySQL SSD Results and More Details on MySQL & SSD drive performance…. It seems that Mtron SSD was the brand reviewed by Matt and Kevin (See 24 Hours with an SSD and MySQL).

Where do we go from here, MySQL as with any database has certain characteristics, large amounts of storage needed, fastest access possible, but generally sequential and random writes, but random reads. Certain MySQL storage engines may benefit with sequential writes, such as PBXT. As with any architecture today, multiple layers, caching and different approaches to storing and retrieving data within your application or website will ultimately be required to maximum the strengths of underlying technologies and minimize the weaknesses.

More information at Wikipedia Solid State Drives.

MySQL & Sun

I have been noticeably absent on my comments in this topic, something I was pinged about yet again today by another colleague.
Did I have nothing to say? No. It’s just I’ve not been able to say much, or actually do anything in the past 7 days due to strong bought of the flu which has kept me in bed near 5 days straight (coincidently aligning perfectly with MySQL’s recent ACM. Go figure that luck).

So what are my thought’s here. (Ok, I’m going to waffle a bit with my point of view, but my key technical points for those of you that just want that, scroll down to the next section heading)

Overall I believe it’s a good thing, on the surface and at the moment. As mentioned MySQL was on an IPO path, I’d would have liked the option to buy my own MySQL shares, be part of a company that got to that point (have worked for 2 failed startup Internet companies previously). I think there would have been many a proud MySQL employee,ex-employee, and user that day, and that’s a shame for the pinnacle victory of Open Source, to become something of worth, to see the light at the end of the tunnel, but be tempted by the fruits enroute. MySQL, like every startup has investors, they have invested in MySQL for quite some time, I’d think probably up to 10 years now, they obviously want their money too.

But the market out there has many larger players, those less understanding of Open Source, more of the corporate giants with larger shares $15 Billion in RDBMS revenues, and willing to either run over, or kill off a smaller weaker competitor by $ size. There has been some press saying it’s a a bad thing, Sun is dead, what a waste of money etc. If Sun did not purchase MySQL I’m sure there was a line that would. It doesn’t take a rocket scientist to think of names such as IBM, Microsoft & Oracle. All competitors ironically. One mentioned that was interesting was Yahoo? Had not thought of that, but in that line of thought, would Google do good, or evil.

I’ve worked more closely with Sun equipment in 2007. Indeed my longest engagement and Californian Client recently mentioned in the press as co customer of Sun & MySQL was a very tough engagement stressing both Solaris 10 and MySQL 5 (with custom built binary). (1) Stressing me too very much, regardless of the New York to San Francisco each week for a decent time. I was exposed to Dtrace, I was exposed to zero skills in MySQL, well perhaps one, but he would not return phone calls or emails, that’s just plain unprofessional even to a colleague. I was exposed the warts of legal not allowing me to get Sun help on DTrace even when that was offered by sUN and was to the clear benefit of MySQL. Luckily cooler heads, weeks (more then we had) and a lot of pain prevailed on this said point, but to my point, what a darn unnecessary pain to start with.

I see more of Sun around. I know of other consulting clients (of course we can’t mention), other large sites of Community collegues, e.g. Fotolog are Sun, be it MySQL 4.x days however. I’ve heard the Sun/MySQL relationship is not that good, that MySQL doesn’t work as well on Sun as it could, that there are clear performance gains that can be made under the Solaris OS.

And historically, I stated working with SUN in the early 90s, it might be good to return after my time with SunOS before there was Solaris. I think Solaris started at 2.5.

(1) I mention custom built binary because many people don’t realize, there are actually three paths of getting features into MySQL. Both the Community and Enterprise path are subject to Engineering and the constraints of “Feature Freeze” for example. A Custom built binary, can include a specific feature or features into a supported version of MySQL just for you. MySQL Professional Services has dedicate staff that work in these areas. Our consulting offerings are listed here

My Key Points (Some personal)

  • Access to a benchmarking group, that’s gotta help. MySQL Inc the company has a number of shortcomings, and little movement to excite me, this is one of them. I’m sure Sun will want to provide MySQL this detail, and this can only be a key strength. (Sun people reading this, I want to help)
  • Access to H/W. Ok, so I can’t get any benchmark results out of MySQL, I can’t even get access to several consistent machines to test stuff myself anyway (as if I had that time), but if I’d got allocated a few days a month of dedicated access to a bank of services I’d sure use it. Especially in the area of RAID performance, LVM/SAN Backup & Recovery. (Again Sun people, pick me, I especially want just a dedicated server to test multiple performance tests with different RAID configurations, and MOST IMPORTANTLY, provide figures of how long to recover a degraded system under load. Want the same under a SAN, but that become more political.
  • Access to Dtrace resources. I see this tool as key in my next level of internal knowledge of the server. This will grant me better access to resources, Luckily my Macbook with 10.5 also has Dtrace, a recent new discovery, so that will make getting Solaris x86 Parallels VM a secondary priority. (Sun people listening, a Parallels/VMWare Solaris is on my wish list)
  • Access to facilities. I live in New York. While I have no desire to want to go into an office day by day, to know now I can, say have meeting a room, an area to work for a few days distracted etc, is golden, and a plus for a larger company.
  • Access to facilities for community events. This is gotta help or fledgling MySQL User Groups and MySQL Camp events. I really hope it can actually flow onto helping significantly in event organization taking the MySQL User Conference from it’s present limitations into bigger and better
  • Access to a better expense system. MySQL’s system sucks. Sorry, no other word for it. Scorn me later. I’m at a disadvantage over many other MySQLer’s, a) I’m one of the few that travel all the time, b) I’m of even less few that live and travel not in my country or origin and so I’m confined to the limitations of said countries financial systems. Specifically I own property in Australia, have 20K CC, can’t get a CC to cover a month’s expeneses here, actually can’t get them to cover a week (already have a few). I can’t use my Australian ones effectively here, two problems, one MySQL could never sought out how to cover my “additional expenses” and I was paying out $150 per month on my own dime, plus I had to physically go to a bank and physically transfer money to Australia (all under a magical $10K AUD limit). When you travel every week, and when I had a brutal travel schedule for a while I was not able to actually get to bank for three weeks.
  • If SUN doesn’t provide a corporate card, or handle expense of the two largest ticket items, airfares and accommodation, there will be hell to pay if I’m not re-reimbursed in very timely fashion. I have a corporate card with MySQL, a sore point actually, it took 12 months to get one, to no fault of my own.
  • Employment Visa. As I mentioned my live at MySQL is more complex then others, I work here in the US under the E3 visa, that is tied to an employer. This will need to change, and that I’m sure is no trivial task.
  • Changed Vacation Time (for the worse). MySQL has a very good Swedish Vacation policy, this goes. For an employee under 2 years my vacation time I hear is like a week less then 2+, and overall vacation time in general is less. They say more US based holidays and forced Christmas/New Year. I’m not the family guy, a day scattered here or there doesn’t work for me, I need blocks of time, the world is out there.
  • Java. My core programming expertize is Java. (sh/awk etc would argue a strong case). My strongest non-scripting language is Java, I’ve missed not being able to remain closer to application development, even stack technology, this may provide some opportunity where consulting engagements are MySQL and development related and I can refresh this skills. Going to another Java One again would be bonus :)
  • Additional Development/Testing Resources. Sun is a company 100 times the size of MySQL. Does this grant us dedicate resources that may be able to help the bleeding of MySQL. (Bleeding the time to market for features, quality therefore, and flexibility for growth). All things I consider still going backwards at MySQL. Additional procedures may make things worse, but MySQL I’m sure will remain a key different component for a long time.
  • Community I’m going to end my points on this. MySQL in my opinion does not do right by the community. I have some specific views I’ll be posting soon on this point. How will community function under SUN, will be allowed to thrive, or will it’s path be dictated as it is now, buy a sales and marketing driven focus. There are some things that need to change in MySQL, this is one of them. Will Marten Mickos, the present CEO the head of the MySQL group at Sun see this and have better direction in store. What I’d expect is to see Sales and Marketing be re-organized (this seems most logical), and then that leaves community to return to a key component such as Engineering & Services and not a side thought. I came from the community before I joined MySQL, I am active (less so) in the community while at MySQL, I stand up for the community at MySQL (again more on this soon), and I’ll be part of the community long after I leave MySQL and/or Sun.

My Ideas for MySQL Camp III

Diary: January 21st 2008 – Martin Luther King Day (Day doctor’s practices are closed BTW.)

“I have a dream”, poetic . Actually I have thumping 5 day straight headache but that’s another story.

I have a dream for MySQL Camp III. A 48 hour Global Hackfest. I ran this by Jay over Thanksgiving, to get back to more the purpose of the Camp, for hackers, coders and the very experience to get to together to share their skills, and for those at the top of our respective game to learn just a little more. MySQL Camp II was a success to attendees in general, but of little value to the experts.

I hope to get us middle to advanced ground. Here is an overview.

  • 48 hour event
  • Say 12pm Friday GMT to 12pm Sunday GMT
  • Global meeting points of two or more to work together, sleep, eat and play together (in fact the goal is not to attend alone, you should really try to get to at least the closest person that’s also attending, it’s designed to be a distributed group event)
  • 4 key areas
    • Getting Starrted. Getting the code, understanding the basics of compiling – Linux, Mac OS/X and Windoze. CLI & IDE debugging options
    • A&D. Analysis of problems, selected bugs, existing patches etc, reviewing procedures, documentation requirements, important of test plans etc. Laying down a plan on what’s going to happen, how long it’s going to take etc.
    • Doing it. Taking a working developer environment, and a set plan, and executing to completion
    • Reviewing it. Getting an insite into MySQL and how bug fixes, community contributions etc are submitted, reviewed, proposed and received
  • I would anticipate we would run say 2 or 3 tracks of these 4 points, so we would repeat stuff, perhaps a different problem, but this enables you to get a real grip, as well as cater for the 24hr cycle.
  • I could see the Doing it interesting, perhaps depending on attendees either a mentor process where a code guru could instruct some youngerlings, or multiple teams working in parallel on the same problem, a little bit of competition.
  • Code base line version and list of bugs/features to be looked at to be pre-determined, so we have a clear structure during the event. This will be proposal format, and may include for example back porting patches for example.
  • We will definitely be having some prizes and some fun, it’s going to be one of those work to 3am in the morning weekends regardless of where you live.

I expect to run this format of MySQL Camp twice, the first to work out any serious problems. I had hoped in November last year to get this before UC2008, fat chance, but I’m proposing a MySQL UC2008 a BOF on the subject.

I had spoken with good friend Farshan Mashraqi that had done some good Sun webinars recently and he was seeking a contact to see if Sun could donate the time/bandwidth for the electronic component or even sponsor. (This was clearly discussed late last year) Seems now this may be easier, or harder with Sun’s involvement.

I have a lot more details, but I want to get this out there into the world, and get some feedback first.
I’d like people’s feedback. Here are 5 questions to start with.

  1. Do you think it’s a good idea?
  2. Would you attended/participate?
  3. Could you contribute in some why? What?
  4. The one thing that appeals the most on the concept?
  5. The one thing that appeals the least/lacks/needs on the concept?

Compiling MySQL 5.0.51 under Ubuntu 7.10

I’ve finally decided to work on a number of improvements in Instrumentation within the MySQL Server I’ve wanted for the first year, See What is the optimal thread specific buffer size?. It’s been a while since I’ve compiled from source, and from these issues, the first under Ubuntu 7.10 (a fresh install). Here are some of the problems, and solutions overcome, just for some others that may experience them.

I should have simply read my own notes from years ago in Compiling MySQL, specifically the pre-requisites list, but it sometimes helps to remember why things are so.

In summary, I needed the following:

apt-get install automake libtool g++ ncurses-dev

Thanks Miademora, I meant to say that, forgot

First error, “aclocal: not found”

$ ./BUILD/compile-pentium-debug
BUILD/check-cpu: Oops, could not find out what kind of cpu this machine is using.
+ make -k distclean
make: *** No rule to make target `distclean'.
+ true
+ /bin/rm -rf */.deps/*.P config.cache innobase/config.cache bdb/build_unix/config.cache bdb/dist/autom4te.cache autom4te.cache innobase/autom4te.cache
+ path=BUILD
+ . BUILD/
+ aclocal
BUILD/ 1: aclocal: not found
+ die Can't execute aclocal
+ echo Can't execute aclocal
Can't execute aclocal
+ exit 1

This needed automake.

$ apt-get install automake
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  autoconf autotools-dev m4
Suggested packages:
  autoconf2.13 autobook autoconf-archive gnu-standards autoconf-doc
Recommended packages:
The following NEW packages will be installed:
  autoconf automake autotools-dev m4
0 upgraded, 4 newly installed, 0 to remove and 24 not upgraded.
Need to get 1085kB of archives.
After unpacking 3899kB of additional disk space will be used.
Do you want to continue [Y/n]? y
WARNING: The following packages cannot be authenticated!
  m4 autoconf autotools-dev automake
Install these packages without verification [y/N]? y
Get:1 gutsy/main m4 1.4.10-0ubuntu2 [207kB]
Get:2 gutsy/main autoconf 2.61-4 [448kB]
Get:3 gutsy/main autotools-dev 20070306.1 [61.6kB]
Get:4 gutsy/main automake 1:1.10+nogfdl-1 [369kB]
Fetched 1085kB in 2s (515kB/s)
Selecting previously deselected package m4.
(Reading database ... 93930 files and directories currently installed.)
Unpacking m4 (from .../m4_1.4.10-0ubuntu2_i386.deb) ...
Selecting previously deselected package autoconf.
Unpacking autoconf (from .../autoconf_2.61-4_all.deb) ...
Selecting previously deselected package autotools-dev.
Unpacking autotools-dev (from .../autotools-dev_20070306.1_all.deb) ...
Selecting previously deselected package automake.
Unpacking automake (from .../automake_1%3a1.10+nogfdl-1_all.deb) ...
Setting up m4 (1.4.10-0ubuntu2) ...

Setting up autoconf (2.61-4) ...

Setting up autotools-dev (20070306.1) ...
Setting up automake (1:1.10+nogfdl-1) ...

Second error, “libtoolize: not found”.

BUILD/check-cpu: Oops, could not find out what kind of cpu this machine is using.
+ make -k distclean
make: *** No rule to make target `distclean'.
+ true
+ /bin/rm -rf */.deps/*.P config.cache innobase/config.cache bdb/build_unix/config.cache bdb/dist/autom4te.cache autom4te.cache innobase/autom4te.cache
+ path=BUILD
+ . BUILD/
+ aclocal
+ autoheader
+ test -f /usr/bin/glibtoolize
+ libtoolize --automake --force
BUILD/ 1: libtoolize: not found
+ die Can't execute libtoolize
+ echo Can't execute libtoolize
Can't execute libtoolize
+ exit 1

Using a nice Ubuntu feature, you can find the right package by typing the command.

$ libtoolize
The program 'libtoolize' is currently not installed.  You can install it by typing:
sudo apt-get install libtool
bash: libtoolize: command not found

This needed the libtool package.

$ apt-get install libtool
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  libc6-dev linux-libc-dev
Suggested packages:
  glibc-doc manpages-dev libtool-doc g77 fortran77-compiler gcj
Recommended packages:
The following NEW packages will be installed:
  libc6-dev libtool linux-libc-dev
0 upgraded, 3 newly installed, 0 to remove and 24 not upgraded.
Need to get 4277kB of archives.
After unpacking 18.3MB of additional disk space will be used.
Do you want to continue [Y/n]? y
WARNING: The following packages cannot be authenticated!
  linux-libc-dev libc6-dev libtool
Install these packages without verification [y/N]? y
Get:1 gutsy-updates/main linux-libc-dev 2.6.22-14.47 [653kB]
Get:2 gutsy-proposed/main libc6-dev 2.6.1-1ubuntu10 [3287kB]
Get:3 gutsy/main libtool 1.5.24-1ubuntu1 [337kB]
Fetched 4277kB in 4s (900kB/s)
Selecting previously deselected package linux-libc-dev.
(Reading database ... 94193 files and directories currently installed.)
Unpacking linux-libc-dev (from .../linux-libc-dev_2.6.22-14.47_i386.deb) ...
Selecting previously deselected package libc6-dev.
Unpacking libc6-dev (from .../libc6-dev_2.6.1-1ubuntu10_i386.deb) ...
Selecting previously deselected package libtool.
Unpacking libtool (from .../libtool_1.5.24-1ubuntu1_i386.deb) ...
Setting up linux-libc-dev (2.6.22-14.47) ...
Setting up libc6-dev (2.6.1-1ubuntu10) ...
Setting up libtool (1.5.24-1ubuntu1) ...

Third error, “preprocessor “/lib/cpp” fails sanity check”

checking how to run the C++ preprocessor... /lib/cpp
configure: error: C++ preprocessor "/lib/cpp" fails sanity check
See `config.log' for more details.
$ more config.log
configure:5336: gcc -c -Wimplicit -Wreturn-type -Wswitch -Wtrigraphs -Wcomment -
W -Wchar-subscripts -Wformat -Wparentheses -Wsign-compare -Wwrite-strings -Wunus
ed-function -Wunused-label -Wunused-value -Wunused-variable -Woverloaded-virtual
 -Wsign-promo -Wreorder -Wctor-dtor-privacy -Wnon-virtual-dtor  -felide-construc
tors -fno-exceptions -fno-rtti  -DUNIV_MUST_NOT_INLINE -DEXTRA_DEBUG -DFORCE_INI
gcc: error trying to exec 'cc1plus': execvp: No such file or directory

This needed the package g++.

$ apt-get install g++
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  g++-4.1 libstdc++6-4.1-dev
Suggested packages:
  g++-multilib g++-4.1-multilib gcc-4.1-doc libstdc++6-4.1-doc
The following NEW packages will be installed:
  g++ g++-4.1 libstdc++6-4.1-dev
0 upgraded, 3 newly installed, 0 to remove and 24 not upgraded.
Need to get 3730kB of archives.
After unpacking 13.7MB of additional disk space will be used.
Do you want to continue [Y/n]? y
WARNING: The following packages cannot be authenticated!
  libstdc++6-4.1-dev g++-4.1 g++
Install these packages without verification [y/N]? y
Get:1 gutsy/main libstdc++6-4.1-dev 4.1.2-16ubuntu2 [1129kB]
Get:2 gutsy/main g++-4.1 4.1.2-16ubuntu2 [2600kB]
Get:3 gutsy/main g++ 4:4.1.2-9ubuntu2 [1440B]
Fetched 3730kB in 4s (871kB/s)
Selecting previously deselected package libstdc++6-4.1-dev.
(Reading database ... 95367 files and directories currently installed.)
Unpacking libstdc++6-4.1-dev (from .../libstdc++6-4.1-dev_4.1.2-16ubuntu2_i386.deb) ...
Selecting previously deselected package g++-4.1.
Unpacking g++-4.1 (from .../g++-4.1_4.1.2-16ubuntu2_i386.deb) ...
Selecting previously deselected package g++.
Unpacking g++ (from .../g++_4%3a4.1.2-9ubuntu2_i386.deb) ...
Setting up g++-4.1 (4.1.2-16ubuntu2) ...
Setting up libstdc++6-4.1-dev (4.1.2-16ubuntu2) ...
Setting up g++ (4:4.1.2-9ubuntu2) ...

Fourth error “checking for termcap functions library… configure: error: No curses/termcap library found”

This needed the package nurses-dev.

This got a successful compile. Again, should have started with the pre-requisites list. A review of the pre-requisites from previously showed gmake no longer necessary, the following is.

automake --version
autoconf --version
libtool --version
m4 --version
gcc --version
bison -version

What is the optimal thread specific buffer size?

So you want to know what join_buffer_size, sort_buffer_size, read_buffer_size and read_rnd_buffer_size values for your application should be? These MySQL thread specific buffers are variables I can never get right because there are insufficient metrics, instrumentation or even abstract details with the present MySQL Versions. These are important because Memory is a resource that you want to maximum towards your database data (the System Global Area), and not towards the Process Global Area in which there is no limitations.

I’ve wanted to know this answer for some time, I’ve asked many people including internal MySQL resources, I’d hoped that when joining MySQL more details would be available, but I’ve never been able to get an answer. I’ve always been meaning to work this out, it’s now 2008 and well the time has now come to do something about it.

The questions I want to answer include?

  • When are these buffers used, i.e. which SQL Statements trigger these. Ok, you can work this out when you review the SQL + Schema but I want something that works for clients.
  • I want to know if the buffers are fully allocated or incrementally allocated. My understanding is that for these 4 buffers, there are pre-allocated unlike internal temporary tables, but since reading some of the code I’m no longer completely convinced.
  • I want to know what size of the buffer (if fully allocated) is actually used.

My initial goal is to add the following new status variables.

  • sort_buffer_count, sort_buffer_usage
  • read_buffer_count, read_buffer_usage
  • read_rnd_buffer_count, read_rnd_buffer_usage
  • join_buffer_count, join_buffer_usage

Now, it’s likely there will be some overlap, for example sort_scan, but nothing like introducing some consistency here.

There per session status variables will allow you when reviewing individual SQL statements to see the impact. Ultimately I’d like to add more smarts into MySQL, because things like “gathering all SQL Statements” can be difficult and often not possible in production. Combined with leverage MySQL Proxy we will get there.

I’m looking forward to getting in the bows of the MySQL Source Code, it’s going to be a steep curve as I do little development these days, less in C++ but nothing like a challenge and questions to a lot of close colleagues for small snippets of help.