5.1.20 Gotcha – The MySQL Error Log

While using the latest MySQL 5.1.20 yesterday I came across another situation that was not expected as with previous editions of MySQL. The background is experimenting with DRBD. When I configured MySQL to startup with a /etc/my.cnf file with data on a DRBD partition I got a failed startup error message with mysqld_safe.

$ bin/mysqld_safe &
[1] 12615
070720 10:10:42 mysqld_safe Starting mysqld daemon with databases from /drbd/data
070720 10:10:42 mysqld_safe mysqld from pid file /drbd/data/newyork.localdomain.pid ended

Ok. Well this happens so I went to the data directory to look for `hostname`.err.

$ cd /drbd/data
$ ls -l

What the! There is no error log. Then the discussion started about this. Apparently mysqld_safe now uses syslog (e.g. /var/log/messages) for logging messages. Ok, but where is the line between mysqld_safe and mysqld. There was some confusion here about this, however investigation of /var/log/messages showed what was necessary to at least find the error.

$ tail -30 /var/log/messages
Jul 19 16:59:51 newyork mysqld_safe[6156]: Starting mysqld daemon with databases from /drbd/data
Jul 19 16:59:54 newyork mysqld[6160]: /usr/local/mysql/bin/mysqld: Can't find file: './mysql/plugin.frm' (errno: 13)
Jul 19 16:59:54 newyork mysqld[6160]: [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
Jul 19 16:59:54 newyork mysqld[6160]: InnoDB: The first specified data file ./ibdata1 did not exist:
Jul 19 16:59:54 newyork mysqld[6160]: InnoDB: a new database to be created!
Jul 19 16:59:54 newyork mysqld[6160]: InnoDB: Setting file ./ibdata1 size to 10 MB
Jul 19 16:59:54 newyork mysqld[6160]: InnoDB: Database physically writes the file full: wait...
Jul 19 16:59:54 newyork mysqld[6160]: InnoDB: Log file ./ib_logfile0 did not exist: new to be created
Jul 19 16:59:54 newyork mysqld[6160]: InnoDB: Setting log file ./ib_logfile0 size to 5 MB
Jul 19 16:59:54 newyork mysqld[6160]: InnoDB: Database physically writes the file full: wait...
Jul 19 16:59:54 newyork mysqld[6160]: InnoDB: Log file ./ib_logfile1 did not exist: new to be created
Jul 19 16:59:54 newyork mysqld[6160]: InnoDB: Setting log file ./ib_logfile1 size to 5 MB
Jul 19 16:59:54 newyork mysqld[6160]: InnoDB: Database physically writes the file full: wait...
Jul 19 16:59:54 newyork mysqld[6160]: InnoDB: Doublewrite buffer not found: creating new
Jul 19 16:59:54 newyork mysqld[6160]: InnoDB: Doublewrite buffer created
Jul 19 16:59:54 newyork mysqld[6160]: InnoDB: Creating foreign key constraint system tables
Jul 19 16:59:54 newyork mysqld[6160]: InnoDB: Foreign key constraint system tables created
Jul 19 16:59:54 newyork mysqld[6160]: InnoDB: Started; log sequence number 0 0
Jul 19 16:59:54 newyork mysqld[6160]: [ERROR] /usr/local/mysql/bin/mysqld: Can't find file: './mysql/host.frm' (errno: 13)
Jul 19 16:59:54 newyork mysqld[6160]: [ERROR] Fatal error: Can't open and lock privilege tables: Can't find file: './mysql/host.frm' (errno: 13)

To ensure backward compatibility now with previous versions of MySQL, it is necessary to specific in my.cnf the log-error parameter.

log-error = /drbd/data/mysql.err

Starting gives you expected error logging.

$ more /drbd/data/mysql.err
070720 10:16:30 mysqld_safe Starting mysqld daemon with databases from /drbd/data
/usr/local/mysql/bin/mysqld: Can't find file: './mysql/plugin.frm' (errno: 13)
070720 10:16:30 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
070720 10:16:30  InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
InnoDB: File name ./ibdata1
InnoDB: File operation call: 'open'.
InnoDB: Cannot continue operation.
070720 10:16:30 mysqld_safe mysqld from pid file /drbd/data/newyork.localdomain.pid ended

So, you need to read the fine print in the Changes in release 5.1.20 (25 June 2007) 5.1 Manual you get.

Incompatible change: mysqld_safe now supports error logging to syslog on systems that support the logger command. The new –syslog and –skip-syslog options can be used in conjunction with the –log-error option to control logging behavior, as described in Section 5.3.1, “mysqld_safe — MySQL Server Startup Script”. The default is to use syslog, which differs from the previous default behavior of writing an error log file. To maintain the older behavior, use the –log-error option. (Bug#4858)

If you read mysqld_safe — MySQL Server Startup Script you also see the following note.


As of MySQL 5.1.20, the default error logging behavior is to write errors to syslog on systems that support the logger program. This differs from the default behavior of writing an error log file for versions prior to 5.1.20. To maintain the older behavior, use the –log-error option.

RTFM those release notes!

MySQL 5.1.20 Installation

MySQL recently released 5.1.20 beta. I just ran into another gotcha, which prompted me to post this draft I’ve had for a while.

rbradford@newyork:/opt/mysql51$ scripts/mysql_install_db
Installing MySQL system tables...
070709 23:24:08 [Note] Plugin 'InnoDB' disabled by command line option
Filling help tables...
070709 23:24:08 [Note] Plugin 'InnoDB' disabled by command line option

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

To do so, start the server, then issue the following commands:
./bin/mysqladmin -u root password 'new-password'
./bin/mysqladmin -u root -h newyork password 'new-password'
See the manual for more instructions.
You can start the MySQL daemon with:
cd . ; ./bin/mysqld_safe &

You can test the MySQL daemon with mysql-test-run.pl
cd mysql-test ; perl mysql-test-run.pl

Please report any problems with the ./bin/mysqlbug script!

The latest information about MySQL is available on the web at


Support MySQL by buying support/licenses at http://shop.mysql.com

Wait, there is an Innodb error message I’ve not seen . I’ll have to look into that.

mysql> show engines;
| Engine     | Support | Comment                                                        | Transactions | XA  | Savepoints |
| InnoDB     | YES     | Supports transactions, row-level locking, and foreign keys     | YES          | YES | YES        |
| MRG_MYISAM | YES     | Collection of identical MyISAM tables                          | NO           | NO  | NO         |
| BLACKHOLE  | YES     | /dev/null storage engine (anything you write to it disappears) | NO           | NO  | NO         |
| CSV        | YES     | CSV storage engine                                             | NO           | NO  | NO         |
| MEMORY     | YES     | Hash based, stored in memory, useful for temporary tables      | NO           | NO  | NO         |
| FEDERATED  | YES     | Federated MySQL storage engine                                 | YES          | NO  | NO         |
| ARCHIVE    | YES     | Archive storage engine                                         | NO           | NO  | NO         |
| MyISAM     | DEFAULT | Default engine as of MySQL 3.23 with great performance         | NO           | NO  | NO         |
8 rows in set (0.00 sec)

mysql> show plugins;
| Name       | Status | Type           | Library | License |
| binlog     | ACTIVE | STORAGE ENGINE | NULL    | GPL     |
| partition  | ACTIVE | STORAGE ENGINE | NULL    | GPL     |
| CSV        | ACTIVE | STORAGE ENGINE | NULL    | GPL     |
| InnoDB     | ACTIVE | STORAGE ENGINE | NULL    | GPL     |
10 rows in set (0.00 sec)

Yet, it works fine! So what does message “[Note] Plugin ‘InnoDB’ disabled by command line option” really mean. On closer inspection it doesn’t say Error, however it’s a different message from previous installs, so it requires more investigate to know what’s happening.

OS Freedom

I’m at some internal MySQL training today at our US HQ. In a room of 11 people we are all using our laptops for access to additional machines. The beauty of this is we have Linux (Ubuntu 5.10, 6.10, 7.04, Red Hat Fedora), Mac OS/X and Windows Operating Systems on various staff machines. It’s great to realize we are not locked into just one infrastructure, and we have the ability to all do our work effectively and MySQL works just as well across these technologies. I can’t however SSH from my iPhone yet, unlike my previous Nokia E62 PDA.

The Hatchery July Event

Wednesday night I attended my second Hatchery event. The Hatchery is an opportunity for organizations and clever individuals seeking venture capital to make a proposal in a formal panel process with venture capitalists. This month’s event included 3 presenters, all 3 different from last month, each presenting in a better style (thanks to being prepped) in a revised format that included a longer presentation time, and an opportunity of questions from the floor. So the presenters.


On the Internet there is “no way to prove who people say they are”. safeTspace.com is an attempt to address this problem starting with the vertical of creating a safer Internet for children. This is a noble pursuit, it’s clearly needed, I applaud the attempt but it’s a battle that I believe can’t be solved via traditional means. I compared this pursuit with a two very common problems. 1: SPAM. This can’t be solved without eliminating the underlying email protocol that is flawed. 2: VIRUSES. This can’t be solved without the underlying Microsoft Operating System that is a virus incubator.

With these two examples there will always be spam filters and spam programs competing to eliminate spam, and the issue for example of false positives. There will be always be anti-virus software attempting to fix as quickly as possible problems or identification *after* a new virus is found and released to the world. Until Microsoft re-defines the way it secures it’s underlying operating system, and application suit from the ground up, those users that use this OS will continue to live with daily concern of viruses. Enough ranting.

It would appear from the presentation there are two steps of the process, authentication and verification. The authentication process involves a physical person verifying the child in question is a physical real person. The verification process is either a fingerprint scanner, or a camera. Fingerprint scanners for example are not secure, and there are plenty of non-standard entry level models already available, so a fingerprint could be faked. Photo recognition, what’s stopping somebody putting a photo of the child in front of the camera.

The problem is the system is only half of two necessary parts. Authenticating a real person to access the Internet doesn’t stop this person from then doing anything like pretending to be somebody else. While the purpose of the pursuit is to ensure the reverse, this first point can also occur. The co-operation of all websites is needed but how can you get a buy in from sites such as MySpace, FaceBook, YouTube etc. I work in this industry, I know people that work for these organizations, I work with large organizations such as these, I know this type of implementation will not happen easily without significant incentive, and there are millions of community web sites, millions.

It was interesting that this project started “from something that came out of an argument”. A comment that was re-iterated by one of the panel saying this was a good thing. I can’t say I agree here, active discussion promotes collaboration, and opinionated views. Arguments rarely achieve anything other then resulting in personal attacks and usually is over something less then very important.

While all three key people were all from the Department of Homeland Security (good to see them all there for the presentation), the comments regarding the proposal included the lack of a technology security expert, and resources with significant Sales & Marketing background. This re-iterates what I have learned from my experiences in working with startup companies and venture capital using the Bell Mason Diagnostic (BMD). This approach tackles 4 different stages of development, 4 quadrants with 12 axis of analysis that covers Technology, CEO, Product, Team, Software Development, Board, Business Plan , Cash, Marketing, Financeability, Sales and Controls. What I know and don’t have in any of my own ventures is contacts and involvement of people covering these areas of required expertise. It’s an important checklist with anybody that has an idea.


The second presenter was for ParkEhiz.com, a quick and easy way to search for and pay in advance for parking. With one click access to your city, Google Maps mashup integration, slider refresh of distance and price (but not rating) the website provides quick access to information and quick filtering, something necessary for a website success, and something that can kill you in performance with a successful site. Still, good points scored from me here for the practical site.

The presenter did a good job. While clearly a technical person, his enthusiasm towards the idea was evident and this is one side to promoting you idea. There were clearly areas of greater analysis in the business plan necessary such as securing major clients, and considering ideal marketing and pricing plans. Panel Investor Hugh Cullman post presentations comments stated in his discussion that the zeal of the presenter and not just the business of the presentation contributed to evaluating proposals early in the funding process. At the ring of the 7 minute timer, when asked to complete the sentence the reply was “it’s going to be a 7 slide sentence”. The presenters know the terms before the presentation, they were also prepared prior. It’s very important that your professionalism includes following the rules.

The ParkWhiz Guarantee is an interesting offer. A 100% guarantee is parking is not honored. This will become an issue if the “phone for an alternative” floods the most likely single phone operator for now. With a dependency on more traditional communication means additional resourcing will always be necessary.

Convenience is a strong selling point. However, having worked for a failed Internet startup from 1999-2002 that had 3 rounds of funding and one significant project that worked with bringing *buyers* and *sellers* together I had a number of points of input. The most significant is meeting the technology capabilities of the “buyer” and “seller”. I saw huge problems here, and this was clearly raised by one of the panel members saying “I know my parking garage just got an answering machine”. In this instance, the reliance on Parking Garages to have Internet access, and to use this in a timely manner will simply never work. Likewise for those looking for last minute parking, people may not both have readily available Internet access not the desire to pay in advance. Combine the management fee for this service of 25%, and I’m sure the desire of less reputable garages for a cash business, as well as the handling of money twice is also and overhead. That is taking money from the buyers, and then passing onto the sellers.

To overcome this I would suggest two things. First, for this to work you need to meet the sellers with the technology they can support, and the most I could see here is cell phone text messaging. No more. The second is providing a monthly fee service for buyers, that again via Cell phone, Text Message, email or PDA version provide a buyer with a list of parking garages with prices and times in the area provided. Would a service of $5-$10 a month work here. Well I guess only trial and error can tell. I did find out that a PDA version of the website existed, however on my iPhone (which is a full web browser), I was stuck with the PDA version.


The final presentation was Newstin, a Global News Aggregator. On first inspection this site showed nothing more then what I get with Google News. The presenter who didn’t stand still (making it interesting for the videographer) was quick to indicate the key differences, the first was the number of feeds being significantly more, and the second being translation capabilities via a machine translation technologies.

This presentation described the direct competition with Googke, Topix.net and Factiva, and similar services as Bloomberg. This information is always necessary, you have to set your apart from your competitors. You have make yourself unique. That uniqueness was the schematic keyword search and integrated translation. This analysis included a patent. The presentation included description of technology partners and it seemed while not clear to indicate that translation was performed by a third party.

The service however is a niche product at $2500 per year. It was also immediately clear that Americans and America can never be considered a primary source for this service. I’ve spoken previously regarding my opinions on the clear lack of World News by the US media, and a clear false view presented to the viewing public in this country. My thoughts on CNN International was also echoed by one of the panel. The key target market is clearly Europe, a point raised many times by the panel. What was not mentioned and I consider an emerging market is Asia. Korea, Japan are powerhouse large Internet communities and I’m sure China and other Asian countries will become likewise.

And to include a quote from the presenter, when asked how much money do you have left from initial funding the response was “Just about enough to get me here to ask for more”. They were clearly asking for a lot and with 30 staff in Prague, Czech Republic resources were also working for peanuts.

Post Presentation

One great thing about this meetup is the opportunity to talk more with the presenters afterwards, to also network and this time (as well as before) and the opportunity for a drink with some people following completion. I had a chance when discussing an idea to mention The Purple Cow by Seth Godin. A quick and easy read, but an inspiration in thinking outside the square. I had an opportunity to show my Moo Cards which last month were on order, but used by one of the presenters. What was surreal was as a showed my cards, one was randomly selected just as the NewsTin presenter approached. The card selected was of the Praque Castle (the only Praque photo of the collection) and Praque being the home city of operations for NewsTin. Coincidence!

The fobar of a Web 2.0 website

Web 2.0 is all about community driven content. Recently eBay purchased Stumble Upon for $75 million. There is a problem here. When I first heard of the site, I looked at. I remember going back the next few days, and I was sure it hadn’t changed. Then I started taking screen shots. Having forgotten about it now for over a month under a discussion today, I took another screenshot. In over a month, from June 3rd to July 15th there has been no content change to the website. There has been a change to the counter of number of stumblers, and a change of image for a Recent Stumblers, but the “content”, the recent popular web site on the first page of the website remains unchanged. Check out my screenshots.

-rw-r--r-- 1 rbradford rbradford 568715 2007-07-15 19:13 stumbleapon.15072007.png
-rw-r--r-- 1 rbradford rbradford 570622 2007-06-03 21:32 stumbleupon.03062007.png
-rw-r--r-- 1 rbradford rbradford 580385 2007-06-04 18:25 stumbleupon.04062007.png
-rw-r--r-- 1 rbradford rbradford 586082 2007-06-07 00:22 stumbleupon.07062007.png
-rw-r--r-- 1 rbradford rbradford 570792 2007-06-15 02:03 stumbleupon.15062007.png

Cool Product – OQO

My flatmate just go an OQO and he hasn’t stopped raving about it.

A small PDA type device, however it runs a full version of Windows XP, has a full keyboard, mouse and keypad, a screen of 800 x 480 and build-in Wi-Fi and wireless broadband that can be connected to a telephone provider. Apparently with Skype installed you could then use it as a phone. He just tested it as we were talking to see if it had a built-in mic. I must admit even I’m impressed. Something I’ll need to play with a little more. Doesn’t beat my iPhone however for the cool alpha geek, in your pocket sized being able to show my photos & view the web device.

MySQL NY Meetup Presentation

Tonight I spoke at the NY MySQL Meetup. The topic “Practical Performance Tips & Tricks” was a full packed 1 hour session, with 4 x 15 minute sections on Beginner, Intermediate, Advanced and MySQL Proxy. The goal to hopefully cover content for different level of attendees. This meeting followed up the large turnout from last month’s meeting with at least 35 people. Thanks again to Logicworks for sponsoring the night and providing the beer and food, especially to adjust for the late arriving presenter.

My Slides are download able in PDF format here.

Some additional links that were discussed during the meeting for reference included.

Details of my previous MySQL presentations can be found in my Articles section.

Other iPhone experiences for corporate users

I am not alone in my experience as a corporate AT&T/Cingular user wanting an iPhone. See my last post iPhone activation – Finally after 3 days. In the past 30 minutes I’ve read the same pain from many people. How can Apple and AT&T get this so wrong?

Some posts of interest:

No iPhones for corporate accounts?!
Anyone else here on an AT&T corporate account?
Apple, er AT&T, tells corporate types to pound sand
No iPhone for business account customers at launch?

iPhone activation – Finally after 3 days

If you have been following my iPhone saga, last time was iPhone for corporate users. What a debacle

So after 3 days, my phone is finally activated, however it was no as simple even with the required hoops mentioned last time.

I started by getting my company to add iPhone as an acceptable device for *our own* company account. The response was “My rep has stated that he can not add the iphone to our corporate account. This is currently not allowed to do it by AT&T corporate. Our only option is #2 which is to release your number and port it back to a personal account. “

What the! So my number is released, and I start the process of getting my iPhone on a personal account. I take the option of calling 1.866.895.1099 rather then trying with iTunes. I explain the situation, and then I’m asked for the account number. I don’t have that, it’s a company account, then I’m asked to type in *#06# on the phone, but of course I’m calling on my phone and that doesn’t work. Then I had to prove I was the owner of the phone by providing 3 recent numbers called. Of course this is in the phone I’m presently using.

Ok we overcome this, then I’m told I may have to pay a security deposit on my phone. What the! More questions including what is your Drivers License Number. Well I don’t have a US drivers license. Anyway, a few more minutes and I’m told I would not have to pay a security deposit.

Continuing on, I was told my account was created, and then I could use iTunes to activate my phone. Wanting to ensure this all worked smoothly I was still online with the operator. I had to agree to the “Apple and Third Party Terms and Conditions” and “AT&T – Terms of Service”. I made a copy of these pages, but didn’t keep the links that they reference unfortunately.

At the end of the process, I click the final button in iTunes and the phone goes dead. Doh!. My Apple iPhone is immediately activated and my present phone is de-activated. A few minutes later I actually had the operator I was talking with call me back, so that counts as one of the few positive experiences in this entire process.

Given this has been a forgettable experience I state I specifically want to make a complaint and I want a number for complaints. I’m told there isn’t one, I have to register online at the AT&T website and then find the email for contacting them. Well, that’s just not customer service.

So finally at like 1am Tue Jul 3rd 3 days after my purchase, my useless paper weight is usable.

iPhone for corporate users. What a debacle

What a debacle. Here is how poorly AT&T have handled the management of the iPhone for the millions of “individual” users that have their bill paided for under a corporate account.

First, prior to release I inquired with my company’s technical people who contacted our local rep. Here is what they had to say.

“I’ve spoken to our Cingular rep and he not going to be able to sell the iPhone to business customers for 1-3 months. I suppose this is to keep stock high in the retail channel. Your best bet, according to him, is to go directly to an Apple store and buy it there. He thinks it’s a better bet then going to a Cingular store.”

So on the night of the release I goto the New York Apple Store, which is huge and open 24 hours, inquire about getting an iPhone given I’m on a corporate account. I have to ask a sales person who asked another more senior who asked another more senior again (it all was rather smoothly handled) until I get the response, “Yes, I can get an iPhone given I’m on an existing corporate account, I simply have to a call AT&T for the activation process and get a new 2 year contract”. That’s it. None of the detail I’m about to discuss. So, as I’ve recently written I did indeed buy the phone. (Includes Pictures & Video)

The first time I call Friday Night (now actually 1am Saturday morning), no answer other then a very boring on hold message for 30 minutes.

I call a second time Saturday midday and again, no answer other then a very boring on hold message for 30 minutes.

On Sunday, call a third time at midday, and surprisingly after 20-25 mins I get an operator. Before that story, it was interesting to observe a change in the on hold message from the previous two times. This one I recorded, I wish I’d had the earlier ones. This time the message clearly states. “Your iPhone is not eligible for any corporate discount.” Wow, just some little detail nobody has at hand before.

Of course, I can’t get my phone activated. What a surprise. As the operator stated, paraphrasing “The iPhone is intended for individuals. We didn’t expect this need for corporate account uses”. I’m sorry, I’m an “individual”, I just have somebody paying the bill on my behalf. AT&T/Apple you blew it here.

So in order to get my iPhone activated I have to contact the appropriate person at my company with the right authority (fat chance that will happen on a weekend, Friday 6pm realize of a product, and nobody considered this, either the did and decided to let the pain happen or the didn’t, either way it’s rather poor product release consideration. So the right person has to contact AT&T and either allow to list the iPhone as a regular device or Release my number (without penalty I’m told). Given there is no corporate discount either option works for me, I just want to play and show off my $600 useless phone. I should also add, my phone account already runs at about $120 per month, I have unlimited data already, so it’s not like switching to an iPhone I would actually be paying more from my account. The operator could see that, so where is the problem. If I had a phone on a corporate account without unlimited data already I could see this being an issue.

I also find out on this call another piece of information I didn’t know. The iPhone is an non insurable device. I’m not sure exactly what that means, but I take it AT&T offer phone insurance, except for the iPhone. Does this mean it won’t last the distance of say 12 months.

Overall the operator was as helpful as she could be, she was courteous and provided all the information I needed, but not with the result I wanted.

AT&T and Apple have done a piss poor job of the release for “individual users” that simply have there phone account paided differently. It’s been a rather unacceptable experience. I’m disappointed in the process and now a full 2 days later I still have a useless iPhone that I can’t activate.

And for the record, I went to the iPhone website to make a complaint. You think after looking at all the contact details I could find a link to do this. Seems, they may not be happy with people making complaints.

Update You can read the final conclusion at iPhone activation – Finally after 3 days