You are here: Home / Archives for Technologies

Damm you Peformance Schema

by

One significant new feature of MySQL 5.5 is the Performance Schema. I recently performed an upgrade from 5.0 to 5.5, however my check of differences in the MySQL variables via mysqladmin variables failed because we now have a new record long variable name “performance_schema_events_waits_history_long_size”.

The full list of new performance_schema variables are:

> | performance_schema_events_waits_history_long_size | 10000                                                                                                                  |
> | performance_schema_events_waits_history_size      | 10                                                                                                                     |
> | performance_schema_max_cond_classes               | 80                                                                                                                     |
> | performance_schema_max_cond_instances             | 1000                                                                                                                   |
> | performance_schema_max_file_classes               | 50                                                                                                                     |
> | performance_schema_max_file_handles               | 32768                                                                                                                  |
> | performance_schema_max_file_instances             | 10000                                                                                                                  |
> | performance_schema_max_mutex_classes              | 200                                                                                                                    |
> | performance_schema_max_mutex_instances            | 1000000                                                                                                                |
> | performance_schema_max_rwlock_classes             | 30

Not really a big problem, and not a complaint, just no longer a simple diff.

The Casual MySQL DBA – Operational Basics

by

So your not a MySQL DBA, but you have to perform like one. If you have a production environment that’s running now, what are the first things you do when it’s not running or reported as not running?

  1. Are the MySQL processes running? (i.e. mysqld and mysqld_safe)
  2. Can you connect locally via cli?
  3. What’s in the MySQL error log?
  4. What are current MySQL threads doing? Locked? long running? how many? idle sources?
  5. Can you connect remotely via cli?
  6. Verify free diskspace?
  7. Verify system physical resources?
  8. If this is a slave, is MySQL replication running? Is it up to date?
  9. What is the current MySQL load, e.g. reads/writes/throughput/network/disk etc?
  10. What is the current InnoDB state and load? (based on if your using InnoDB)

After you do this manually more then once you should be scripting these commands to be productive for future analysis and proactive monitoring?

Is a problem obvious? Does the output look different to what a normal environment looks like? (HINT: This list is not just for when there is a problem)

So moving forward?

  1. Is disk/memory/cpu/network bottleneck an issue you can resolve?
  2. Can you improving locking statements (if applicable)?
  3. Can you identify, analyse and tune long running statements?
  4. Do you know how to restart MySQL?
  5. Do you know who to call when you have a non working environment?
  6. When did your backup last run?
  7. Does your last backup work?

In order to support any level of production MySQL environment you need to know the answers to these questions? If you don’t, then this is your homework checklist for MySQL DBA operations 101. There a number of resources where you can find the answers, and this help can be available online, however never assume the timeliness of responses, especially if your expecting if for FREE! Open source software can be free, open source support rarely is.

MySQL HandlerSocket under Ubuntu

by

Starting with the great work of Yoshinori-san Using MySQL as a NoSQL – A story for exceeding 750,000 qps on a commodity server and Golan Zakai who posted Installing Dena’s HandlerSocket NoSQL plugin for MySQL on Centos I configured and tested HandlerSocket under Ubuntu 10.04 64bit.

NOTE: This machine already compiles MySQL and Drizzle. You should refer to appropriate source compile instructions for necessary dependencies.

# Get Software
cd /some/path
export DIR=`pwd`
wget http://download.github.com/ahiguti-HandlerSocket-Plugin-for-MySQL-1.0.6-10-gd032ec0.tar.gz
wget http://mysql.mirror.iweb.ca/Downloads/MySQL-5.1/mysql-5.1.52.tar.gz
wget  http://mysql.mirror.iweb.ca/Downloads/MySQL-5.1/mysql-5.1.52-linux-x86_64-glibc23.tar.gz

# Install MySQL Binaries
tar xvfz mysql-5.1.52-linux-x86_64-glibc23.tar.gz
ln -s mysql-5.1.52-linux-x86_64-glibc23 mysql5152
rm -rf mysql5152/mysql-test
rm -rf mysql5152/sql-bench
cd mysql5152
scripts/mysql_install_db
bin/mysqld_safe &

# Install MySQL Source
tar xvfz mysql-5.1.52.tar.gz

# Install HandlerSocket
tar xvfz ahiguti-HandlerSocket-Plugin-for-MySQL-1.0.6-10-gd032ec0.tar.gz
cd ahiguti-HandlerSocket-Plugin-for-MySQL-d032ec0/
./autogen.sh
./configure --with-mysql-source=${DIR}/mysql-5.1.51 --with-mysql-bindir=${DIR}/mysql5152/bin
make
sudo make install

# libtool: install: /usr/bin/install -c .libs/handlersocket.lai /home/rbradfor/projects/handlersocket/mysql5152/lib/plugin/handlersocket.la

# Install the Perl dependency
cd perl-Net-HandlerSocket
perl Makefile.PL
Checking if your kit is complete...
Warning: the following files are missing in your kit:
	lib/HandlerSocket.pm
Please inform the author.
Writing Makefile for Net::HandlerSocket
# For some reason I had to run this twice, after error I check, saw nothing wrong, ran a second time and it worked ???
perl Makefile.PL
Writing Makefile for Net::HandlerSocket
make
sudo make install

Files found in blib/arch: installing files in blib/lib into architecture dependent library tree
Installing /usr/local/lib/perl/5.10.1/auto/Net/HandlerSocket/HandlerSocket.so
Installing /usr/local/lib/perl/5.10.1/auto/Net/HandlerSocket/HandlerSocket.bs
Installing /usr/local/lib/perl/5.10.1/Net/HandlerSocket.pm
Installing /usr/local/lib/perl/5.10.1/Net/HandlerSocket/Pool.pm
Installing /usr/local/man/man3/Net::HandlerSocket.3pm
Appending installation info to /usr/local/lib/perl/5.10.1/perllocal.pod

cp lib/Net/HandlerSocket.pm blib/lib/Net/HandlerSocket.pm
cp lib/Net/HandlerSocket/Pool.pm blib/lib/Net/HandlerSocket/Pool.pm
/usr/bin/perl /usr/share/perl/5.10/ExtUtils/xsubpp  -typemap /usr/share/perl/5.10/ExtUtils/typemap  HandlerSocket.xs > HandlerSocket.xsc && mv HandlerSocket.xsc HandlerSocket.c
Please specify prototyping behavior for HandlerSocket.xs (see perlxs manual)
g++ -fPIC -c  -I. -I../libhsclient -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -O3 -Wall -Wno-unused   -DVERSION="0.01" -DXS_VERSION="0.01" -fPIC "-I/usr/lib/perl/5.10/CORE"   HandlerSocket.c
Running Mkbootstrap for Net::HandlerSocket ()
chmod 644 HandlerSocket.bs
rm -f blib/arch/auto/Net/HandlerSocket/HandlerSocket.so
LD_RUN_PATH="/usr/local/lib" g++ -fPIC  -shared -O2 -g -L/usr/local/lib -fstack-protector HandlerSocket.o  -o blib/arch/auto/Net/HandlerSocket/HandlerSocket.so 	
	   -lhsclient  	

chmod 755 blib/arch/auto/Net/HandlerSocket/HandlerSocket.so
cp HandlerSocket.bs blib/arch/auto/Net/HandlerSocket/HandlerSocket.bs
chmod 644 blib/arch/auto/Net/HandlerSocket/HandlerSocket.bs
Manifying blib/man3/Net::HandlerSocket.3pm

Now all built, it’s as simple as the following to install:

${DIR}/mysql5152/bin/mysql -uroot
mysql> INSTALL PLUGIN HandlerSocket SONAME 'handlersocket.so';
mysql> SHOW PLUGINS;
+---------------+----------+----------------+------------------+---------+
| Name          | Status   | Type           | Library          | License |
+---------------+----------+----------------+------------------+---------+
| binlog        | ACTIVE   | STORAGE ENGINE | NULL             | GPL     |
| partition     | ACTIVE   | STORAGE ENGINE | NULL             | GPL     |
| ARCHIVE       | ACTIVE   | STORAGE ENGINE | NULL             | GPL     |
| BLACKHOLE     | ACTIVE   | STORAGE ENGINE | NULL             | GPL     |
| CSV           | ACTIVE   | STORAGE ENGINE | NULL             | GPL     |
| FEDERATED     | DISABLED | STORAGE ENGINE | NULL             | GPL     |
| MEMORY        | ACTIVE   | STORAGE ENGINE | NULL             | GPL     |
| InnoDB        | ACTIVE   | STORAGE ENGINE | NULL             | GPL     |
| MyISAM        | ACTIVE   | STORAGE ENGINE | NULL             | GPL     |
| MRG_MYISAM    | ACTIVE   | STORAGE ENGINE | NULL             | GPL     |
| handlersocket | ACTIVE   | DAEMON         | handlersocket.so | BSD     |
+---------------+----------+----------------+------------------+---------+
11 rows in set (0.00 sec)

However, I found it not actually running even after restarting MySQL. You need the following configuration

cd ${DIR}/mysql5152
echo "[mysqld]
plugin-load=handlersocket.so
loose_handlersocket_port = 9998 # the port number to bind to (for read requests)
loose_handlersocket_port_wr = 9999 # the port number to bind to (for write requests)
loose_handlersocket_threads = 16 # the number of worker threads (for read requests)
loose_handlersocket_threads_wr = 1 # the number of worker threads (for write requests)" >> my.cnf

After restarting you get the following confirmation.

mysql>SHOW PROCESSLIST;
+----+-------------+-----------------+---------------+---------+------+-------------------------------------------+------------------+
| Id | User        | Host            | db            | Command | Time | State                                     | Info             |
+----+-------------+-----------------+---------------+---------+------+-------------------------------------------+------------------+
|  1 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
|  2 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
|  3 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
|  4 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
|  5 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
|  6 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
|  7 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
|  8 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
|  9 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
| 10 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
| 11 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
| 12 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
| 13 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
| 14 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
| 15 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
| 16 | system user | connecting host | NULL          | Connect | NULL | handlersocket: mode=rd, 0 conns, 0 active | NULL             |
| 17 | system user | connecting host | handlersocket | Connect | NULL | handlersocket: mode=wr, 0 conns, 0 active | NULL             |
| 18 | root        | localhost       | NULL          | Query   |    0 | NULL                                      | SHOW PROCESSLIST |
+----+-------------+-----------------+---------------+---------+------+-------------------------------------------+------------------+
18 rows in set (0.00 sec)

My Test Data

USE test
CREATE TABLE user (
  user_id INT UNSIGNED NOT NULL,
  name VARCHAR(50) NOT NULL,
  email VARCHAR(255) NOT NULL,
  created TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY(user_id)
) ENGINE=InnoDB;

INSERT INTO user (user_id,name, email) VALUES
(100,'Mickey Mouse','[email protected]'),
(101,'Marvin the Martian','[email protected]'),
(102,'Spiderman','[email protected]');

SELECT user_id,name,email FROM user WHERE user_id=101;
+---------+--------------------+---------------------+
| user_id | name               | email               |
+---------+--------------------+---------------------+
|     101 | Marvin the Martian | [email protected] |
+---------+--------------------+---------------------+

My test retrieval program is basically unchanged from the original post.

$ cat retrieve.pl
#!/usr/bin/perl

use strict;
use warnings;
use Net::HandlerSocket;

#1. establishing a connection
my $args = { host => '10.0.0.6', port => 9998 };
my $hs = new Net::HandlerSocket($args);

#2. initializing an index so that we can use in main logics.
 # MySQL tables will be opened here (if not opened)
my $res = $hs->open_index(0, 'test', 'user', 'PRIMARY',
    'name,email,created');
die $hs->get_error() if $res != 0;

#3. main logic
 #fetching rows by id
 #execute_single (index id, cond, cond value, max rows, offset)
$res = $hs->execute_single(0, '=', [ '101' ], 1, 0);
die $hs->get_error() if $res->[0] != 0;
shift(@$res);
for (my $row = 0; $row < 1; ++$row) {
  my $user_name= $res->[$row + 0];
  my $user_email= $res->[$row + 1];
  my $created= $res->[$row + 2];
  print "$user_namet$user_emailt$createdn";
}

#4. closing the connection
$hs->close();

If your HandlerSocket is not running you will see:

$ perl retrieve.pl
FATAL_EXIT: getaddrinfo failed: ip_to_remote_host:9998
$ telnet 10.0.0.6 9998
Trying 10.0.0.6...
telnet: Unable to connect to remote host: Connection refused

When working

$ telnet 10.0.0.6 9998
Trying 10.0.0.6...
Connected to 10.0.0.6.
Escape character is '^]'.
P	0	test	user	PRIMARY	name,email,created
0	1
0	=	1	101
0	3	Marvin the Martian	[email protected]	2010-11-05 13:11:34


$ perl retrieve.pl
Marvin the Martian	[email protected]	2010-11-05 13:11:34

Again, great work by Yoshinori-san to have this code as production deployable. Now to run some real benchmarks.

Improving MySQL Insert thoughput

by

There are three simple practices that can improve general INSERT throughput. Each requires consideration on how the data is collected and what is acceptable data loss in a disaster.

General inserting of rows can be performed as single INSERT’s for example.

INSERT INTO table (col1, col2, col3) VALUES (?, ?, ?);
INSERT INTO table (col1, col2, col3) VALUES (?, ?, ?);
INSERT INTO table (col1, col2, col3) VALUES (?, ?, ?);

While this works, there are two scalability limitations. First is the network overhead of the back and forth of each SQL statement, the second is the synchronous nature, that is your code can not continue until your INSERT is successfully completed.

The first improvement is to use MySQL’s multi values capability with INSERT. That is you can insert multiple rows with a single INSERT statement. For example:

INSERT INTO table (col1, col2, col3) VALUES (?, ?, ?),  (?, ?, ?), (?, ?, ?);

Depending on how you collect the information to be inserted, you may be able to easily implement this. The benefit, as previously mentioned in The value of multi insert values shows an 88% improvement in performance.

One system variable to be aware of is max_allowed_packet. You may have to adjust this for larger INSERT statements.

Next is the ability to INSERT data based on information already in another table. You can also leverage for example another storage engine like MEMORY to batch up data to be inserted via this approach.

INSERT INTO table (col1, col2, col3) SELECT col1, col2, col3 FROM anothertable

The third option is to batch load your data from a flat file. Depending on how you source the information you are inserting, that may also be a significant improvement in throughput in bulk inserting data.

LOAD DATA [CONCURRENT] INFILE 'file'
INTO TABLE (col1, col2, col3)

On a closing note, the choice of storage engine can also have a significant effect on INSERT throughput. MySQL also offers other non ANSI options including DELAYED, IGNORE and LOW_PRIORITY. These should definitely be avoided.

OTN MySQL conference slides

by

2010 has been the first year I have re-presented any of my developed MySQL presentations. Historically I have always created new presentations, however Paul Vallee gave me some valuable advice at UC 2010. In the past two weeks I’ve traveled to seven countries in South America on the OTN LA tour where I have been speaking about and promoting MySQL.

My three current presentations have been improved and even simplified, more future improvements are planned. There is definitely a benefit in repeating a good presentation multiple times.

My SQL Idiosyncrasies That Bite OTN
MySQL Best Practices – OTN
MySQL Scalability Mistakes – OTN

MySQL Support Options

by

Oracle has released news about changing policies of MySQL Enterprise Support effectively dropping annual support for Basic and Silver. The entry level support is now $3000 per server per year. The MySQL support team now part of Oracle has great resources however Oracle is in the business of making money. When a general company question for OOW is company income, and the first option is < $50 million it highlights that startups, and smaller companies are clearly not a focus.

The success of MySQL as an open source company has lead to other leading providers that now can provide enterprise level support. More importantly many organizations over per-incident support which is more cost effect. News in the past week has included Percona - New Support Option for MySQL, and today SkySQL with Kaj Arno’s Joining SkySQL Ab, back in the startup business. (Motivation for me to publish this draft)

For reference, this is what I received from Oracle prompting this post.

For those of you using Basic and Silver support we’re being told those options will no longer be available. If you wish to continue with Basic or Silver you will need to sign a multi-year agreement and you would be able to keep using Basic or Silver for up to another 3 years.

MySQL Best Practices for DBAs and Developers

by

This is one of the MySQL presentations I’m doing on the OTN LAD Tour in South America, starting today in Lima, Peru.

MySQL Best Practices for DBAs and Developers

Learn the right techniques to maximize your investment in MySQL by knowing the best practices for DBAs and Developers. Understand what subtle differences between MySQL and other RDBMS products are essential to understand in order to maximize the benefits and strengths of MySQL. We will be covering areas including the minimum MySQL configuration, ideal SQL, MySQL security and schema optimizations.

  • MySQL Configuration default settings including SQL_MODE
  • Documenting, formatting and future proofing your SQL
  • Developing and reviewing all SQL paths
  • MySQL physical and user security
  • The best schema optimizations
  • Essential Monitoring and Instrumentation
  • The locking essentials for different storage engines
  • Managing your Disk I/O with optimal storage and access
MySQL Best Practices – OTN LAD Tour

MySQL 5.5 and transaction management

by

Announced at MySQL Sunday was the Release Candidate edition of MySQL 5.5.6. Also noted by Geert where he points out the default storage engine is now InnoDB.

However, for those from a background other then MySQL there is still a gotcha.

mysql> show global variables like 'autocommit';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| autocommit    | ON    |
+---------------+-------+

Unlike Oracle for example, the default autocommit is on.

Doing some other boundary conditions, it is no longer possible to disable InnoDB on startup which you would of course expect.

$ bin/mysqld_safe --skip-innodb &
$ tail error.log

101003 15:33:32 [Note] Plugin 'InnoDB' is disabled.
101003 15:33:32 [ERROR] Unknown/unsupported storage engine: InnoDB
101003 15:33:32 [ERROR] Aborting

MyISAM however can’t be removed and can’t be disabled. This is a question I’ve been asked by Oracle resources.

101003 15:34:55 [ERROR] /Users/rbradfor/mysql/mysql-5.5.6-rc-osx10.5-x86_64/bin/mysqld: ambiguous option '--skip-myisam' (--skip-myisam-block-size)
101003 15:34:55 [ERROR] Parsing options for plugin 'MyISAM' failed.
101003 15:34:55 [ERROR] Failed to initialize plugins.
101003 15:34:55 [ERROR] Aborting

mysql> select table_schema, table_name from information_schema.tables where engine='myisam';
+--------------------+---------------------------+
| table_schema       | table_name                |
+--------------------+---------------------------+
| information_schema | COLUMNS                   |
| information_schema | EVENTS                    |
| information_schema | PARAMETERS                |
| information_schema | PARTITIONS                |
| information_schema | PLUGINS                   |
| information_schema | PROCESSLIST               |
| information_schema | ROUTINES                  |
| information_schema | TRIGGERS                  |
| information_schema | VIEWS                     |
| mysql              | columns_priv              |
| mysql              | db                        |
| mysql              | event                     |
| mysql              | func                      |
| mysql              | help_category             |
| mysql              | help_keyword              |
| mysql              | help_relation             |
| mysql              | help_topic                |
| mysql              | host                      |
| mysql              | ndb_binlog_index          |
| mysql              | plugin                    |
| mysql              | proc                      |
| mysql              | procs_priv                |
| mysql              | servers                   |
| mysql              | tables_priv               |
| mysql              | time_zone                 |
| mysql              | time_zone_leap_second     |
| mysql              | time_zone_name            |
| mysql              | time_zone_transition      |
| mysql              | time_zone_transition_type |
| mysql              | user                      |
+--------------------+---------------------------+
30 rows in set (0.06 sec)

Common MySQL Scalability Mistakes

by

This week I was one of the presenters at the first Surge Scalability Conference in Baltimore. An event that focused not just on one technology but on what essential tools, technologies and practices system architects need to know about for successfully scaling web applications.

While MySQL is an important product in many environments, it is only one component for a successful technology stack and for many organizations is one of several products that manage your data.

My presentation was on the common MySQL scalability mistakes and how to avoid them. This is a problem/solution approach and is a companion talk with Successful MySQL Scalability which describes design for successfully scalability from the ground up.

OTN Interview about MySQL

by

I was interviewed by Justin Kestelyn the OTN Senior Director about MySQL at Oracle Open World this week.

Some highlights of the questions asked:

  • 0:55 Since the close of acquisition has there been any change in direction?
  • 2:23 How have your clients and customers responded to the acquisition?
  • 3:53 You mentioned that Oracle will bring added advantages and could infuse invocation.
  • 5:15 InnoDB and MySQL are now both owned by Oracle. What do you see as the development advantages?
  • 6:47 What were your thoughts on the first MySQL Sunday?
  • 8:58 Forks?
  • 11:04 Contact Details

You can view the Video online or play below.

Successful MySQL Scalability Presentation

by

Last night I was the invited guest at the SF MySQL Meetup. In my presentation “Successful MySQL Scalability” I talked about a set of principles to ensure appropriate system architecture, data availability and best practices to build an ideal solution for your business. The presentation was also live streamed and is available online.

Successful MySQL Scalability

MySQL South America tour

by

DISCLAIMER: This post contains no technical MySQL content however it is good news for the MySQL Community.

MySQL content will be included for the first time with the LAOUC (Latin American Oracle Usergroups Council) Oracle tour that is being organized in conjunction with OTN (Oracle Technology Network).

I have no idea what MySQL user communities are in South America however if you live in any of the following cities, please feel free to contact me. I am happy to have additional discussion regarding MySQL or help in some way if there is interest in any cities.

This seven country tour includes:

  • Oct 12 – Lima, Peru
  • Oct 14 – Santiago, Chile
  • Oct 16 – Montevideo, Uruguay
  • Oct 18 – São Paulo, Brazil
  • Oct 20 – Bogota, Colombia
  • Oct 22 – Quito, Ecuador
  • Oct 25 – San Jose, Costa Rica

More details on the specific locations in each city will be available when finalized.

I would be very happy if anybody wants to translate this to Spanish or Portuguese for readers in South America.


View OTN Latin America in a larger map

2011 MySQL Conferences

by

Next year will mark a significant change for the MySQL community. At least three major conferences will have dedicated MySQL content that is great for attendees getting the best information on how to use MySQL from the experts in the field.

O’Reilly MySQL Conference & Expo

The 9th Annual MySQL conference will be held at is usual home of recent years. Colin will again be back as committee chair for a 3rd year and this will be my 6th straight MySQL conference.

Date: April 11 – 14, 2011
Location: Hyatt Regency, Santa Clara, California
Website: There is no website at this time
Call for Papers: There are no details for call for papers
Program Chairs: Colin Charles from Monty Program AB and Brian Aker.

Collaborate 11

Collaborate is a larger conference (4,000-5,000 attendees) that is actually three separate conferences in one run by the IOUG, OAUG and Quest. The IOUG content is generally a focus for Oracle DBA’s. Last year marked the first year with any MySQL sessions, and this year Collaborate will have dedicated MySQL tracks chaired by fellow ACE director Sheeri Cabral who is well known for her work in the MySQL community.

Date: April 10 – 14, 2011
Location: Orange County Convention Center West, Orlando, Florida
Website: http://collaborate11.ioug.org/
Call for Papers: Now open. Closes Friday October 1, 2010
Program Chair: Sheeri Cabral

KScope 11

ODTUG Kaleidoscope (Kscope for short) is a conference (1500 attendees) that is very focused on delivering the best content from the top community contributors for the communities benefit. 2010 was my first Kaleidoscope conference and I felt completely at home. Great people, great events and the best conference food I’ve had in many years.

With a dedicated MySQL track in 2010 for the first time I will again be the MySQL Program Chair in 2011 with an extended format for the MySQL developer and DBA. The focus will be the best way to develop successful applications with MySQL and will include Architecture, Performance Tuning, Best Practices, Case Studies and Hands-On streams.

Date: June 26 – 30, 2011
Website: http://kscope11.com
Location: Long Beach, California
Call for Papers: Closes Tuesday October 26, 2010
Program Chair: Ronald Bradford – Independent Consultant

Recap

2010 is also not over. MySQL Sunday at OOW promises to be a great event in San Francisco in under 2 weeks. You can still register at a very cheap price of $75 for 4 dedicated tracks of MySQL content. Open SQL Camp being organized also by Sheeri in Boston in October will continue the tradition of a small but focused and free event for the MySQL community.

The case against using rpm packaging for MySQL

by

In some environments using a distro package management system may* provide benefits including handling dependencies and providing a simpler approach when there are no dedicated DBA or SA resources.

However, the incorrect use can result in pain and in this instance production downtime. Even with dedicated resources at an unnamed premium managed hosting provider, the simple mistake of assumption resulted in over 30 minutes of unplanned downtime during peak time.

One of the disadvantages of using a system such as rpm is the lack of control in managing the starting and stopping of your MySQL instance, and the second is unanticipated package dependency upgrades.

So what happened with this client. When attempting to use the MySQL client on the production server, I got the following error.

$ mysql -uxxx -p
error while loading shared libraries: libmysqlclient.so.10: cannot open shared object file: No such file or directory

The server was running MySQL 5.0.27 via an rpm install.

$ rpm -qa | grep -i mysql
MySQL-server-standard-5.0.27-0.rhel3
MySQL-shared-standard-5.0.27-0.rhel3
MySQL-devel-standard-5.0.27-0.rhel3
MySQL-shared-compat-5.0.27-0.rhel3
MySQL-client-standard-5.0.27-0.rhel3

With no access to this managed server the information was relayed to the hosting provider and some time later we found the production website down. Some 30 minutes later we found that to fix the rpm problem, a dependency upgrade has also caused an automatic upgrade from 5.0.27 to 5.0.88.

While upgrading is not necessarily a bad thing, the lack of planning including a backup, a scheduled window of downtime and any level of testing is simply a poor cowboy approach to DBA management.

Upcoming MySQL Conferences

by

Unlike previous years when the number of conferences with MySQL content diminishes after the O’Reilly MySQL and OSCON conferences (Open SQL Camp excluded), this year has a lot on offer.

This month:

Upcoming next month in September:

  • MySQL Sunday at Oracle Open World on September 18 in San Francisco includes 4 tracks and around 15 quality speakers. (Big numbers of attendees also rumored but yet unconfirmed).
  • The inaugural Surge Scalability conference in Baltimore will include presentations by myself and Baron Schwartz (Percona being sponsors) as well as talks from other popular sites using MySQL.

If your in SF for the MySQL Sunday you may also want to come for the SF MySQL Meetup on the preceeding Thursday night where I’ll be giving my talk on “Common MySQL Scalability problems, and how to fix them”.

In October:

  • Open SQL Camp in Boston from Friday, Oct 15th in the evening, ending Sunday Oct 17th

Europeans will be busy in November where you will find dedicated MySQL tracks with multiple speakers at DOAG and UKOUG. Other MySQL talks can be found at SAPO Codebits 2010 and BGOUG.

And for South America, stay tuned. October will be your month!

There is also a great event calendar maintained by the MySQL community team on the Forge.

Why GRANT ALL is bad

by

A common observation for many LAMP stack products is the use of poor MySQL security practices. Even for more established products such as WordPress don’t always assume that the provided documentation does what it best for you. As per my earlier posts where I detailed installation instructions and optimal permissions for both WordPress and Mediawiki, and not just directed readers to online documentation.

In this post I will detail why GRANT ALL is bad.

Let’s start with what GRANT ALL [PRIVILEGES] provides. As per the MySQL 5.1 Reference Manual you get the following:

ALTER, ALTER ROUTINE, CREATE, CREATE ROUTINE, CREATE TEMPORARY TABLES, CREATE USER, CREATE VIEW, DELETE, DROP, EVENT, EXECUTE, FILE, INDEX, INSERT, LOCK TABLES, PROCESS, REFERENCES, RELOAD, REPLICATION CLIENT, REPLICATION SLAVE, SELECT, SHOW DATABASES, SHOW VIEW, SHUTDOWN, SUPER, TRIGGER, UPDATE, USAGE

I am going to focus on just one privilege that is included with ALL, and that is SUPER. This privilege can do the following which can be destructive for an application level user:

  • Bypasses read_only
  • Bypasses init_connect
  • Can Disable binary logging
  • Change configuration dynamically
  • No reserved connection

User Permissions

This is how a user should be created, granting only the required permissions to a given schema.

CREATE USER goodguy@localhost IDENTIFIED BY 'sakila';
GRANT CREATE,SELECT,INSERT,UPDATE,DELETE ON odtug.* TO goodguy@localhost;

This is what is commonly seen.

CREATE USER superman@'%';
GRANT ALL ON *.* TO superman@'%';

Bypasses read_only

Many MySQL replication environments rely on ensuring the MySQL slave is consistent with the master. Did you know that an application can bypass this security when read_only=true is used?

$ mysql -ugoodguy -psakila odtug
mysql> insert into test1(id) values(1);
ERROR 1290 (HY000): The MySQL server is running with the --read-only option so it cannot execute this statement

$ mysql -usuperman odtug
mysql> insert into test1(id) values(1);
Query OK, 1 row affected (0.01 sec)

GRANT ALL is bad for data consistency.

Bybasses init_connect

A common practices used for UTF8 communications is to use the init_connect configuration variable.

#my.cnf
[client]
init_connect=SET NAMES utf8

$ mysql -ugoodguy -psakila odtug

mysql> SHOW SESSION VARIABLES LIKE 'ch%';
+--------------------------+----------+
| Variable_name            | Value    |
+--------------------------+----------+
| character_set_client     | utf8     |
| character_set_connection | utf8     |
| character_set_database   | latin1   |
| character_set_filesystem | binary   |
| character_set_results    | utf8     |
| character_set_server     | latin1   |
| character_set_system     | utf8     |
+--------------------------+----------+

$ mysql -usuperman odtug

mysql> SHOW SESSION VARIABLES LIKE 'character%';
+--------------------------+----------+
| Variable_name            | Value    |
+--------------------------+----------+
| character_set_client     | latin1   |
| character_set_connection | latin1   |
| character_set_database   | latin1   |
| character_set_filesystem | binary   |
| character_set_results    | latin1   |
| character_set_server     | latin1   |
| character_set_system     | utf8     |
+--------------------------+----------+

GRANT ALL is bad for data integrity.

Disables Binary Logging.

$ mysql -usuperman odtug

mysql> SHOW MASTER STATUS;
+-------------------+----------+--------------+------------------+
| File              | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-------------------+----------+--------------+------------------+
| binary-log.000001 |      354 |              |                  |
+-------------------+----------+--------------+------------------+

mysql> DROP TABLE time_zone_leap_second;
mysql> SET SQL_LOG_BIN=0;
mysql> DROP TABLE time_zone_name;
mysql> SET SQL_LOG_BIN=1;
mysql> DROP TABLE time_zone_transition;
mysql> SHOW MASTER STATUS;
+-------------------+----------+--------------+------------------+
| File              | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-------------------+----------+--------------+------------------+
| binary-log.000001 |      674 |              |                  |
+-------------------+----------+--------------+------------------+

$ mysqlbinlog binary-log.000001 --start-position=354 --stop-position=674

# at 354
#100604 18:00:08 server id 1  end_log_pos 450 Query thread_id=1 exec_time=0 error_code=0
use mysql/*!*/;
SET TIMESTAMP=1275688808/*!*/;
DROP TABLE time_zone_leap_second
/*!*/;
# at 579
#100604 18:04:31 server id 1  end_log_pos 674 Query thread_id=2 exec_time=0 error_code=0
use mysql/*!*/;
SET TIMESTAMP=1275689071/*!*/;
DROP TABLE time_zone_transition
/*!*/;
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;

Should that statement be run on MySQL Slaves?
Is the binary log used for any level of auditing?

GRANT ALL is bad for slave consistency.

The reserved connection

MySQL reserved one connection for an administrator to be able to login to a server. For example.

$ mysql -uroot

mysql> show global variables like 'max_connections';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 3     |
+-----------------+-------+
1 row in set (0.07 sec)

mysql> show global status like 'threads_connected';
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| Threads_connected | 4     |
+-------------------+-------+

mysql> SHOW PROCESSLIST;
+----+------+-----------+-------+---------+------+------------+---------------
| Id | User | Host      | db    | Command | Time | State      | Info
+----+------+-----------+-------+---------+------+------------+---------------
| 13 | good | localhost | odtug | Query   |  144 | User sleep | UPDATE test1 ...
| 14 | good | localhost | odtug | Query   |  116 | Locked     | select * from test1
| 15 | good | localhost | odtug | Query   |   89 | Locked     | select * from test1
| 15 | root | localhost | odtug | Query   |   89 | Locked     | SHOW PROCESSLIST

However if all application users are already using the SUPER privilege, the administrator will get.

$ mysql -uroot
ERROR 1040 (HY000): Too many connections

There is no way to be able to login and see what’s happening, or kill threads for example. In this case you either wait, or you are required to kill the mysqld process, or fine the client threads to kill. The result of the former may lead to a corrupt database requiring additional recovery.

GRANT ALL is bad for system administration and monitoring.

Conclusion

Don’t use GRANT ALL for application users. For more information, including why I only listed just 5 issues, check out my MySQL Idiosyncrasies that BITE presentation. I will also be presenting this talk at MySQL Sunday at Oracle Open World in September.

You have to love the Planet MySQL voting system

by

Within a few hours my post Installing Mediawiki on Oracle Enterprise Linux LAMP stack got 5 negative votes.

Wow, I’d be glad if these people could felt so passionately about all the other CRUD on Planet MySQL that has ZERO to do actually do with MySQL.

Using a LAMP product, and providing instructions for operation can’t be a negative voting offense. So it can only be the words “Oracle Enterprise Linux”.

For those negative people out there that care enough to physically mark blogs let me share some facts with you. RedHat Enterprise Linux (RHEL) is the most widely used and support platform for production MySQL environments. CentOS and Oracle Enterprise Linux (OEL) provide via the freedom of Open Source, their own offerings of RHEL with various other features including support or specific additional features for other products. Oracle also provides and commits work to the Linux Kernel, so they are just as much an allies to open source as other companies.

So are you bagging the operating system or just the word Oracle. Oracle is hear to stay, in fact those attending the MySQL Sunday event at Oracle Open World may get a welcome wake up shock. And for reference these articles are being written for a published Oracle Magazine article which meets the comfort level of the reader and introduces MySQL in a positive way. Your actions show just how much you are unwilling to embrace the larger community.

Installing Mediawiki on Oracle Enterprise Linux LAMP stack

by

A company wiki can be easily configured in under 10 minutes using Mediawiki the open source LAMP software that powers the top 10 website Wikipedia.

A company wiki is an ideal means for a centralized and user contributed documentation system. The following steps show you how to download, configure and get your Mediawiki site operational.

Software Pre-Requisites

Software Installation

su -
cd /tmp
wget http://download.wikimedia.org/mediawiki/1.16/mediawiki-1.16.0.tar.gz
cd /var/www/html
tar xfz /tmp/mediawiki*.tar.gz
mv mediawiki* wiki
chmod 777 wiki/config

NOTE: You should check the Mediawiki Downloads page for the latest version.

You can now visit http://localhost/wiki and you will be presented with a message of an un-configured Mediawiki environment. You can streamline the MySQL portion of this configuration with the following commands.

mysql -uroot -p -e "DROP SCHEMA IF EXISTS wikidb;CREATE SCHEMA wikidb"
mysql -uroot -p -e "CREATE USER wikiuser @localhost IDENTIFIED BY 'sakila'"
mysql -uroot -p -e "GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,INDEX on wikidb.* TO wikiuser@localhost"

You can now complete the configuration on the Configuration Page. You will need to enter the following information.

  • Wiki name = Example Wiki
  • Contact e-mail = your email address
  • Admin user password = enter password
  • DB password = sakila

After successful installation, one additional step is needed.

mv wiki/config/LocalSettings.php wiki/
chmod 400 wiki/LocalSettings.php
chmod 500 wiki/config

You can now start using your new Wiki at http://localhost/wiki. You will find documentation at the links provided on the displayed home page and also at www.mediawiki.org.

MySQL Structures

Looking at the tables that are created by the installation process:

$ mysql -uwikiuser -psakila blog

mysql> show tables;
+-------------------+
| Tables_in_wikidb  |
+-------------------+
| archive           |
| category          |
| categorylinks     |
| change_tag        |
| external_user     |
| externallinks     |
| filearchive       |
| hitcounter        |
| image             |
| imagelinks        |
| interwiki         |
| ipblocks          |
| job               |
| l10n_cache        |
| langlinks         |
| log_search        |
| logging           |
| math              |
| objectcache       |
| oldimage          |
| page              |
| page_props        |
| page_restrictions |
| pagelinks         |
| protected_titles  |
| querycache        |
| querycache_info   |
| querycachetwo     |
| recentchanges     |
| redirect          |
| revision          |
| searchindex       |
| site_stats        |
| tag_summary       |
| templatelinks     |
| text              |
| trackbacks        |
| transcache        |
| updatelog         |
| user              |
| user_groups       |
| user_newtalk      |
| user_properties   |
| valid_tag         |
| watchlist         |
+-------------------+
45 rows in set (0.00 sec)

mysql> SELECT table_name,engine,table_rows FROM information_schema.tables WHERE table_schema = 'wikidb';
+-------------------+--------+------------+
| table_name        | engine | table_rows |
+-------------------+--------+------------+
| archive           | InnoDB |          0 |
| category          | InnoDB |          0 |
| categorylinks     | InnoDB |          0 |
| change_tag        | InnoDB |          0 |
| external_user     | InnoDB |          0 |
| externallinks     | InnoDB |          0 |
| filearchive       | InnoDB |          0 |
| hitcounter        | MEMORY |          0 |
| image             | InnoDB |          0 |
| imagelinks        | InnoDB |          0 |
| interwiki         | InnoDB |         95 |
| ipblocks          | InnoDB |          0 |
| job               | InnoDB |          0 |
| l10n_cache        | InnoDB |       3686 |
| langlinks         | InnoDB |          0 |
| log_search        | InnoDB |          0 |
| logging           | InnoDB |          0 |
| math              | InnoDB |          0 |
| objectcache       | InnoDB |          2 |
| oldimage          | InnoDB |          0 |
| page              | InnoDB |          1 |
| page_props        | InnoDB |          0 |
| page_restrictions | InnoDB |          0 |
| pagelinks         | InnoDB |          0 |
| protected_titles  | InnoDB |          0 |
| querycache        | InnoDB |          0 |
| querycache_info   | InnoDB |          0 |
| querycachetwo     | InnoDB |          0 |
| recentchanges     | InnoDB |          0 |
| redirect          | InnoDB |          0 |
| revision          | InnoDB |          1 |
| searchindex       | MyISAM |          0 |
| site_stats        | InnoDB |          1 |
| tag_summary       | InnoDB |          0 |
| templatelinks     | InnoDB |          0 |
| text              | InnoDB |          1 |
| trackbacks        | InnoDB |          0 |
| transcache        | InnoDB |          0 |
| updatelog         | InnoDB |          0 |
| user              | InnoDB |          1 |
| user_groups       | InnoDB |          2 |
| user_newtalk      | InnoDB |          0 |
| user_properties   | InnoDB |          0 |
| valid_tag         | InnoDB |          0 |
| watchlist         | InnoDB |          0 |
+-------------------+--------+------------+
45 rows in set (0.01 sec)

Sort URL for this post rb42.com/oel-install-mediawiki

Will Oracle kill MySQL?

by

I get asked this question often. It was mentioned again recently in a NYTECH executive breakfast with RedHat CIO Lee Congdon.

The short answer is No.

There is clear evidence that in the short to medium term Oracle will continue to promote and enhance MySQL. Some of these indicators include:

It is clear from these sources that Oracle intends to incorporate MySQL into Oracle Backup and Security Vault products. Both a practical and necessary step. There is also a clear mention of focusing on the Microsoft platform, a clear indicator that SQL Server is in their sights without actually saying it.

What is unknown is exact how and when features will be implemented. Also important is how much these may cost the end user. Oracle is in the business of selling, now an entire H/W and S/W stack. They also have a complicated pricing model of different components with product offerings. I assume this will continue. There are already two indications, InnoDBbackup included for Enterprise Backup (from April Keynote) and 5.1 enterprise split. (Note: while this split may have existed prior to Oracle, it is now more clearly obvious).

MySQL can never be seen as drawing away from any Oracle sales of the core entry level database product. It is likely Oracle will provide a SQL Syntax compatibility layer for MySQL within 2 years, however it will I’m sure be a commercial add-on. Likewise, I would suspect a PL/SQL lite layer within 5 years, but again at a significant cost to offset the potential loss of sales in the low end of the server market. There continues to be active development in the MySQL Enterprise Monitor, MySQL Workbench and MySQL Connectors which is all excellent news for users.

Moving forward, how long will this ancillary development of free tools continue? What will happen to the commercial storage engine, OEM and licensing model after the 5 year commitment? How will the MySQL ecosystem survive.? There is active development in Percona, MariaDB and Drizzle forks, however unless all players that want to provide a close MySQL compatible solution work together, progress will continue to be a disappointing disjointed approach. The 2011 conference season will also see a clear line with competing MySQL conferences in April scheduled at the same time, the O’Reilly MySQL conference in Santa Clara California and the Oracle supported(*) Collaborate 2011 in Orlando, Florida.

I have a number of predictions on what Oracle ME MySQL may look like in 5 years however this is a topic for a personal discussion.

Speaking at Surge Scalability 2010 – Baltimore, MD

by

I will be joining a great list of quality speakers including John Allspaw, Theo Schlossnagle, Rasmus Lerdorf and Tom Cook at Surge 2010 in Baltimore, Maryland on Thu 30 Sep, and Fri Oct 1st 2010. Surge 2010 Speaker - Baltimore, MD

My presentation on “The most common MySQL scalability mistakes, and how to avoid them.” will include discussing various experiences observed in the field as a MySQL Consultant and MySQL Performance Tuning expert.

Abstract:

The most common mistakes are easy to avoid however many startups continue to fall prey, with the impact including large re-design costs, delays in new feature releases, lower staff productivity and less then ideal ROI. All growing and successful sites need to achieve higher Availability, seamless Scalability and proven Resilience. Know the right MySQL environments to provide a suitable architecture and application design to support these essential needs.

Overview:

Some details of the presentation would include:

  • The different types of accessible data  (e.g. R/W, R, none)
  • What limits MySQL availability (e.g software upgrades, blocking statements, locking etc)
  • The three components of scalability – Read Scalability/Write Scalability/Caching
  • Design practices for increasing scalability and not physical resources
  • Disaster is inevitable. Having a tested and functional failover strategy
  • When other products are better (e.g. Static files, Session management via Key/Value store)
  • What a lack of accurate monitoring causes
  • What a lack of breakability testing causes
  • What does “No Downtime” mean to your organization.
  • Implementing a successful “failed whale” approach with preemptive analysis
  • Identifying when MySQL is not your bottleneck

If you missed MySQL Idiosyncrasies that BITE

by

I recently gave a webinar to the LAOUC and NZOUG user groups on MySQL Idiosyncrasies that BITE.

For the benefit of many viewers that do not use English as a first language my slides include some additional information from my ODTUG Kaleidoscope presentation in June.

Thanks to Francisco Munoz Alvarez for organizing.

MySQL Idiosyncrasies That Bite 2010.07

3 webinars on Upgrading MySQL

by

The IOUG Online Education Series: Get Real with Upgrades will include next week 3 different MySQL webinars. These are:

  • MySQL 5.1: Why and How to Upgrade by Sheeri Cabral on Tuesday, July 27, 12:00 p.m. – 1:00 p.m. CT
  • MySQL Upgrades With No Downtime by Sean Hull on Wednesday, July 28, 12:00 p.m. – 1:00 p.m. CT
  • MySQL Upgrade Best Practices by Matt Yonkovit on Thursday, July 29, 12:00 p.m. – 1:00 p.m. CT

Each speaker will be covering different areas and reasons for considering and implementing a successful MySQL Upgrade.

Installing WordPress on Oracle Enterprise Linux LAMP stack

by

A company blog can be easily configured in under 10 minutes using WordPress, a popular open source LAMP product that runs a reported 12+ million blogs including those found at CNN, NY Times, Wall Street Journal (WSJ), ZDNet, MTV, People Magazine, Playstation and eBay.

A company blog is a great way for the dissemination of information to your user base as well as enabling a means of user feedback via comments.

The following steps show you how to download, configure and get your WordPress blog operational.

Software Pre-Requisites

Software Installation

su -
cd /tmp
wget  http://wordpress.org/latest.tar.gz
cd /var/www/html
tar xfz /tmp/latest.tar.gz
mv wordpress blog

You can now visit http://localhost/blog and you will be presented with a message of an un-configured WordPress environment. You can streamline the MySQL portion of this configuration with the following commands.

cd blog
sed -e "s/database_name_here/blog/;s/username_here/blog_user/;s/password_here/sakila/" wp-config-sample.php > wp-config.php
mysql -uroot -p -e "CREATE SCHEMA blog"
mysql -uroot -p -e "CREATE USER blog_user @localhost IDENTIFIED BY 'sakila'"
mysql -uroot -p -e "GRANT SELECT,INSERT,UPDATE,DELETE,CREATE on blog.* TO blog_user@localhost"

Returning now to http://localhost/blog you simply only need to specify a Title, password and email address, click Save and your Blog at http://localhost/blog is complete and operational.



MySQL Structures

Looking at the tables that are created by the installation process:

$ mysql -ublog_user -psakila blog

mysql> show tables;
+-----------------------+
| Tables_in_blog        |
+-----------------------+
| wp_commentmeta        |
| wp_comments           |
| wp_links              |
| wp_options            |
| wp_postmeta           |
| wp_posts              |
| wp_term_relationships |
| wp_term_taxonomy      |
| wp_terms              |
| wp_usermeta           |
| wp_users              |
+-----------------------+
11 rows in set (0.00 sec)

mysql> SELECT table_name,engine,table_rows FROM information_schema.tables WHERE table_schema = 'blog';
+-----------------------+--------+------------+
| table_name            | engine | table_rows |
+-----------------------+--------+------------+
| wp_commentmeta        | MyISAM |          0 |
| wp_comments           | MyISAM |          1 |
| wp_links              | MyISAM |          7 |
| wp_options            | MyISAM |        109 |
| wp_postmeta           | MyISAM |          1 |
| wp_posts              | MyISAM |          2 |
| wp_term_relationships | MyISAM |          8 |
| wp_term_taxonomy      | MyISAM |          2 |
| wp_terms              | MyISAM |          2 |
| wp_usermeta           | MyISAM |         13 |
| wp_users              | MyISAM |          1 |
+-----------------------+--------+------------+
11 rows in set (0.00 sec)

Additional References

Short URL for this post rb42.com/oel-install-wordpress

Reviewing your MySQL installation on Oracle Enterprise Linux

by

After successfully Installing MySQL, let us take a look at an operational MySQL instance on your Oracle Enterprise Linux server.

User Management

By default there will be a new mysql user and group created. This user is used to run the mysqld process is generally not used for any other purpose.

$ grep mysql /etc/{passwd,shadow,group}
/etc/passwd:mysql:x:27:27:MySQL Server:/var/lib/mysql:/bin/bash
/etc/shadow:mysql:!!:14796::::::
/etc/group:mysql:x:27:

Binaries

MySQL binaries are found in /usr/bin.

$ ls -l /usr/bin/mysql*
-rwxr-xr-x 1 root root  314568 Feb 16 17:45 /usr/bin/mysql
-rwxr-xr-x 1 root root  110776 Feb 16 14:39 /usr/bin/mysqlaccess
-rwxr-xr-x 1 root root   35144 Feb 16 17:45 /usr/bin/mysqladmin
-rwxr-xr-x 1 root root  112944 Feb 16 17:45 /usr/bin/mysqlbinlog
-rwxr-xr-x 1 root root    7632 Feb 16 17:45 /usr/bin/mysqlbug
-rwxr-xr-x 1 root root   30576 Feb 16 17:45 /usr/bin/mysqlcheck
-rwxr-xr-x 1 root root    7632 Feb 16 17:45 /usr/bin/mysql_config
-rwxr-xr-x 1 root root    3670 Feb 16 17:44 /usr/bin/mysql_convert_table_format
-rwxr-xr-x 1 root root   22522 Feb 16 17:44 /usr/bin/mysqld_multi
-rwxr-xr-x 1 root root   13073 Feb 16 17:44 /usr/bin/mysqld_safe
-rwxr-xr-x 1 root root   75184 Feb 16 17:45 /usr/bin/mysqldump
-rwxr-xr-x 1 root root    6356 Feb 16 17:44 /usr/bin/mysqldumpslow
-rwxr-xr-x 1 root root   11648 Feb 16 17:44 /usr/bin/mysql_explain_log
-rwxr-xr-x 1 root root    3245 Feb 16 14:39 /usr/bin/mysql_find_rows
-rwxr-xr-x 1 root root     483 Feb 16 17:44 /usr/bin/mysql_fix_extensions
-rwxr-xr-x 1 root root    5834 Feb 16 17:44 /usr/bin/mysql_fix_privilege_tables
-rwxr-xr-x 1 root root   31431 Feb 16 17:44 /usr/bin/mysqlhotcopy
-rwxr-xr-x 1 root root   26160 Feb 16 17:45 /usr/bin/mysqlimport
-rwxr-xr-x 1 root root   13659 Feb 16 17:44 /usr/bin/mysql_install_db
-rwxr-xr-x 1 root root    6586 Feb 16 17:44 /usr/bin/mysql_secure_installation
-rwxr-xr-x 1 root root   16687 Feb 16 17:44 /usr/bin/mysql_setpermission
-rwxr-xr-x 1 root root   28224 Feb 16 17:45 /usr/bin/mysqlshow
-rwxr-xr-x 1 root root   14473 Feb 16 14:39 /usr/bin/mysql_tableinfo
-rwxr-xr-x 1 root root  158192 Feb 16 17:45 /usr/bin/mysqltest
-rwxr-xr-x 1 root root   42360 Feb 16 17:45 /usr/bin/mysqltestmanager
-rwxr-xr-x 1 root root   15464 Feb 16 17:45 /usr/bin/mysqltestmanagerc
-rwxr-xr-x 1 root root   13448 Feb 16 17:45 /usr/bin/mysqltestmanager-pwgen
-rwxr-xr-x 1 root root 1312064 Feb 16 17:45 /usr/bin/mysql_tzinfo_to_sql
-rwxr-xr-x 1 root root   54160 Feb 16 17:45 /usr/bin/mysql_upgrade
-rwxr-xr-x 1 root root    5753 Feb 16 17:44 /usr/bin/mysql_upgrade_shell
-rwxr-xr-x 1 root root  112136 Feb 16 17:45 /usr/bin/mysql_waitpid
-rwxr-xr-x 1 root root    3818 Feb 16 17:44 /usr/bin/mysql_zap

The mysqld binary is found in /usr/libexec

Error Log

The MySQL error log is found in /var/log/mysqld.log

The content after an initial start of MySQL will look similar to:

cat /var/log/mysqld.log
100705 22:09:03  mysqld started
InnoDB: The first specified data file ./ibdata1 did not exist:
InnoDB: a new database to be created!
100705 22:09:03  InnoDB: Setting file ./ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
100705 22:09:03  InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 5 MB
InnoDB: Database physically writes the file full: wait...
100705 22:09:03  InnoDB: Log file ./ib_logfile1 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile1 size to 5 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
100705 22:09:03  InnoDB: Started; log sequence number 0 0
100705 22:09:03 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution

On the first invocation of MySQL, the InnoDB storage engine will create a default tablespace and redo logs. This is the majority of messages in the above log.

Processes

MySQL is a multi-threaded single process called mysqld. A second wrapper process mysqld_safe is generally found. This process logs stderr and also will restart the mysqld process if not found.

ps -ef | grep mysql
root     14733     1  0 Jul05 pts/1    00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --user=mysql
mysql    14783 14733  0 Jul05 pts/1    00:00:10 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --socket=/var/lib/mysql/mysql.sock

Memory Usage

MySQL can have a very low memory footprint. By default the mysqld process has a 175M virtual size.

$ ps -eopid,fname,rss,vsz,user,command | grep -e "RSS" -e "mysql"
  PID COMMAND    RSS    VSZ USER     COMMAND
14275 grep       720  61136 root     grep -e RSS -e mysql
14733 mysqld_s  1192  63820 root     /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --user=mysql
14783 mysqld   27004 179496 mysql    /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --socket=/var/lib/mysql/mysql.sock

Disk Usage

The MySQL data files will be stored on a default installation in /var/lib/mysql

$ du -sh /var/lib/mysql
22M     /var/lib/mysql

$ ls -ld /var/lib/mysql
drwxr-xr-x 4 mysql mysql 4096 Jul 13 11:50 /var/lib/mysql

$ ls -l /var/lib/mysql
total 20552
-rw-rw---- 1 mysql mysql 10485760 Jul  5 22:09 ibdata1
-rw-rw---- 1 mysql mysql  5242880 Jul  5 22:09 ib_logfile0
-rw-rw---- 1 mysql mysql  5242880 Jul  5 22:09 ib_logfile1
drwx------ 2 mysql mysql     4096 Jul  5 22:09 mysql
srwxrwxrwx 1 mysql mysql        0 Jul  5 22:09 mysql.sock

The MySQL data directory includes the InnoDB tablespace datafile (ibdata1), redo logs (ib_logfile?), and the mysql directory corresponding to the mysql schema containing instance meta data.

This directory also contains the socket file, which is actually a poor location as this opens the security of this directory for world access. This will be discussed later in securing your installation.

Running MySQL

The best means of controlling the starting and stopping of mysql is to use the provided service init script mysqld

$ ls -l /etc/init.d/mysqld
-rwxr-xr-x 1 root root 4286 Feb 16 17:45 /etc/init.d/mysqld

Configuration

For OEL the MySQL configuration can be found in /etc.
NOTE: MySQL can use multiple configuration files.

$ ls -l /etc/my.cnf
-rw-r--r-- 1 root root 441 Feb 16 14:39 /etc/my.cnf

MySQL includes a minimalistic configuration file by default. The configuration file format is variable=value pairs for a given number of different sections, in this file [mysqld] and [mysqld_safe].

$ cat /etc/my.cnf

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1

# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links=0

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

Audit

A full audit of all MySQL related files.

find / -name "*mysql*"
/etc/rc.d/rc3.d/S64mysqld
/etc/rc.d/rc5.d/S64mysqld
/etc/rc.d/rc6.d/K36mysqld
/etc/rc.d/init.d/mysqld
/etc/rc.d/rc0.d/K36mysqld
/etc/rc.d/rc4.d/S64mysqld
/etc/rc.d/rc1.d/K36mysqld
/etc/rc.d/rc2.d/S64mysqld
/etc/php.d/pdo_mysql.ini
/etc/php.d/mysql.ini
/etc/php.d/mysqli.ini
/etc/ld.so.conf.d/mysql-x86_64.conf
/etc/ld.so.conf.d/mysql-i386.conf
/usr/lib64/mysql
/usr/lib64/mysql/mysqlbug
/usr/lib64/mysql/libmysqlclient_r.so.15.0.0
/usr/lib64/mysql/libmysqlclient.so.15
/usr/lib64/mysql/libmysqlclient_r.so.15
/usr/lib64/mysql/mysql_config
/usr/lib64/mysql/libmysqlclient.so.15.0.0
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/Bundle/DBD/mysql.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/DBD/mysql
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/DBD/mysql/mysql.so
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/DBD/mysql
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/DBD/mysql.pm
/usr/lib64/php/modules/mysql.so
/usr/lib64/php/modules/pdo_mysql.so
/usr/lib64/php/modules/mysqli.so
/usr/libexec/mysqld
/usr/libexec/mysqlmanager
/usr/share/mysql
/usr/share/mysql/mysql_system_tables.sql
/usr/share/mysql/mysql_system_tables_data.sql
/usr/share/mysql/mysql_fix_privilege_tables.sql
/usr/share/mysql/mysql_test_data_timezone.sql
/usr/share/vim/vim70/syntax/mysql.vim
/usr/share/man/man8/mysqld.8.gz
/usr/share/man/man8/mysqlmanager.8.gz
/usr/share/man/man1/mysql.1.gz
/usr/share/man/man1/mysql.server.1.gz
/usr/share/man/man1/mysql_tableinfo.1.gz
/usr/share/man/man1/mysql_upgrade.1.gz
/usr/share/man/man1/mysqlaccess.1.gz
/usr/share/man/man1/mysql_waitpid.1.gz
/usr/share/man/man1/mysql_fix_extensions.1.gz
/usr/share/man/man1/mysqlman.1.gz
/usr/share/man/man1/mysqlbinlog.1.gz
/usr/share/man/man1/mysql_install_db.1.gz
/usr/share/man/man1/mysql_tzinfo_to_sql.1.gz
/usr/share/man/man1/mysql_secure_installation.1.gz
/usr/share/man/man1/mysqld_safe.1.gz
/usr/share/man/man1/mysqladmin.1.gz
/usr/share/man/man1/mysqlimport.1.gz
/usr/share/man/man1/mysql_zap.1.gz
/usr/share/man/man1/msql2mysql.1.gz
/usr/share/man/man1/mysqlshow.1.gz
/usr/share/man/man1/mysqldump.1.gz
/usr/share/man/man1/safe_mysqld.1.gz
/usr/share/man/man1/mysql_explain_log.1.gz
/usr/share/man/man1/mysql_config.1.gz
/usr/share/man/man1/mysqlbug.1.gz
/usr/share/man/man1/mysqld_multi.1.gz
/usr/share/man/man1/mysql_setpermission.1.gz
/usr/share/man/man1/mysqlhotcopy.1.gz
/usr/share/man/man1/mysql_find_rows.1.gz
/usr/share/man/man1/mysql_convert_table_format.1.gz
/usr/share/man/man1/mysql_fix_privilege_tables.1.gz
/usr/share/man/man1/mysqldumpslow.1.gz
/usr/share/man/man1/mysqltest.1.gz
/usr/share/man/man1/mysqlcheck.1.gz
/usr/share/man/man3/Bundle::DBD::mysql.3pm.gz
/usr/share/man/man3/DBD::mysql.3pm.gz
/usr/share/man/man3/DBD::mysql::INSTALL.3pm.gz
/usr/share/doc/mysql-server-5.0.77
/usr/share/doc/mysql-5.0.77
/usr/share/doc/selinux-policy-2.4.6/html/services_mysql.html
/usr/share/pixmaps/comps/mysql.png
/usr/share/info/mysql.info.gz
/usr/share/selinux/devel/include/services/mysql.if
/usr/bin/mysql_fix_extensions
/usr/bin/mysql
/usr/bin/mysqltestmanager
/usr/bin/mysqldumpslow
/usr/bin/mysql_upgrade_shell
/usr/bin/mysql_convert_table_format
/usr/bin/mysqlimport
/usr/bin/mysqldump
/usr/bin/mysqltestmanager-pwgen
/usr/bin/mysql_tzinfo_to_sql
/usr/bin/mysqlbug
/usr/bin/mysqlhotcopy
/usr/bin/mysqlaccess
/usr/bin/mysqltest
/usr/bin/mysqladmin
/usr/bin/mysql_upgrade
/usr/bin/mysqltestmanagerc
/usr/bin/mysqld_safe
/usr/bin/mysql_zap
/usr/bin/mysql_waitpid
/usr/bin/msql2mysql
/usr/bin/mysql_secure_installation
/usr/bin/mysql_fix_privilege_tables
/usr/bin/mysqlshow
/usr/bin/mysql_config
/usr/bin/mysql_setpermission
/usr/bin/mysql_tableinfo
/usr/bin/mysql_find_rows
/usr/bin/mysqld_multi
/usr/bin/mysqlcheck
/usr/bin/mysqlbinlog
/usr/bin/mysql_install_db
/usr/bin/mysql_explain_log
/usr/lib/mysql
/usr/lib/mysql/mysqlbug
/usr/lib/mysql/libmysqlclient_r.so.15.0.0
/usr/lib/mysql/libmysqlclient.so.15
/usr/lib/mysql/libmysqlclient_r.so.15
/usr/lib/mysql/mysql_config
/usr/lib/mysql/libmysqlclient.so.15.0.0
/usr/lib/python2.4/site-packages/sos/plugins/mysql.pyo
/usr/lib/python2.4/site-packages/sos/plugins/mysql.pyc
/usr/lib/python2.4/site-packages/sos/plugins/mysql.py
/var/log/mysqld.log
/var/run/mysqld
/var/run/mysqld/mysqld.pid
/var/lock/subsys/mysqld
/var/lib/mysql
/var/lib/mysql/mysql
/var/lib/mysql/mysql.sock
/root/.mysql_history
/selinux/booleans/mysqld_disable_trans
/selinux/booleans/allow_user_mysql_connect

Installing a LAMP stack on Oracle Enterprise Linux

by

After successfully installing MySQL on Oracle Enterprise Linux installing a LAMP (Linux/Apache/MySQL/PHP) stack can also be performed with a single command:

$ yum install -y httpd php php-mysql
# Start the Apache Httpd Process
$ /etc/init.d/httpd start

To test and confirm Apache Httpd and PHP, we can use the CLI browser lynx:

$ yum install -y lynx
$ echo "<? phpinfo() ?>" > /var/www/html/phpinfo.php
$ lynx http://localhost/phpinfo.php

If successful, you will find a web page that contains the following.

phpinfo() (p1 of 31)

   PHP Logo

PHP Version 5.1.6

   System Linux localhost.localdomain 2.6.18-164.el5 #1 SMP Thu Sep 3 04:15:13
   EDT 2009 x86_64
   Build Date Feb 11 2010 19:07:36
   Configure   Command   './configure'  '--build=x86_64-redhat-linux-gnu'
   '--host=x86_64-redhat-linux-gnu'    '--target=x86_64-redhat-linux-gnu'
   '--program-prefix=''--prefix=/usr''--exec-prefix=/usr''--bindir=/usr/bin'
   '--sbindir=/usr/sbin'    '--sysconfdir=/etc'    '--datadir=/usr/share'
   '--includedir=/usr/include'                      '--libdir=/usr/lib64'
   '--libexecdir=/usr/libexec'                     '--localstatedir=/var'
   '--sharedstatedir=/usr/com'                  '--mandir=/usr/share/man'
   '--infodir=/usr/share/info'             '--cache-file=../config.cache'
   '--with-libdir=lib64'                   '--with-config-file-path=/etc'

It is important to note that PHP is also a standalone scripting language that doesn’t require a web browser. You can use PHP on the command line, for example:

$ php --version
PHP 5.1.6 (cli) (built: Feb 11 2010 19:06:40)
Copyright (c) 1997-2006 The PHP Group
Zend Engine v2.1.0, Copyright (c) 1998-2006 Zend Technologies

$ echo "<?phpinfo()?>" | php | grep -i mysql
Configure Command =>  './configure' '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/usr/com' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--cache-file=../config.cache' '--with-libdir=lib64' '--with-config-file-path=/etc' '--with-config-file-scan-dir=/etc/php.d' '--disable-debug' '--with-pic' '--disable-rpath' '--without-pear' '--with-bz2' '--with-curl' '--with-exec-dir=/usr/bin' '--with-freetype-dir=/usr' '--with-png-dir=/usr' '--enable-gd-native-ttf' '--without-gdbm' '--with-gettext' '--with-gmp' '--with-iconv' '--with-jpeg-dir=/usr' '--with-openssl' '--with-png' '--with-pspell' '--with-expat-dir=/usr' '--with-pcre-regex=/usr' '--with-zlib' '--with-layout=GNU' '--enable-exif' '--enable-ftp' '--enable-magic-quotes' '--enable-sockets' '--enable-sysvsem' '--enable-sysvshm' '--enable-sysvmsg' '--enable-track-vars' '--enable-trans-sid' '--enable-yp' '--enable-wddx' '--with-kerberos' '--enable-ucd-snmp-hack' '--with-unixODBC=shared,/usr' '--enable-memory-limit' '--enable-shmop' '--enable-calendar' '--enable-dbx' '--enable-dio' '--with-mime-magic=/usr/share/file/magic.mime' '--without-sqlite' '--with-libxml-dir=/usr' '--with-xml' '--with-system-tzdata' '--enable-force-cgi-redirect' '--enable-pcntl' '--with-imap=shared' '--with-imap-ssl' '--enable-mbstring=shared' '--enable-mbstr-enc-trans' '--enable-mbregex' '--with-ncurses=shared' '--with-gd=shared' '--enable-bcmath=shared' '--enable-dba=shared' '--with-db4=/usr' '--with-xmlrpc=shared' '--with-ldap=shared' '--with-ldap-sasl' '--with-mysql=shared,/usr' '--with-mysqli=shared,/usr/lib64/mysql/mysql_config' '--enable-dom=shared' '--with-dom-xslt=/usr' '--with-dom-exslt=/usr' '--with-pgsql=shared' '--with-snmp=shared,/usr' '--enable-soap=shared' '--with-xsl=shared,/usr' '--enable-xmlreader=shared' '--enable-xmlwriter=shared' '--enable-fastcgi' '--enable-pdo=shared' '--with-pdo-odbc=shared,unixODBC,/usr' '--with-pdo-mysql=shared,/usr/lib64/mysql/mysql_config' '--with-pdo-pgsql=shared,/usr' '--with-pdo-sqlite=shared,/usr' '--enable-dbase=shared'
/etc/php.d/mysql.ini,
/etc/php.d/mysqli.ini,
/etc/php.d/pdo_mysql.ini,
mysql
MySQL Support => enabled
MYSQL_MODULE_TYPE => external
MYSQL_SOCKET => /var/lib/mysql/mysql.sock
MYSQL_INCLUDE => -I/usr/include/mysql
MYSQL_LIBS => -L/usr/lib64/mysql -lmysqlclient
mysql.allow_persistent => On => On
mysql.connect_timeout => 60 => 60
mysql.default_host => no value => no value
mysql.default_password => no value => no value
mysql.default_port => no value => no value
mysql.default_socket => no value => no value
mysql.default_user => no value => no value
mysql.max_links => Unlimited => Unlimited
mysql.max_persistent => Unlimited => Unlimited
mysql.trace_mode => Off => Off
mysqli
MysqlI Support => enabled
MYSQLI_SOCKET => /var/lib/mysql/mysql.sock
mysqli.default_host => no value => no value
mysqli.default_port => 3306 => 3306
mysqli.default_pw => no value => no value
mysqli.default_socket => no value => no value
mysqli.default_user => no value => no value
mysqli.max_links => Unlimited => Unlimited
mysqli.reconnect => Off => Off
PDO drivers => mysql, sqlite
pdo_mysql
PDO Driver for MySQL, client library version => 5.0.77

Short URL: rb42.com/oel-install-lamp

Installing MySQL on Oracle Enterprise Linux

by

One of the significant benefits of MySQL is it’s ease of use. Generally already installed on most Linux systems, MySQL can be installed by a single command if not yet present. On Oracle Enterprise Linux 5.4 you can use the following commands to check for MySQL, configure your yum repository and install MySQL.

# Check if already installed
$ rpm -qa | grep -i mysql

# Configure yum repository on new server
$ su -
$ cd /tmp
$ wget http://public-yum.oracle.com/public-yum-el5.repo
$ sed -e "s/enabled=0/enabled=1/" public-yum-el5.repo > /etc/yum.repos.d/public-yum-el5.repo

# Install MySQL
$ yum install -y mysql-server mysql

# Start and test MySQL Instance
$ /etc/init.d/mysqld start
$ mysql -uroot -e "SELECT VERSION()"

+-----------+
| VERSION() |
+-----------+
| 5.0.77    |
+-----------+

You can find more information about the Oracle public yum repository at http://public-yum.oracle.com You will also note that the version installed is 5.0. The current GA version of MySQL is 5.1, however Red Hat is notorious for taking time to update repositories more regularly. You can always use more current rpm files available at the MySQL downloads page.

If you want MySQL to be configured to automatically start and stop on your server you need to run the following command.

$ chkconfig mysqld on
$ chkconfig --list mysqld
mysqld          0:off   1:off   2:on    3:on    4:on    5:on    6:off

You can link to this post with the short url rb42.com/oel-install-mysql

Related Posts

Speaking at Oracle Open World 2010

by


I will be one of 18 MySQL speakers at Oracle Open World 2010 at the first ever MySQL Sunday. With a great diversity of technical, community and product talks this will be a great opportunity to get a cross section of MySQL content. Combined with Java One, this year’s Oracle Open World will include a lot more opportunity of technical and developer content then the more regular marketing material.

As the program chair for the first dedicated MySQL track at last month’s ODTUG Kaleidoscope 2010 our MySQL community now includes a larger number of target people. From the Oracle community come many highly technical and skilled resources, some with an understanding or appreciation of MySQL and many that are new to MySQL.

This is a great opportunity to share our knowledge and experience with MySQL.

References

Optimizing SQL Performance – The Art of Elimination

by

The most efficient performance optimization of a SQL statement is to eliminate it. Cary Millsap’s recent Kaleidoscope presentation again highlighted that improving performance is function of code path. Removing code will improve performance.

You may think that it could be hard to eliminate SQL, however when you know every SQL statement that is executed in your code path obvious improvements may be possible. In the sequence SQL was implemented sometimes easy observations can lead to great gains. Let me provide some actual client examples that were discovered by using the MySQL General Log.

Example 1

5 Query   SELECT *  FROM `artist`
5 Query   SELECT *  FROM `artist`
5 Query   SELECT *  FROM `artist`
5 Query   SELECT *  FROM `artist`
5 Query   SELECT *  FROM `artist`
5 Query   SELECT *  FROM `artist` WHERE (ArtistID = 196 )
5 Query   SELECT *  FROM `artist` WHERE (ArtistID = 2188 )
5 Query   SELECT *  FROM `artist`
5 Query   SELECT *  FROM `artist`
5 Query   SELECT *  FROM `artist`

In this example, the following was executed for a single page load. Not only did I find a bug where full-table scans occurred rather then being qualified, there were many repeating and unnecessary occurrences.

Example 2

SELECT option_name, option_value FROM wp_options WHERE autoload = 'yes'
SELECT option_value FROM wp_options WHERE option_name = 'aiosp_title_format' LIMIT 1
SELECT option_value FROM wp_options WHERE option_name = 'ec3_show_only_even' LIMIT 1
SELECT option_value FROM wp_options WHERE option_name = 'ec3_num_months' LIMIT 1
SELECT option_value FROM wp_options WHERE option_name = 'ec3_day_length' LIMIT 1
SELECT option_value FROM wp_options WHERE option_name = 'ec3_hide_event_box' LIMIT 1
SELECT option_value FROM wp_options WHERE option_name = 'ec3_advanced' LIMIT 1
SELECT option_value FROM wp_options WHERE option_name = 'ec3_navigation' LIMIT 1
SELECT option_value FROM wp_options WHERE option_name = 'ec3_disable_popups' LIMIT 1
SELECT option_value FROM wp_options WHERE option_name = 'sidebars_widgets' LIMIT 1

This is a stock WordPress installation and highlights a classic Row at a Time (RAT) processing.

Example 3

SELECT * FROM activities_theme WHERE theme_parent_id=0
SELECT * FROM activities_theme WHERE theme_parent_id=1
SELECT * FROM activities_theme WHERE theme_parent_id=2
SELECT * FROM activities_theme WHERE theme_parent_id=11
SELECT * FROM activities_theme WHERE theme_parent_id=16

In this client example, again RAT processing, I provided a code improvement to run these multiple queries in a single statement, otherwise known as Chunk At a Time (CAT) processing. It’s not rocket science however the elimination of the network component of several SQL statements can greatly reduce page load time.

SELECT *
FROM   activities_theme
WHERE  theme_parent_id in  (0,1,2,11,16) 

Example 4

The following represents one of the best improvement. During capture, the following query was executed 6,000 times over a 5 minute period. While you make think this is acceptable, the value passed wae 0. The pages_id is an auto_increment column which by definition does not have a 0 value. In this instance, a simple boundary condition in the code would eliminate this query.

SELECT pages_id, pages_livestats_code, pages_title,
       pages_parent, pages_exhibid, pages_theme,
       pages_accession_num
FROM pages WHERE pages_id = 0

There are many tips to improving and optimizing SQL. This is the simplest and often overlooked starting point.

Related articles include: The RAT and the CAT and We need more CAT’s