Understanding your RAID Configuration

For any production MySQL Database system, running RAID is a given these days. Do you know what RAID your database is? Are you sure?. Ask for quantifiable reproducible output from your systems provider or your System Administrator.

As a consultant I don’t always know the specific tools for the clients deployed H/W, but I ask the question. On more the one occasion the actual result differed from the clients’ perspective or what they were told, and twice I’ve discovered that clients when asked if their RAID was running in a degraded mode, it actually was and they didn’t know.

You can read about various benchmarks at MySQL blogs such as BigDBAHead and MySQL Performance Blog however getting first hand experience of your actually RAID configuration, the H/W and S/W variables is critical to knowing how your technology works. You can then build on this to run your own benchmarks.

Over 50% of my clients run on DELL equipment, most using local storage or shared storage options such as Dell MD1000, Dell MD3000, NetApps or EMC. I’ve had the opportunity to spend a few days looking into the more details of RAID, specifically the DELL PERC 5/i Raid Controllers, and I’ve started a few MySQL Cheatsheets for my own reference that others may also benefit from.

Understanding PERC RAID Controllers gives an overview of using the MegaCLI tools to retrieve valuable information on the Adapter, Physical Drives, Logical Drives and the all important Battery Backed Cache.

There are several Google search results out there about finding the MegaCLI tools, I found them to be all outdated. There is of course other tools including Dell OpenManage Server Administrator (GUI and CLI) and an Open Source project called megactl.

Here is just a summary of a few lines of each that yields valuable information:

Adapter Details

$ /opt/MegaRAID/MegaCli/MegaCli64 -AdpAllInfo -aALL

RAID Level Supported             : RAID0, RAID1, RAID5, RAID10, RAID50
Max Stripe Size         : 128kB
Stripe Size                    : 64kB

Physical Details

$ /opt/MegaRAID/MegaCli/MegaCli64 -LDPDInfo -aall

Adapter #0
Number of Virtual Disks: 1

RAID Level: Primary-5, Secondary-0, RAID Level Qualifier-3
Size:208128MB
Stripe Size: 64kB
Number Of Drives:4
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU

Raw Size: 70007MB [0x88bb93a Sectors]
Inquiry Data: FUJITSU MAY2073RC       D108B363P7305KAU
Inquiry Data: FUJITSU MAY2073RC       D108B363P7305KAJ
Inquiry Data: FUJITSU MAY2073RC       D108B363P7305JSW
Inquiry Data: FUJITSU MAY2073RC       D108B363P7305KB1

Battery

$ MegaCli -AdpBbuCmd -aALL
  Fully Charged           : Yes
  Discharging             : Yes

BBU Capacity Info for Adapter: 0
Relative State of Charge: 100 %
Absolute State of charge: 88 %
Run time to empty: 65535 Min

BBU Properties for Adapter: 0

Auto Learn Period: 7776000 Sec
Next Learn time: 304978518 Sec

The big detail that was missing was the details in this ouput of the drive speed, such as 7.2K, 10K, 15K. What is the impact? Well that’s the purpose of the next step.

Following this investigation, testing of the RAID configuration with Bonnie++ was performed to determine the likely performance of various configurations, and to test RAID0, RAID1, RAID5 and RAID10.

Further testing that would be nice would include for example RAID 5 with 3 drives verses 4 drives. The speed of the drives, the performance in a degraded situation, and the performance during a disk rebuild.

This still leaves the question about how to test the performance with and without the Battery Backed Cache. You can easily disable this via CLI tools, but testing an actually database test, and pulling the power plug for example with and without would yield some interesting results. More concerning is when Dell specifically discharges the batters, and it takes like 8 hours to recharge. In your production environment you are then running in degraded mode. Disaster always happens at the worse time.

MySQL Book winner – for 5 configuration options

If you had to configure a WordPress MU installation without access to any details of your MySQL Configuration, what would you do?

What top five configuration settings would you use?

I asked the community this question, see For MySQL DBA fame and glory. Prize included and a number of brave soles responded for a chance to win a free copy of MySQL Administrators Bible by MySQL Community She-BA Sheeri Cabral.

There is no perfect answer and of course you would want to set more then five options, however the purpose of the competition was to seek what people would do with limited information and a limited choice of actions to enable people to focus on what is important.

Part of choosing the winner was for those that accurately read the question. There were a number of triggers. These included:

  • Give an answer with five options. That was the question.
  • WordPress MU gives you an indication of the schema size (about 300 tables), the default storage engine of MyISAM, and the technology stack includes PHP.
  • The current MySQL configuration included just two existing options.

What would I do?

If I was asked this question, here is what I’d do.

  1. Demand more information about the H/W the Mysql server running on. I also would do my own physical verification as client description != actual implementation.
  2. Confirm if the server is shared or dedicated.
  3. Get SHOW GLOBAL VARIABLES output.
  4. Get 2 x SHOW GLOBAL STATUS output, separated from some time, e.g. 1 minute, 10 minutes, 1 hour.

Without the above information I’m not really prepared to tune a system in isolation. I’ll spend more time at a later time explaining why.

So my answers for reference.

key_buffer_size

We have as a pre-cursor to the top 5, the key_buffer_size. The key_buffer_size is used to store MyISAM Indexes, and is assigned as MySQL server startup. Setting this value to large is a waste of space, setting this too small will increase the amount of disk I/O. In this example, the total index space was around 100MB, so setting the value to 384-512M supports adequate growth. The default value for 5.0.67 is 8M. When you read on, see point 5, you may well consider lowering this value.

1. log-bin=/path/to/dir/file-prefix

If you are storing data in a database I’m sure you want to keep it. log-bin is necessary for point in time recovery. Period. You should also specify the actually path, and store this outside of the default MySQL data directory. The default is NOT enabled.

2. myisam_recover = force,backup

MyISAM is not the most optimal storage engine for online systems. This is a much larger discussion however for the stock default WP and general LAMP products we start with MyISAM. Setting myisam_recover will improve one disadvantage of MyISAM and that is the recovery on startup. The default is NOT enabled.

3. query_cache_size = 64M

In absence of greater information, enabling the Query Cache in a generally high read environment is a good starting point. You have the ability to set and unset this dynamically on a running server, so it is easy to remove if performance is not the optimal. You should also not go overboard with the size of the query cache. Start with a modest amount, such as 64M, assess the impact of the QC and re-assess the value. The default is NOT enabled. The Query Cache also uses the variables query_cache_type, which defaults to ON, and query_cache_limit which defaults to 1M.

4. table_cache=1024

The MU part of the WordPress is a trigger that the system will have several hundred tables. Monitoring Opened_tables, and Open_tables is as easy check to determine a more appropriate table_cache value. The default value on 5.0.67 is 64.

5. max_connections=150

This was a difficult choice, the first 4 are obvious for the current environment.

I have to question the reason why max_connections=500. Given that there is no other configuration settings, why would this value be so large. Start with a more realistic figure, such as the default of 150, and monitor max_used_connections, threads_running for a while. A review of the H/W confirmed this is unrealistic, the system has 16GB of RAM, but only a 32bit OS.

Moving forward

Tuning MySQL is not a trivial or simple task. It takes a certain amount of knowledge about the system. Recently I have seen some very unusual MySQL configurations, and I have to pass on this information for all readers and DBA’s.

Gather information about your environment, and the setting in question before making a change. Document this on an internal wiki. It is important, when in a year’s time somebody asks, why did you change this?

The bonus prize was for anybody that questioned the current configuration which only included two parameters.

I had to look up –safe-show-database. No wonder I didn’t know it, Deprecated since 4.0.2. I wonder sometimes when people add uncommon options. Shlomi the only person to at least raise a ? on that on.

And the winner, I had to consider 3 people that had 3 of my 5. Shlomi Noach made such a plea I have to give it to him, and I’ll throw in international shipping.

(Mis)Understanding RAID configuration

Update 8 June 2009
Be sure you to check out some of my research including Understanding your RAID Configuration and Understanding PERC RAID Controllers.

Last week I got caught thinking a MySQL production system wasn’t configured with RAID when it actually was. The following is the Linux dmesg output of 3 different client MySQL servers, where the H/W is all the same, a Dell 2950.

In the first two cases you will see individual references to the internal disks, however in the third you do not. This threw me, but as Matt Yonkovit correctly pointed out to me, the sizes are not standard drive sizes.

A deeper level of understanding via CLI tools, or actually console output would appear necessary to understand the disk configuration in the future.


Nov 2007 – Dell 2950 Running RHEL 4. Note the individual SEAGATE drive details and single sda (70G) and sdb(300G)

scsi0 : LSI Logic SAS based MegaRAID driver
  Vendor: SEAGATE   Model: ST973451SS        Rev: SM04
  Type:   Direct-Access                      ANSI SCSI revision: 05
  Vendor: SEAGATE   Model: ST973451SS        Rev: SM04
  Type:   Direct-Access                      ANSI SCSI revision: 05
  Vendor: SEAGATE   Model: ST973451SS        Rev: SM04
  Type:   Direct-Access                      ANSI SCSI revision: 05
  Vendor: SEAGATE   Model: ST973451SS        Rev: SM04
  Type:   Direct-Access                      ANSI SCSI revision: 05
usb 4-1: new high speed USB device using ehci_hcd and address 2
usb 4-1: configuration #1 chosen from 1 choice
hub 4-1:1.0: USB hub found
hub 4-1:1.0: 2 ports detected
  Vendor: SEAGATE   Model: ST973451SS        Rev: SM04
  Type:   Direct-Access                      ANSI SCSI revision: 05
  Vendor: SEAGATE   Model: ST973451SS        Rev: SM04
  Type:   Direct-Access                      ANSI SCSI revision: 05
  Vendor: SEAGATE   Model: ST973451SS        Rev: SM04
  Type:   Direct-Access                      ANSI SCSI revision: 05
  Vendor: DP        Model: BACKPLANE         Rev: 1.05
  Type:   Enclosure                          ANSI SCSI revision: 05
  Vendor: DELL      Model: PERC 5/i          Rev: 1.03
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sda: 142082048 512-byte hdwr sectors (72746 MB)
sda: Write Protect is off
sda: Mode Sense: 1f 00 10 08
SCSI device sda: drive cache: write through w/ FUA
SCSI device sda: 142082048 512-byte hdwr sectors (72746 MB)
sda: Write Protect is off
sda: Mode Sense: 1f 00 10 08
SCSI device sda: drive cache: write through w/ FUA
 sda: sda1 sda2 sda3 sda4 < sda5 >
sd 0:2:0:0: Attached scsi disk sda
  Vendor: DELL      Model: PERC 5/i          Rev: 1.03
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sdb: 568328192 512-byte hdwr sectors (290984 MB)
sdb: Write Protect is off
sdb: Mode Sense: 1f 00 10 08
SCSI device sdb: drive cache: write through w/ FUA
SCSI device sdb: 568328192 512-byte hdwr sectors (290984 MB)
sdb: Write Protect is off
sdb: Mode Sense: 1f 00 10 08
SCSI device sdb: drive cache: write through w/ FUA

Oct 2008 – Dell 2950 running Ubuntu 8.04. Note the individual SEAGATE drive details and single sda (600GB)

[   88.069326] scsi0 : LSI Logic SAS based MegaRAID driver
[   88.087755] scsi 0:0:0:0: Direct-Access     SEAGATE  ST3300656SS      HS09 PQ: 0 ANSI: 5
[   88.104931] scsi 0:0:1:0: Direct-Access     SEAGATE  ST3300656SS      HS09 PQ: 0 ANSI: 5
[   88.122079] scsi 0:0:2:0: Direct-Access     SEAGATE  ST3300656SS      HS09 PQ: 0 ANSI: 5
[   88.139064] scsi 0:0:3:0: Direct-Access     SEAGATE  ST3300656SS      HS09 PQ: 0 ANSI: 5
[   88.205346] scsi1 : ata_piix
[   88.205387] scsi2 : ata_piix
[   88.205409] ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0xfc00 irq 14
[   88.205411] ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0xfc08 irq 15
[   88.240207] scsi 0:0:32:0: Enclosure         DP       BACKPLANE        1.05 PQ: 0 ANSI: 5
[   88.252370] scsi 0:2:0:0: Direct-Access     DELL     PERC 6/i         1.21 PQ: 0 ANSI: 5
[   88.263532] scsi 0:0:32:0: Attached scsi generic sg0 type 13
[   88.263547] scsi 0:2:0:0: Attached scsi generic sg1 type 0
[   88.265390] Driver 'sd' needs updating - please use bus_type methods
[   88.265671] sd 0:2:0:0: [sda] 1169686528 512-byte hardware sectors (598880 MB)
[   88.265700] sd 0:2:0:0: [sda] Write Protect is off
[   88.265702] sd 0:2:0:0: [sda] Mode Sense: 1f 00 10 08
[   88.265771] sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
[   88.266135] sd 0:2:0:0: [sda] 1169686528 512-byte hardware sectors (598880 MB)
[   88.266173] sd 0:2:0:0: [sda] Write Protect is off
[   88.266175] sd 0:2:0:0: [sda] Mode Sense: 1f 00 10 08
[   88.266232] sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
[   88.266240]  sda: sda1 sda2 sda3
[   88.270085] sd 0:2:0:0: [sda] Attached SCSI disk

May 2009 – Dell 2950 running RHEL 5. Note no individual drive details, only partition sizes for sda (300G) and sdb (600G).

scsi0 : LSI Logic SAS based MegaRAID driver
  Vendor: DP        Model: BACKPLANE         Rev: 1.05
  Type:   Enclosure                          ANSI SCSI revision: 05
  Vendor: DELL      Model: PERC 5/i          Rev: 1.03
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sda: 584843264 512-byte hdwr sectors (299440 MB)
SCSI device sda: drive cache: write through
SCSI device sda: 584843264 512-byte hdwr sectors (299440 MB)
SCSI device sda: drive cache: write through
 sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 sda8 sda9 sda10 sda11 >
Attached scsi disk sda at scsi0, channel 2, id 0, lun 0
  Vendor: DELL      Model: PERC 5/i          Rev: 1.03
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sdb: 1169686528 512-byte hdwr sectors (598880 MB)
SCSI device sdb: drive cache: write through
SCSI device sdb: 1169686528 512-byte hdwr sectors (598880 MB)
SCSI device sdb: drive cache: write through
 sdb: sdb1 sdb2 sdb3

Leveraging the power of Twitter

Last week I posted the following twitter request“Can somebody loan me (or buy me) a Dell 2950 decked out so I can run and publish some benchmarks. Please!”

In a same day response I was offered access to use 2 x Dell 1950’s, and today I’m now actually using these machines for my own testing. I would like to thank cafemom (Barry, Anthony & Dan) for the loan of hardware.

And now the chance to better understand the RAID configuration of the DELL PERC Controllers, trying out some different RAID types, LVM configurations and disk tests. When I’m done with my System Administrator refresher, I’m then be trying some different MySQL Benchmarks to test various MySQL configuration settings including using the new Juice benchmark.

Free MySQL Book giveway – Current Progress

I’ve decided to give people two more days for a chance to win a free MySQL Book — Sheeri Cabral’s MySQL Administrators Bible.

I have had five people so far provide recommendations for a simple MySQL configuration question as stated in For MySQL DBA fame and glory. Prize included. Shlomi Noach the current front runner.

Try your MySQL Performance Tuning skills. This is a good opportunity for new MySQL DBA’s and experienced DBA’s to provide basic input.

More Basic MySQL Security

The reason for yesterday’s Basic OS/MySQL Security was a request to review a system and I was given the production server ‘root’ password in an email. Never email a ‘root’ password, especially including the hostname as well. Email is an insecure protocol that can be monitored by hackers. However, today’s basic security tip following a look at the system is:

Never store the MySQL ‘root’ user password in a ~root/.my.cnf file.

There is simply no reason to do so, and you expose your database to destruction or manipulation when a user has access to the ‘root’ OS user, for example via sudo.

I’ve heard excuses why the ‘root’ MySQL password has to be in a file, I’ve yet to be convinced.

Do you need to store a MySQL password in a file? Yes. Connection management for your application is an example, however that use should never be ‘root’ user to connect to your application.
You may also need to run scripts to backup your data with mysqldump. The solution is to create a dedicated user .e.g. ‘backup’, and then grant that user only the permissions necessary to do the specific task at hand. By default, a simple mysql command will never grant access without any authentication, the user will need to find the password.

As David Minor pointed out in comments, you can also with MySQL change the ‘root’ user name, which is not a bad idea for improved security. The follow steps perform this.

$ mysql -uroot -p[password]
mysql> create user [email protected] identified by '[newpassword]';
mysql> grant all on *.* to [email protected] with grant option;
mysql> exit

# Check you can really login
$ mysql -udba -pnewpassword
mysql> select host,user from mysql.user;
# Drop 'root' users as listed, generally
mysql> drop user [email protected];
mysql> drop user [email protected];
mysql> drop user [email protected][hostname];
mysql> select host,user from mysql.user;
mysql> exit

More information see the MySQL 5.1 Reference Manual at MySQL Privileges, Create User and Grant Syntax.

For MySQL DBA fame and glory. Prize included.

I came across the following configuration today on a Production MySQL system (5.0.67) running 30+ blogs using WordPress MU.

$ cat /etc/my.cnf
[mysqld]
set-variable = max_connections=500
safe-show-database

No I did not truncate the output. I could say I’ve seen worse, but that’s a stretch.


So the quiz and a prize for the best response, for the next 48 hours I’ll accept your comments as responses to this post for the top 5 settings you would add, and additionally what information you may need to add these settings. Bonus points for giving a reason why you would add the settings as well.

For example, I’ll give you the most obvious.

key_buffer_size = ????

To determine a key_buffer_size to start with I would look at the size of all Indexes via I_S, and combine with some estimate of growth, say 2x-5x.

For the best answer with the top 5 settings, I’ll send you a copy of Sheeri Cabral’s book MySQL Administrators Bible which even I as a resident MySQL Expert has enjoyed scanning and learning something new.

Basic OS/MySQL Security

If you can do either of these on your MySQL production server, you need to correct immediately.

1. Login directly to your MySQL server as the ‘root’ Linux Operating System user. For example:

$ ssh [email protected]
Password:  ************

2. Connect to MySQL database as the ‘root’ MySQL user without a password.

$ mysql -uroot

Here are the 60 second fixes to address these major security flaws.
To disable direct root access to your server, first ensure you can login as a normal user, then su – or sudo su – appropriately. Then, disable ssh root access with the following configuration change.

$   vi /etc/ssh/sshd_config
    # ensure this is commented out and set to no
    PermitRootLogin no

$   /etc/init.d/sshd restart

This will stop any brute force attack on your server by automated bots and password generators.

Second, the default installation of MySQL *DOES NOT SET A PASSWORD*. Apart from being crazy, I’ve seen production systems without a MySQL ‘root’ user password. To set a password run:

$ mysqladmin -uroot password SOMEPASSWORD
$ mysqladmin -uroot -pSOMEPASSWORD -hlocalhost password SOMEPASSWORD

Transcending Technology Specific Boundaries

I had the pleasure to sit on the Performance Panel at the recent Percona Performance Conference. While the panel contained a number of usual MySQL suspects, one person was not familiar, that being Cary Millsap from Method R.

An expert in optimizing Oracle performance, Cary also gave an session on Day 2 that I attended. While he opened professing not to be an expert in MySQL, his talk provided valuable foundation knowledge irrespective of whether you use MySQL or another database product.

Having come myself from 7 straight years in system architecture and performance tuning in Ingres, then a further 6 years in Oracle again heavily involved in system architecture and performance tuning, a lot of my experience in the 10 years of providing my own MySQL consulting is drawn from my past RDBMS experiences. In addition much of what I actually provide to clients today is common sense that I don’t see applied.

A summary of the excellent content provided by Cary.

The common technology agnostic problem we need to address is:

  • Users say that everything is slow, but I don’t know where to begin
  • Users are complaining but all the monitoring dials are green

From a user’s perspective, their experience consist simply of two elements.

  1. Task
  2. Time

In general, business people simply don’t care about the “system” except thought the specific tasks that make up their pressing business needs. And for these users, performance is all about the time to complete this task.

Throughput can be stated as tasks per time.
Response time is the time taken per task.

Cary also quoted Donald Knuth — “The universal experience of programmers who have been using measurements tools has been that there intuitive guesses fail.”

Performance is easy if you stop guessing where your code is slow. A few best practice tips are:

  • You have to insist on seeing where time goes for any task you think is important
  • You need to look at the sequence diagram of the task
  • What individual part takes the most time, then look at the task before that. The fastest way to do something is don’t.
  • To drill down, you need to attack the skew of each part, not the average.

In Summary the closing points were:

  • Performance is about time and tasks
  • Not all tasks are created equal
  • Read “The Goal”
  • Don’t guess, your probably wrong
  • Measure response time before you optimize anything – Insist on it

Performance is easy when code measures it’s own time and tasks. This closing statement on instrumentation I completely concur with.

SHOW WARNINGS woes

Recently on a client site I had to fight the pain of having no way to confirm loss of data integrity when optimizing data types. Due to MySQL’s ability to perform silent conversion of data, when converting a number of columns we enabled sql_mode to catch any truncations as errors.

sql_mode=STRICT_ALL_TABLES

This ensured that should any data truncations occur, an error is thrown not a warning. The following shows an example case study for converting an INT to TINYINT UNSIGNED and shows that without sql_mode silent conversions occur.

mysql> drop schema if exists tmp;
Query OK, 25 rows affected (0.40 sec)

mysql> create schema tmp;
Query OK, 1 row affected (0.01 sec)

mysql> use tmp
Database changed
mysql> create table t1(i1 INT NULL);
Query OK, 0 rows affected (0.15 sec)

mysql> insert into t1 values(1),(2),(3),(256),(65536),(NULL);
Query OK, 6 rows affected (0.06 sec)
Records: 6  Duplicates: 0  Warnings: 0

mysql> select * from t1;
+-------+
| i1    |
+-------+
|     1 |
|     2 |
|     3 |
|   256 |
| 65536 |
|  NULL |
+-------+
6 rows in set (0.03 sec)

mysql> set session sql_mode=STRICT_ALL_TABLES;
Query OK, 0 rows affected (0.05 sec)

mysql> alter table t1 modify i1 TINYINT UNSIGNED NULL;
ERROR 1264 (22003): Out of range value for column 'i1' at row 4
mysql> set session sql_mode='';
Query OK, 0 rows affected (0.03 sec)

mysql> alter table t1 modify i1 TINYINT UNSIGNED NULL;
Query OK, 6 rows affected, 2 warnings (0.07 sec)
Records: 6  Duplicates: 0  Warnings: 2

mysql> show warnings;
+---------+------+---------------------------------------------+
| Level   | Code | Message                                     |
+---------+------+---------------------------------------------+
| Warning | 1264 | Out of range value for column 'i1' at row 4 |
| Warning | 1264 | Out of range value for column 'i1' at row 5 |
+---------+------+---------------------------------------------+
2 rows in set (0.00 sec)

mysql> select * from t1;
+------+
| i1   |
+------+
|    1 |
|    2 |
|    3 |
|  255 |
|  255 |
| NULL |
+------+
6 rows in set (0.00 sec)

The process however for the client included converting a number of DATETIME columns to DATE columns. The input data specification called for the input values to be dates only so storing and using DATETIME was simply a waste of space.

Unfortunately the use of sql_mode didn’t help here, so the combination of changing multiple columns in one time simply meant it was impossible to determine if any truncations occurred.

This combined with the sheer data volume, tables between 10GB – 500GB, with 10 – 100 million rows.
When you see 14+ millions rows modified, 35+million rows and of course SHOW WARNINGS by default only shows a very small number of actual warnings

It is of course impossible to run individual ALTER statements on columns. Of the top 10 tables, excluding the largest, alter times were between 3 and 15 hrs.

The only solution is to run pre SQL checks on the data to search for the boundary conditions manually. This is not the ideal solution for a customer.

The MySQL crystal ball says …

As the recipient of the 2009 MySQL Community Member of the Year award I received a MySQL crystal ball. While it looks good in my bookcase, unfortunately the best advice I can offer during this time of uncertainty is “watch this space”.

A number of topics where information is still very much unknown and I’m either asked about, or am following includes:

  • The Oracle acquisition of Sun, owner of MySQL.
  • MySQL 5.4 Alpha release and schedule for production release
  • The end of MySQL 5.0 Community/Enterprise split
  • The future of Falcon in MySQL 6.0?

Thanks to Julian Cash of the Human Creativity Project of his photograph at the 2009 MySQL Conference.

Ronald Bradford - MySQL Community Member of the Year

View Photo. View all 2009 MySQL Conference photos.

MySQL for the Oracle DBA Resources

The announcement last month of Oracle to acquire Sun continues to warrant a lot of discussion over exactly what Oracle will do with MySQL. Only time will tell what will happen with the official product, however it is important to remember that MySQL is GPL, there will always be a free version of MySQL available for popular LAMP stack products such as WordPress and Drupal and new and existing startup’s will continue to use MySQL.

This announcement will see Oracle resources begin to better understand and evaluate MySQL. As a resident MySQL Expert, I also have a strong background in Oracle having also worked for Oracle Corporation. I have also delivered several successful one day and half day workshops on MySQL/Oracle related content including:

  • 2009 – Best Practices for Migrating to MySQL from Oracle and SQL Server environments. 4 hour workshop for the Federal Government – Washington DC
  • 2008 – MySQL for the Oracle DBA Bootcamp. 1 day workshop for the Federal Government – Washington DC
  • 2007 – MySQL for the Oracle DBA Bootcamp. 1 day workshop New York, NY & San Francisco, CA
  • 2007 – MySQL for Oracle DBA’s and Developers. 90 minute presentation at MySQL Conference and Expo, Santa Clara, CA
  • 2006 – MySQL for Oracle Developers. 45 minute presentation at MySQL Users Conference, Santa Clara, CA
  • 2006 – Know your Competitor. A MySQL developers guide to Oracle 10g express edition. 60 minutes presentation at MySQL Users Group, Brisbane, Australia

In addition I have proposed 2 MySQL related talks at Oracle Open World 2009

  • Integrating MySQL into your Oracle DBA management processes
  • An overview for evaluating migrating from Oracle to MySQL

For more information check out my MySQL for the Oracle DBA Resources.

HiTCHO Top tech tips

I recent visit with old Brisbane friend HiTCHO which I met at the Brisbane MySQL Users Group in 2005, has lead to this cool list of some hardware and software technologies he used that I am now considering or have already implemented or purchased.

Software

  1. xmarks.com – Bookmark-Powered Web Discovery
  2. Pulse – Smart Pen
  3. Quicksilver Mac windows manager
  4. MailPlane – Brings Gmail to your Mac desktop
  5. Evernote – Remember Everything, with Firefox plugin and iPhone App
  6. Textmate – The missing editor for Mac OS/X
  7. Screen flow Professional screencasting Studio
  8. Snoop – A GNU/Linux file descriptor monitoring tool inspired by FreeBSD’s ‘watch’.

Hardware

  1. Drobo – Storage that manages itself
  2. Canon PowerShot SX1. True HD in a Canon compact digital camera.
  3. LiveScribe – Never miss a word

One advantage of Oracle/Sun/MySQL

This weeks’ announcement Oracle to by Sun was a major talking point at the 2009 MySQL Conference & Expo. While it is too early to even speculate what the future holds with the official MySQL product, for myself a speaker on MySQL topics, Oracle Open World is now a target market.

In addition to many years of providing MySQL for the Oracle DBA Resources I have with the recent closure of call for papers submitted two sessions for consideration.

Integrating MySQL into your Oracle DBA management processes

Most large enterprise organizations use more then one RDBMS product to service business requirements. With the increase in MySQL usage for web based applications such as self-service content, Oracle DBA’s need to understand and appreciate the minimum to ensure performance and availability meets client expectations.

Just how to you integrate MySQL into existing and existing Oracle database infrastructure and management monitoring process?
What are the critical monitoring components? How do these compare to current Oracle Best Practices.
Understand the various end user tools support multiple RDBMS products including Oracle and MySQL.

In this session, DBA’s will leave with the essential knowledge and appreciation of MySQL management.

An overview for evaluating migrating from Oracle to MySQL

MySQL is becoming increasing popular RDBMS for web based applications due to it’s ease of use, availability within the the LAMP stack and large number of open source applications. While implementing MySQL for a new development project may be easy, migrating existing databases, data and applications to use MySQL is not. In this presentation, we will answer questions including:

What are the major challenges to overcome to consider MySQL for some portion of your business?
What are the issues in application portability?
What are ideal applications to consider for migration?
Tools, Products and options for easy migration?
What Oracle features are not supported?

Learn how to read and write to MySQL directly via Oracle Heterogeneous services.

Announcing Drizzle on EC2

I have published the very first sharable Drizzle Amazon Machine Image (AMI) for AWS EC2, based on the good feedback from my discussion at the Drizzle Developer Day on what options we should try.

This first version is a 32bit Developer instance, showcasing Drizzle and all necessary developer tools to build Drizzle from source.

What you will find on drizzle-ami/intrepid-dev32 – ami-b858bfd1

Ubuntu 8.10 Intrepid 32 bit base server installation:

  • build tools
  • drizzle dependencies
  • bzr 1.31.1

From the respective source trees the following software is available:

  • drizzle 2009.04.997
  • libdrizzle 0.0.2
  • gearman 0.0.4
  • memcached 1.2.8
  • libmemcached 0.28

Drizzle has been configured with necessary dependencies for PAM authentication, http_auth, libgearman and MD5 but these don’t seem to be available in the binary distribution.

I will be creating additional AMI’s including 64bit and LAMP ready binary only images.

The following example shows using drizzle on this AMI. Some further work is necessary for full automation, parameters and logging. I’ve raised a number of issues the Drizzle Developers are now hard at work on.

1. Starting Drizzle

ssh [email protected]
sudo /etc/init.d/drizzle-server.init start &

2. Testing Drizzle (the sakila database has been installed)

$ drizzle
Welcome to the Drizzle client..  Commands end with ; or g.
Your Drizzle connection id is 4
Server version: 2009.04.997 Source distribution

Type 'help;' or 'h' for help. Type 'c' to clear the buffer.

drizzle> select version();
+-------------+
| version()   |
+-------------+
| 2009.04.997 |
+-------------+
1 row in set (0 sec)

drizzle> select count(*) from sakila.film;
+----------+
| count(*) |
+----------+
|     1000 |
+----------+
1 row in set (0 sec)

3. Compiling Drizzle

sudo su - drizzle
ls
deploy  drizzle  libdrizzle  sakila-drizzle
cd drizzle
./configure --help
Description of plugins:

   === HTTP Authentication Plugin ===
  Plugin Name:      auth_http
  Description:      HTTP based authentications
  Supports build:   static and dynamic

   === PAM Authenication Plugin ===
  Plugin Name:      auth_pam
  Description:      PAM based authenication.
  Supports build:   dynamic

   === compression UDFs ===
  Plugin Name:      compression
  Description:      UDF Plugin for compression
  Supports build:   static and dynamic
  Status:           mandatory

   === crc32 UDF ===
  Plugin Name:      crc32
  Description:      UDF Plugin for crc32
  Supports build:   static and dynamic
  Status:           mandatory

   === Error Message Plugin ===
  Plugin Name:      errmsg_stderr
  Description:      Errmsg Plugin that sends messages to stderr.
  Supports build:   dynamic

   === Daemon Example Plugin ===
  Plugin Name:      hello_world
  Description:      UDF Plugin for Hello World.
  Supports build:   dynamic

   === Gearman Logging Plugin ===
  Plugin Name:      logging_gearman
  Description:      Logging Plugin that logs to Gearman.
  Supports build:   dynamic

   === Query Logging Plugin ===
  Plugin Name:      logging_query
  Description:      Logging Plugin that logs all queries.
  Supports build:   static and dynamic
  Status:           mandatory

   === Syslog Logging Plugin ===
  Plugin Name:      logging_syslog
  Description:      Logging Plugin that writes to syslog.
  Supports build:   static and dynamic
  Status:           mandatory

   === MD5 UDF ===
  Plugin Name:      md5
  Description:      UDF Plugin for MD5
  Supports build:   static and dynamic

   === One Thread Per Connection Scheduler ===
  Plugin Name:      multi_thread
  Description:      plugin for multi_thread
  Supports build:   static
  Status:           mandatory

   === Old libdrizzle Protocol ===
  Plugin Name:      oldlibdrizzle
  Description:      plugin for oldlibdrizzle
  Supports build:   static
  Status:           mandatory

   === Pool of Threads Scheduler ===
  Plugin Name:      pool_of_threads
  Description:      plugin for pool_of_threads
  Supports build:   static
  Status:           mandatory

   === Default Signal Handler ===
  Plugin Name:      signal_handler
  Description:      plugin for signal_handler
  Supports build:   static
  Status:           mandatory

   === Single Thread Scheduler ===
  Plugin Name:      single_thread
  Description:      plugin for single_thread
  Supports build:   static
  Status:           mandatory

   === Archive Storage Engine ===
  Plugin Name:      archive
  Description:      Archive Storage Engine
  Supports build:   static
  Status:           mandatory

   === Blackhole Storage Engine ===
  Plugin Name:      blackhole
  Description:      Basic Write-only Read-never tables
  Supports build:   static and dynamic
  Configurations:   max, max-no-ndb

   === CSV Storage Engine ===
  Plugin Name:      csv
  Description:      Stores tables in text CSV format
  Supports build:   static
  Status:           mandatory

   === Memory Storage Engine ===
  Plugin Name:      heap
  Description:      Volatile memory based tables
  Supports build:   static
  Status:           mandatory

   === InnoDB Storage Engine ===
  Plugin Name:      innobase
  Description:      Transactional Tables using InnoDB
  Supports build:   static and dynamic
  Configurations:   max, max-no-ndb
  Status:           mandatory

   === MyISAM Storage Engine ===
  Plugin Name:      myisam
  Description:      Traditional non-transactional MySQL tables
  Supports build:   static
  Status:           mandatory


Report bugs to <http://bugs.launchpad.net/drizzle>.

Compiling libdrizzle

Compiling libdrizzle is a rather trivial task. The following are the steps I undertook on Ubuntu 8.10 Intrepid 32 bit.

There was one pre-requisite from the most basic installed developer tools.

sudo apt-get install -y  automake
bzr clone lp:libdrizzle
cd libdrizzle
./config/autorun.sh
./configure
make
make install

And there they are:

$ ls -l /usr/local/lib/libdrizzle*
-rw-r--r-- 1 root root 1122710 2009-04-26 18:10 /usr/local/lib/libdrizzle.a
-rwxr-xr-x 1 root root     940 2009-04-26 18:10 /usr/local/lib/libdrizzle.la
lrwxrwxrwx 1 root root      19 2009-04-26 18:10 /usr/local/lib/libdrizzle.so -> libdrizzle.so.0.0.2
lrwxrwxrwx 1 root root      19 2009-04-26 18:10 /usr/local/lib/libdrizzle.so.0 -> libdrizzle.so.0.0.2
-rwxr-xr-x 1 root root 1003734 2009-04-26 18:10 /usr/local/lib/libdrizzle.so.0.0.2

I added the following to enable other programs using libdrizzle to find the libraries in the system path.

echo "/usr/local/lib" > /etc/ld.so.conf.d/drizzle.conf
ldconfig

Drizzle/bzr dependency

A number of developers had problems on Friday at the Drizzle Developer Day with compiling bzr. The distro in question I was helping with was CentOS 5 32-bit. I had no issues on CentOS 5 64bit.

Today while creating the first deployed Drizzle AWS AMI I discovered the same problem using Ubuntu 8.10 Intrepid 32 bit.

The solution was actually rather trivial. Installing the python-dev package solved the problem.

apt-get install python-dev
Bzr 1.13.1 Compiling error

building 'bzrlib._btree_serializer_c' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c bzrlib/_btree_serializer_c.c -o build/temp.linux-i686-2.5/bzrlib/_btree_serializer_c.o
bzrlib/_btree_serializer_c.c:4:20: error: Python.h: No such file or directory
bzrlib/_btree_serializer_c.c:5:26: error: structmember.h: No such file or directory
bzrlib/_btree_serializer_c.c:35: error: expected specifier-qualifier-list before ‘PyObject’
....
bzrlib/_btree_serializer_c.c:1651: error: request for member ‘f_lineno’ in something not a structure or union
bzrlib/_btree_serializer_c.c:1651: warning: statement with no effect
bzrlib/_btree_serializer_c.c:1652: warning: implicit declaration of function ‘PyTraceBack_Here’

  Cannot build extension "bzrlib._btree_serializer_c".
  Use "build_ext --allow-python-fallback" to use slower python implementations instead.

error: command 'gcc' failed with exit status 1

Adding a Drizzle Plugin

I joined about 50 others including a number of core MySQL developers and MySQL community members today for the 2009 Drizzle developers day at Sun Microsystems Santa Clara campus.

In addition to a number of presentations and various group discussions most of my individual hacking time was under the guidance of Drizzle team developer Stewart Smith were Patrick Galbraith and myself started the porting of Patrick’s memcached UDF functions for MySQL.

Leveraging some existing Drizzle plugin’s such as CRC32() and UNCOMPRESS() we were easily able to navigate the src/plugin/memcached directory plug.in, Makefile.am and drizzle_declare_plugin definition in the new get.cc to get a working stub ‘Hello World Example';

plug.in

$ more plug.in
DRIZZLE_PLUGIN(memcached,[memcached UDF],
        [UDF Plugin for memcached])
DRIZZLE_PLUGIN_STATIC(memcached,   [libmemcachedudf.a])
DRIZZLE_PLUGIN_MANDATORY(memcached)  dnl Default
DRIZZLE_PLUGIN_DYNAMIC(memcached,   [libmemcachedudf.la])
Makefile.am
$ more Makefile.am
# Copyright (C) 2006 MySQL AB
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA

EXTRA_LTLIBRARIES =	libmemcachedudf.la
pkgplugin_LTLIBRARIES =	@[email protected]
libmemcachedudf_la_LDFLAGS =	-module -avoid-version -rpath $(pkgplugindir)
libmemcachedudf_la_LIBADD =		$(LIBZ)
libmemcachedudf_la_CPPFLAGS=	$(AM_CPPFLAGS) -DDRIZZLE_DYNAMIC_PLUGIN
libmemcachedudf_la_SOURCES =	get.cc


EXTRA_LIBRARIES =	libmemcachedudf.a
noinst_LIBRARIES =	@[email protected]
libmemcachedudf_a_SOURCES=	$(libmemcachedudf_la_SOURCES)
$ more get.cc
/* Copyright (C) 2009 Patrick Galbraith, Ronald Bradford
...
*/
#include <drizzled/server_includes.h>
#include <drizzled/sql_udf.h>
#include <drizzled/item/func.h>
#include <drizzled/function/str/strfunc.h>

#include <stdio.h>
#include <libmemcached/memcached.h>

using namespace std;

/* memc_get */
class Item_funcmemc_get : public Item_str_func
{
public:
  Item_funcmemc_get() : Item_str_func() {}
  const char *func_name() const { return "memc_get"; }
  bool check_argument_count(int n) { return (n==1); }
  String *val_str(String*);
  void fix_length_and_dec() {
    max_length=32;
    args[0]->collation.set(
      get_charset_by_csname(args[0]->collation.collation->csname,
                            MY_CS_BINSORT), DERIVATION_COERCIBLE);
  }

};


String *Item_funcmemc_get::val_str(String *str)
{
  assert(fixed == 1);
  String * sptr= args[0]->val_str(str);
  str->set_charset(&my_charset_bin);
  if (sptr)
  {
    null_value=0;
    str->set("hello memcached test", 20,system_charset_info);
    return str;
  }
  null_value=1;
  return 0;
}


Create_function memc_get_factory(string("memc_get"));

static int memcached_plugin_init(PluginRegistry &registry)
{
  registry.add(&memc_get_factory);
  return 0;
}

drizzle_declare_plugin(memcached)
{
  "memcached",
  "0.1",
  "Patrick Galbraith, Ronald Bradford",
  "memcached plugin",
  PLUGIN_LICENSE_GPL,
  memcached_plugin_init, /* Plugin Init */
  NULL,   /* Plugin Deinit */
  NULL,   /* status variables */
  NULL,   /* system variables */
  NULL    /* config options */
}
drizzle_declare_plugin_end;

Percona Performance Conference Talk

My final presentation during the 2009 MySQL Conference and Expo week was with the Percona Performance Conference on the topic of The Ideal Performance Architecture. My talk included discussions on Technology, Disk, Memory, Indexes, SQL and Data.

Updated 09/18/09
you can now see video of the event at Percona TV.

MySQL Monitoring 101

At the 2009 MySQL Conference and Expo I presented to a full room on MySQL Monitoring 101.

This presentation focused on the following four goals.

  • Know what to monitor
  • Know how you can monitor
  • Learn practices to diagnose problems
  • Have a foundation of historical information

Updated 09/18/09
You can also find additional materials at:

A change in the MySQL Binary distributions

Yesterday was the surprise announcement of MySQL 5.4 at the 2009 MySQL Conference and Expo. It was unfortunate that the supporting information was not that forthcoming on the MySQL website. I tried for several hours to try and download, but no mirrors were initially available. Today I see some information on the mysql.com home page and finally able to get the binary.

What I found most significant with this new major version release is a change in the binary distribution, as seen on the Download page.

MySQL 5.4 is only available on 3 platforms:

  • Linux (AMD64 / Intel EM64T)
  • Solaris 10 (SPARC, 64-bit)
  • Solaris 10 (AMD64 / Intel EM64T, 64-bit)

I was also surprised that this beta release highlights the emphasis of community contributions (long overdue), yet the community and indeed many employees of Sun/MySQL were simply unaware of this work. This is clearly a change in involving the community. While I applaud the beta status, hopefully a more stable product to start with, it’s development was done in a very closed company model.

Setting up MySQL on Amazon Web Services (AWS) Presentation

On Tuesday at the MySQL Camp 2009 in Santa Clara I presented Setting up MySQL on Amazon Web Services (AWS).

This presentation assumed you know nothing about AWS, and have no account. With Internet access via a Browser and a valid Credit Card, you can have your own running Web Server on the Internet in under 10 minutes, just point and click.

We also step into some more detail online click and point and supplied command line tools to demonstrate some more advanced usage.

What's happening with InnoDB

I have moved on to InnoDB: Innovative Technologies for Performance and Data Protection by Ken Jacobs at MySQL Conference and Expo.

With a brief history lesson of inception from 1994, inclusion in MySQL in 2000 and acquired by Oracle in 2005. Most of the work was done by one person. InnoDB is based on sound database computer science using Gray & Reuters definitive text on database design.

Some key points in Ken’s discussion.

  • Adaptive Hash indexing for frequent queries on keys.
  • In plugin Adaptive Hash is configurable
  • Insert Buffering – Deferring secondary index writes
  • Fast Index Create – doesn’t requires all indexes to be rebuilt
  • Table Compression – Changing the page size

The InnoDB plugin available in 5.1 has a number of new benefits.

  • fast index creation
  • table compression
  • info schema tables
  • new row storage format
  • file format management

All InnoDB 1.0.3 plugin features will be available in MySQL 5.4

The big announcement is a new product – Embedded InnoDB. This has the high performance, reliability and rich functionality of InnoDB, has a flexible programmatic API. No SQL, No security.

Search at Craigslist

I am now sitting in on MySQL and Search at Craigslist by Jeremy Zawodny at MySQL Users Conference

Some of the technical difficulties that required addressing.

  • High churn rate
  • half life can be very short
  • Growth
  • Traffic
  • Need to archive postings, e.g. 100M but be searchable
  • Internationalization and UTF-8

Some of the Craigslist Goals

  • Open Source
  • Easy and approachable
  • be green with energy use

A review of the Internals server configuration

  • Load Balancer (perlbal like)
  • Read Proxy Array (perl+memcached)
  • Web Read Array (apache 1.3 + mod_perl)
  • Object Cache (Perl + memcached)
  • Read DB Cluster (MySQL 5.0.x)
  • Search Cluster (Sphinx)

Clusters of DB servers have good vertical partitioning by Roles. These being

  • Users
  • Classified
  • Forums
  • Stats
  • Archive

Sphinx is a full standalone full text search that is used. Did compare with Apache Solr, but it seemed more complex and complicated. The Sphinx configuration:

  • Partitioned based on cities (people search locally)
  • Attributes v Keywords
  • Persistent Connections
  • Minimal stopword list
  • Partition in 2 clusters (1 master, 4 slaves)

The results of implementing Sphinx were:

  • decrease in 25 MySQL boxes to 10 sphinx boxes
  • no locking
  • 1,000+ qps
  • 50M queries per day
  • Better separation of code

MySQL Users Conference Opening Lines

Opening introduction from Colin Charles got us started. Karen Tegan Padir VP MySQL & Software Infrastructure was the opening keynote.

She comes from a strong tech background and is passionate about open source, the communities and how to make a successful product.

There isn’t a person that doesn’t go a day without interacting with a website or hardware system that uses a MySQL database.

The big news was the announcement of MySQL 5.4 – Performance & Scalability. Key features include.

  • InnoDb scalability 16way x86 and 64 way CMT servers
  • subquery optimization
  • new query algorithms
  • improved stored procedures, and prepared statements
  • enhanced Information Schema
  • improved DTrace Support

More information at MySQL 5.4 Announcement Details….

Other key points includes:

1. Ken Jacobs announces today an Embedded Innodb with a powerful API (not SQL based). Read more at Innobase Introduces Embedded InnoDB
2. MySQLCluster 7.0 is also released today. Some benchmarks 4.3x improvements. New features also include LDAP support.
3. The next release of MySQL Query Analyzer, 2.1 announced.
4. Sun announces a commitment to accept contributions from the community.
5. Community also gets the Monthly Rapid Updates.
6. MySQL Drizzle Project is discussed as a technology incubator.

Partners of the year: Intel, Infobright and Lifeboard.
Appliation of the year: Zappos, Alcatel-lucent and Symantec.
Community members of the year: Marc Delisle, Ronald Bradford, Shlomi Noach.

Where is the MySQL in Sun's announcement

I find it surprising that in the official Sun Announcement there is no mention of MySQL for two reasons. Firstly, this was Sun largest single purchase of $1 billion only 12 months ago. Second, MySQL’s largest competitor is Oracle.

While the Sun website shows the news in grandeur, the MySQL website is noticeably absent in any information of it’s owners’ acquisition.

On my professional side, as an independent speaker for Sun Microsystems with plans for upcoming webinars and future speaking on “Best Practices in Migrating to MySQL from Oracle”, this news does not benefit my bottom line.

Drizzle + PHP = Sweet

I’ve just successfully configured Drizzle with the PHP Extension and successfully retrieve data to present on a web page.

Qudos to Eric Day for his work. I was able to identify a problem with the current tar release, and a quick confirmation on #drizzle at IRC confirmed a fix had already been commited.

I’m looking forward to evaluating WordPress and Drupal, two popular and common LAMP stack applications that run on MySQL, and to provide any feedback to the community for future support of Drizzle.

What questions do you ask?

When you have to evaluate a MySQL System & Environment, what questions do you ask in order to determine critical information about the environment and evaluate the business success and viability. You don’t have to be a consultant to ask these questions, ask them of your own environment. Do any of the answers shock or concern you?

I was prompted to write about this from a conversation with a colleague about “accepting risk”. His comment was, “every IT server on the planet is vulnerable regardless of best practices.”

Here is my list of questions for you based on an immediate response from this discussion?

Technology

  • What is your full technology stack, i.e. Operating System, Database, Application Server, Development Language(s) and other essential components?
  • What are the versions of these technologies?
  • What new technologies or versions of existing technologies are you presently evaluating?

Disaster Recovery

  • What is your Backup and Recovery strategy including your Database, Application and Administration?
  • Have you tested your Backup and Recovery strategy?
  • Have you really tested your Backup and Recovery strategy from end to end? How long ago? How long did it take?
  • What RAID do you run? Have you physically verified that? When did you confirm you are not running in a degraded RAID situation?
  • What does your website look like when it’s unavailable? What is the physical content on the website. Let’s pretend your entire data center is unavailable for 40 hours.
  • Have you ever had a major disaster? What did you learn from this experience?

Development Processes

  • Do you use a version control system? Which one? Is everything under version control?
  • Do you have a controlled and reproducible build and release process? Is it automated in any way?
  • What are your levels of testing. Unit test for coding? Regression testing for new features? Volume Testing? End User Testing?
  • Do you have a proper test environment (which is not production) where you can accurately evaluate production software and production problems?

Infrastructure

  • How do you know when there is a problem with your site? Do you have monitoring and alert notification in place?
  • When you have a performance problem, can you evaluate if it is new, re-occuring or a gradually worsening problem?
  • What are you two biggest performance problems right now? What are the specific details of the problem? “My website is slow” is not an answer
  • Can you roll out new features without taking your website down for general use?

Business Viability

  • How long would a customer stay with your site if it was unavailable?
  • Can your clients be satisfied with the Twitter “failed whale” approach or will they leave?
  • When will your system crash under load? Do you know this figure? What is the load today and the projections to this failure point?

Given more time I’d probably revise the list, but this was just an initial response.

This post is part of 31 Days to Build a Better BlogWrite a List Post.

mysql.com search is so broken

Today, while on the MySQL manual page, I typed in ‘select’ in the search manual box to confirm the SELECT syntax.

The result was not what I expected, the “SELECT” command. Instead I only got two options “Speed of SELECT …” and “Optimizing SELECT and …”.

Ok, well that’s not what I want, there is a suggestion box to the right so I pick the top option “mysql select”. Not only is this worse with “Type Conversion in …”, “Searching on Two Keys” I also get 3 totally useless “Keymatch” records

Download MySQL -http://dev.mysql.com/downloads/
MySQL Training – http://www.mysql.com/training/ KeyMatch
Buy MySQL Enterprise -http://shop.mysql.com/enterprise/

I know in the past just entering ‘SELECT’ worked, because I’ve been presently writing tests on JOIN syntax and I wanted to link in my blog reference.

Images of my searches.