You are here: Home / Archives for Technologies

QMail Admin

by

For some reason my vqadmin failed to create a new mail domain?? I know I had some /home permissions problems, but that was quite some time ago, perhaps I haven’t added a domain for mail in that time. Anyway, time to refresh my skills and software.

vqadmin

Home Page

cd /src
wget http://www.inter7.com/vqadmin/vqadmin-2.3.2.tar.gz
tar xvfz vqadmin-2.3.2.tar.gz
cd vqadmin-2.3.2
./configure --enable-cgibindir=/home/vqadmin/cgi-bin --enable-htmldir=/home/vqadmin/www
make && make install-strip

vi /opt/httpd/conf/httpd.include

        deny from all
        Options ExecCGI
        AllowOverride AuthConfig
        Order deny,allow


vi /home/vqadmin/cgi-bin/vqadmin/.htaccess
AuthType Basic
AuthUserFile /home/vqadmin/.passwd
AuthName vQadmin
require valid-user
satisfy any

chown nobody /home/vqadmin/cgi-bin/vqadmin/.htaccess
chmod 644 /home/vqadmin/cgi-bin/vqadmin/.htaccess
htpasswd -bc /home/vqadmin/.passwd admin admin_password
chmod 644 /home/vqadmin/.passwd
apachectl stop
apachectl start

All fixed. For a step by step guide to all Qmail software I recommend http://qmailrocks.org

Using Cascade in Foreign Keys

by

For those that are using a Referential Integrity based Storage Engine in MySQL, the use of Foreign Keys can provide an extra level of data integrity within your application. There are of course caveats to this statement, such as correct design, proper transactional use etc, but that’s a topic for another time. The use of CASCADE has an effect on this.

So, just a quick refresher, using the MySQL Sakila Sample Database (if you don’t have it, get it. This can be a valuable tool in many other areas). Let’s first check out the Schema Diagram. We will look at the first relationship in the top left corner of the diagram.

There is a defined relationship with the actor and film_actor tables. An actor record can belong to one or more film_actor records, and a film_actor record must have one corresponding actor record.

Schema Definition

CREATE TABLE actor (
  actor_id SMALLINT UNSIGNED NOT NULL AUTO_INCREMENT,
  first_name VARCHAR(45) NOT NULL,
  last_name VARCHAR(45) NOT NULL,
  last_update TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY  (actor_id),
  KEY idx_actor_last_name (last_name)
)ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE film_actor (
  actor_id SMALLINT UNSIGNED NOT NULL,
  film_id SMALLINT UNSIGNED NOT NULL,
  last_update TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY  (actor_id,film_id),
  KEY idx_fk_film_id (`film_id`),
  CONSTRAINT fk_film_actor_actor FOREIGN KEY (actor_id)
         REFERENCES actor (actor_id) ON DELETE RESTRICT ON UPDATE CASCADE,
  CONSTRAINT fk_film_actor_film FOREIGN KEY (film_id)
         REFERENCES film (film_id) ON DELETE RESTRICT ON UPDATE CASCADE
)ENGINE=InnoDB DEFAULT CHARSET=utf8;

Explanation

The above shows the CREATE TABLE syntax as provided in Version 0.8 of the Sakila Sample Database. Looking specifically at the constraint we wish analyse.

  CONSTRAINT fk_film_actor_actor FOREIGN KEY (actor_id)
         REFERENCES actor (actor_id) ON DELETE RESTRICT ON UPDATE CASCADE,

The FOREIGN KEY for a specific column REFERENCES an appropiate table and column, and includes optional UPDATE and DELETE actions.

The ON DELETE RESTRICT is indeed optional as this is the default operation, but is a good practice for both readibility and future compatibility.
The ON UPDATE CASCADE in this case, is indeed unwarranted as the actor primary key is an AUTO_INCREMENT column and should never be updated. By good design definition, a PRIMARY KEY should never be updated in a table, and then should be set to RESTRICT. If however you ever wish to update a primary key value in RESTRICT mode, you can’t as to first UPDATE children before the parent is to set the value to an unknown value that values validation. The best approach here is always use AUTO_INCREMENT primary keys, and never update the values.

Use

So let’s take the current implementation for a spin to see how the default RESTRICTED implementation operates.

mysql> select count(*) from film_actor where actor_id=1;
+----------+
| count(*) |
+----------+
|       19 |
+----------+
1 row in set (0.05 sec)
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> delete from actor where actor_id=1;
ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails
    (`sakila/film_actor`, CONSTRAINT `fk_film_actor_actor` FOREIGN KEY (`actor_id`) REFERENCES `actor` (`actor_id`)
   ON UPDATE CASCADE)
mysql> rollback;
Query OK, 0 rows affected (0.06 sec)

As you can see, an attempt to delete an actor fails due to a foreign key constraint failure.

The correct implementation for this design is to delete child relationships first. For example.

mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> delete from film_actor where actor_id=1;
Query OK, 19 rows affected (0.00 sec)
mysql> delete from actor where actor_id=1;
Query OK, 1 row affected (0.00 sec)
mysql> rollback;
Query OK, 0 rows affected (0.06 sec)

Let’s now try CASCADE.

mysql> alter table film_actor drop foreign key fk_film_actor_actor;
Query OK, 5462 rows affected (0.78 sec)
Records: 5462  Duplicates: 0  Warnings: 0
mysql> alter table film_actor add CONSTRAINT `fk_film_actor_actor` FOREIGN KEY (`actor_id`) REFERENCES `actor` (`actor_id`) ON DELETE CASCADE;
Query OK, 5462 rows affected (0.69 sec)
Records: 5462  Duplicates: 0  Warnings: 0
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> select count(*) from film_actor where actor_id=1;
+----------+
| count(*) |
+----------+
|       19 |
+----------+
1 row in set (0.01 sec)
mysql> delete from actor where actor_id=1;
Query OK, 1 row affected (0.00 sec)
mysql> select count(*) from film_actor where actor_id=1;
+----------+
| count(*) |
+----------+
|        0 |
+----------+
1 row in set (0.00 sec)
mysql> rollback;
Query OK, 0 rows affected (0.00 sec)

As you can see, by changing the definition to CASCADE, the requirement to delete all child rows is nullified.

Considerations

When designing I try to avoid the use of CASCADE. While it has it’s benefits, it is a “catch” by the database, and can hide poor design and coding techniques. In can however in more heavily loaded systems, provide a performance improvement, as additional commands for deleting don’t incur the network communications overhead. I’ll need to add this as a note to my performance testing guidelines to see what effect this has.

Another benefit on a more strict approach is MySQL for example currently does not fire triggers resulting on cascading constraint operations.

Try it yourself

As you can see it’s easy to use. To try it yourself.

  • Use an appropiate storage engine, e.g. InnoDB. You can use ALTER TABLE [table] ENGINE=InnoDB; on your tables.
  • Add an index to the target (foreign key) column, your source (primary key) will have an index by definition of being a primary key.
  • Ensure your source and target columns are the same data type
  • Create the appropiate constraint.

While the MySQL Manual refers to both the source (primary) and target (foreign) columns must have Indexes, I’ve found this not to be the case in 5.1. An ALTER TABLE will auto create the target (foreign) columns index if it doesn’t exist.

Just note, traps for young players. Both the source (primary) and target (foreign) columns must have the identical datatype definition. A common problem I’ve seen is that UNSIGNED is used for a Primary Key definition and not for a foreign key definition.

References
Wikipedia definition of “Referential Integrity”
MySQL Manual Page – FOREIGN KEY Constraints
MySQL Sakila Sample Database Forum
MySQL Forge Wiki on Sakila Sample Database

Guidelines for managing embedded external project dependencies

by

I’ve yet to find any Java project that doesn’t have dependancies on some other Open Source external libraries. I’ve yet to find a Java project that manages these external dependencies appropiately for support and integration at an enterprise level.

As with most projects, understanding an applying sound principles that scale will help you at a later date, and generally the cost of implementation is minimual at the start, but of course becomes more expensive when it’s really needed. The classic case is Version Control. For over 10 years, even on small single developer projects, I’ve used Version Control, it should be taught at university as an introduction to good programming design, it would greatly benefit software development and maintenance.

Back onto the topic of hand. Let’s use a moderate Java Web Based application, and for the purposes of this discussion the following Open Source external libraries are incoporated. Log4J, JUnit, Canoo WebTest, MySQL JDBC, Apache Commons (Collections, DHCP, Pool, HTTPClient, Taglibs Mailer). I could continue, but this will suffice for the demonstration.

It’s very easy for your project to include the appropiate jar’s such as (log4j.jar, junit.jar, commons-pool.jar etc), however this is where support and integration with other products fall down.

A Controlled Approach

You need to keep a seperate repository (under source code control of course) of your external libraries, and this becomes the source across all corporate projects. This is to include the following for each library:

  • The actual deployed jar
  • The matching source code of the deployed jar
  • Java Documentation of deployed jar

Versioning

Log4J is an example of an Open Source project that does version their jars, will many other open source projects do not. Why don’t they? Well one reason is to enable people to upgrade easily but simply overriding existing versions, and processes that have specific CLASSPATH’s are not affected. Generally today, implementations of software include all jar’s within a specified directory so I don’t see the problem.
Log4J gives in this example a log4j-1.2.12.jar for deployment purposes. When libraries do not include a version number, the are to be specifically added. This adds another small delemma of standards. The general practice is to use the hyphen ‘-‘. followed by the product version using the full-stop ‘.’, however there are projects that don’t follow this.

Version Recording

So now we have for each external library, an appropiately versioned jar, and matching source and documentation. This is the initial baseline. What’s needed is a simple HTML index that manages this information for use. The Index should include:

  • Product Name
  • Product URL
  • Repository Version
    • Version Number
    • Version Date
    • Download Date
  • Latest Version
    • Version number
    • Date
    • Comment

You may ask, why do you record the Latest Version, when the practice should be to always get the latest version. JUnit is a good example, the present version 4.x, requires a JDK 1.5.x deployment, and if your application is running only 1.4.x, then you can only use older 3.8.x versions.s mentioned earlier,

Management

Having an index of external libraries is one thing, correct use and management is the most important step. Let’s assume we have taken the time to download and document the required libraries from our example, and everything has been deployed into our first project.

Now, a 2/3 month task of checking for updated versions can be scheduled. Withing this process, newer versions can be downloaded and recorded appropiately. In our example, Log4J now has 1.2.13. Updating the external libraries repository is the simple part, the next step is to notify all coporate projects of the new version, and to encourage uptake. This may not always occur in a timely manner, but with at least this baseline in place and when there are issues, standardisation on the known coporate version is the first step.

Dependencies

Within each project libaries, a readme that details which versions of which external jars are included andwhen they were last updated from the repository should be done. Noting this information with the both the external libraries repository and the project repository provides a paper trail. In addition, should there be any exceptions, this is the place this information can be reported.

External Projects

Canoo WebTest is a good example of an external project that also includes other external libraries such as Log4J, JUnit, HtmlUnit, NekoHtml.
Problems arise when these products may use and implement older or unknown versions of libraries.

Internal Projects

Having internal projects that are dependent on other internal projects is nothing new in a large corporate enterprise. The problem arises when a spaghetti of undocumented dependancies causes a management nightmare. Let’s take this real life example.

Product A has included jars of Product B, Product C, Product D and Product E. The Product C actually has a different version of Product D embedded within in. Product D which now is included twice also includes Product E, so there are now three copies of this, all are different in size, and all a binary only with no version numbers, and no corresponding matching source code. Does this sound bad? Well it is. How it ever worked is still amazing.

This mess could have been managed first with Version Control (a basic 101 in software development), and an appropiate management of external libraries, and a similar approach to internal libraries.

An Example

This is a great example to highlight the cost in lack of appropiate management. I was supporting an existing large scale project (1000+ users) (let’s call this Product A), and the integration of a new project (let’s call this Product B) had been passed from the development team for implementation, testing and release. A threaded process, it would simply just hang after some initialisation, no notice, no errors, just nothing. Not withstanding that something should have been better reported for the errors. Due to 7 possible log files between the software application and the application server, nothing was reported, but that’s another topic.

The final result was Product B had introduced the use of org.apache.log4j.Logger.trace(), a new more granular logging then debug(). The appropiate Log4J jar had been included in the product, and this was Version 1.2.12. However, Product A, which was using Product B, was bundled with an earlier version of Log4J, Version 1.2.8, and this version didn’t support this new method.

While it took a few hours of debugging to find this problem, it was made easier because at least these jar’s were version, of the 20-30 jars across products only 3-4 were versioned. Similar problems with QName and XMLBeans unamed jars prior to this took days to resolve (indeed one had to be worked around as it couldn’t be resolved).

A further complication in this process was when Product B was introduced. This was developed and built under Linux, while Product A was still being maintained under Windows. From the experience of Integration is was found that the order of loading within the classloader of a commerical application server differed between operating systems.

What constitutes a good error message to the user?

by

Today, will go down in my professional history as quite possibly the lowest I would ever think of a software developer. I’ve carefully avoided the term “fellow coder”, speaking of a IT industry sticking by fellow IT people, but not today.

I presently support an existing production system, some 1000+ users that’s been in place around 3 years in which I’ve had no prior involvement until recently. Integration with other external parties within the system have provided an esclation in errors being reported in this external communication, and lack of adequate feedback to the user is another topic. Email is the means of reporting administratively of user errors, again another topic of issue. Within these emails, which are almost impossible to manage due to the limited internal GUI only toolset and lack of access to actual email account files to automate scripting (yet another topic? Do you see a trend here), is some relevent information regarding the transaction, and then most importantly the error message. The thing I need to investigate the problem.

The line reads (removing some stuff):


Error Code: 234567892

Ok, well a little cryptic, but surely you can work out from the external system what this means. Investigation of some more errors, in the mail GUI product, yet another series of open windows (you can’t view messages like a regular mail client with a summary list and a detail panel), provides for a trend in the error messages:


Error Code: 1235492567
Error Code: -434783134
Error Code: 34345199

The trend being there is none. Of course today by mid morning the email error count is into the hundreds, and I’m none the wiser. Well time to closely investigate the code management (as I’ve already contacted the external party, and asked if I can provide some error codes to receive greater information).

The following constitutes the two lines of code taken in the determination of the error messages I’ve shown so far. Note, this code takes the external system response, and then “attempts to determine usefull error content for presentation back to the user”.


errorNo = new Random().nextInt();
error = “Error Code: ” + errorNo;

Now while everybody laughed out loud, including fellow developers, DBA, IT manager, the Business Owners and Users (which can’t read code but could understand the first of these two lines I highlighted), and yes it really was funny in context with a bigger picture, but really it wasn’t funny at all. Some things make my blood boil, and this was one of them. With all the time lost between multitudes of people, users, call centre etc, I’d never felt a stronger conviction to hunt down the developer that wrote this.

The end of the story is after even trolling old CVS repository entries I was unable to piece sufficient information to determine the author. Most likely done pre version control, and then that trail leads to only a few names I’ve heard mentioned before.

I’d like to see somebody top that one!

Linux One Liner – Finding Stuff

by

Let’s say you created a file in your home directory but can’t work out which directory you put it in.


$ find ~ -name somefile.txt

You can replace ~ (tilda) with a directory e.g. / (slash) for search all locations on your system.

Let’s say you want to find all the JPEG’s you have.

$ find ~ -name "*.jpg"

Now to improve on this, I know I put a JPEG somewhere in the past few days, give me just the files from the past 3 days.

$ find . -name "*.jpg" -mtime -3

And what if you only wanted files greater then 2MB

$ find . -name "*.jpg" -mtime -3 -size +2M

If you want to look at a more detailed listing of the files found, like the format you are familar with using ls, try this.

$ find . -name "*.jpg" -mtime -3 -exec ls -l {} ;

You can find more then files, lets find all the directories you have.

$ find ~ -type d

I haven’t added it, but historically you needed to add -print on the end to print the results, seems nowadays this is optional.

I briefly used the -exec option above, I used it for various purposes. Here are a few.

$ find /backup-location -name "backup.*.tar.gz" -mtime +3 -print -exec rm -f {} ;
$ find . -name CVS -exec rm -rf {} ;

The first I run against my backup directory, that removes the online backups older then 3 days. Of course I’ve also got offline backups.
The second is when I checkout stuff from CVS and I want to prune all the CVS information. NOTE: Using the rm -rf command is very dangerous, you should only use this when you know your stuff. Used in the wrong way you delete everything, if you don’t have backups, there ain’t any UNDO in Linux. Also if you do it as root, you can effectively kill your installation in a split second.

There are other commands that perform various level of finding (e.g. commands via path) and other various stuff. A topic for another time, but to entice you.


$ which find
$ whereis find
$ locate find

Linux One Liner – Parsing long HTML urls

by

Ever wanted to look at a long HTML URL more easily, say to investigate a parameter. Here is a search from MapQuest.

http://www.mapquest.com/maps/map.adp?formtype=address&addtohistory=&address=10%20Market%20St&city=San%20Francisco&state=CA&zipcode=94111%2d4801&country=US&geodiff=1


$ echo "[insert url here]" | | tr "&?" "n"

This produced for the above URL the following output.

http://www.mapquest.com/maps/map.adp

formtype=address
addtohistory=
address=10%20Market%20St
city=San%20Francisco
state=CA
zipcode=94111%2d4801
country=US
geodiff=1

The Translate command tr does however replace both the & and ? characters. There are of course many more approaches like.


echo "[insert url here]" | sed -e "s/&/\n/g" -e "s/?/\n/g"

You can easily preserve the & and ? characters extending the syntax with

echo "[insert url here]" | sed -e "s/&/\n&/g" -e "s/?/\n?/g

This produces.

http://www.mapquest.com/maps/map.adp

?formtype=address
&addtohistory=
&address=10%20Market%20St
&city=San%20Francisco
&state=CA
&zipcode=94111%2d4801
&country=US
&geodiff=1

Now don’t get me started with the awk command. One of my popular books is Sed & Awk. If you do any detailed Shell scripting, this is a very handy guide.

Linux One Liner – Security

by

Here are a few useful one liners for Linux Security. View current packet filtering rules. (i.e. what can and can’t access your computer.

$ iptables -L

On older distros, iptables may not be in place. Try ipchains. A good reference and tools on iptables can be found at www.iptablesrocks.org.

Identity open ports on your installation using the Network exploration tool and security scanner.


$ nmap -p 1-65535 localhost

On my computer this returned

Starting nmap 3.70 ( http://www.insecure.org/nmap/ ) at 2006-06-11 12:22 EST
Interesting ports on lamda.arabx (127.0.0.1):
(The 65525 ports scanned but not shown below are in state: closed)
PORT STATE SERVICE
22/tcp open ssh
25/tcp open smtp
111/tcp open rpcbind
139/tcp open netbios-ssn
445/tcp open microsoft-ds
631/tcp open ipp
901/tcp open samba-swat
8005/tcp open unknown
32769/tcp open unknown
34315/tcp open unknown

That’s a cause for a bit of concern. Will need to look into that more.

Looking into more detail, I know what runs samba-swat but let’s confirm.


$ fuser -n tcp 901

This provides a confirmation and the Process id of the process using this port. A more susync output would be.

$ ps -ef | grep `fuser -n tcp 901 | tail -1 | cut -d: -f2` | grep -v grep

This gives me.

root 3356 1 0 Jun10 ? 00:00:00 xinetd -stayalive -pidfile /var/run/xinetd.pid

Which is exactly right, Samba Swat (the web interface for Samba) which you access at http://localhost:901 is configured using xinetd.

Now to investigate some ports I didn’t know were open.

Linux One Liner – Using the manual

by

For users of Linux regardless of the skill level, using the OS manual is invaluable. Frank gives an example using crontab at Viewing a specific version of a man page, but as with Linux there is always more then one way to skin a cat.

To view a man page of a command e.g. du.

$ man du

The Unix Manual is generally broken down into 9 sections, and sometimes a manual page is in multiple sections. These section are:

  • Section 1 – Commands
  • Section 2 – System Calls
  • Section 3 – Library Calls
  • Section 4 – Special Files
  • Section 5 – File Formats and Conversions
  • Section 6 – Games for Linux
  • Section 7 – Macro Packages and Conventions
  • Section 8 – System Management Commands
  • Section 9 – Kernel Routines

As in Franks example, crontab is in both Section 1 and 5. Crontab tab the Linux Command, and the file format used for crontab. To get access to the later.

$ man -s 5 crontab

Frank made reference to a syntax of man crontab.5 which didn’t work in my distro, so again, different implementations may be possible.

Say you remember the command associated with cron but not the full name. You can search the man pages with.

$ man -k cron

This produced in my distro.

/etc/anacrontab [anacrontab] (5) - configuration file for anacron
anacron (8) - runs commands periodically
cron (8) - daemon to execute scheduled commands (ISC Cron V4.1)
crontab (1) - maintain crontab files for individual users (ISC Cron V4.1)
crontab (5) - tables for driving cron (ISC Cron V4.1)
hinotes (1) - Syncronize your Hi-Notes database with your desktop machine. Hi-Notes must be installed on your Palm handheld (and at least one entry must exist within Hi-Notes)
read-todos (1) - Syncronize your Palm ToDo application's database with your desktop machine

Of course you should not discount that a manual page exists for the man command.

$ man man

Linux One Liner – Calculating Used Diskspace

by

You can easily see the state of diskspace used with the command.


$ df

However, often you want to know where most of the diskspace is being taken. A high level summary can be performed with.

$ du -k -s /* | sort +0nr -1

Producing results like.

23450208        share
9369212 home
3803504 usr
2395876 var
2015380 opt
920121  proc
815476  src
...

A more indepth review of the worst offending directories can be done with.


$ du -k / | sort +0nr -1 | head -30

This view does however show all offending directories so you normally have to ignore the higher levels as the are inclusive of the more specific directories where the most diskspace is.

You get a result like

47642425        /
23450208        /share
9799580 /home
9153228 /home/rbradfor
8497152 /share/bittorrent
7065840 /share/bittorrent/Stargate.SG-1.Season.9
4986368 /home/rbradfor/vmplayer
4837136 /usr
3659200 /opt
2559836 /home/rbradfor/vmplayer/ultimateLAMP
2447692 /var
2426364 /home/rbradfor/vmplayer/ultimateLAMP-0.1
2377732 /usr/lib
2335428 /var/lib
2213440 /var/lib/vmware
2213432 /var/lib/vmware/Virtual Machines
2174928 /share/lib
2174912 /share/lib/vmware
2174896 /share/lib/vmware/Virtual Machines
1972900 /home/rbradfor/download
1945576 /var/lib/vmware/Virtual Machines/XP Pro Dell 5150
1868016 /share/UltimateLAMP
1604032 /usr/share
...

References

  • df – report filesystem disk space usage
  • du – estimate file space usage
  • sort – sort lines of text files

MySQL Ideas

by

Seems I have over time, thought of many ideas, jotted some notes on some, and even done some work, but everybody knows that “home projects” can take a long time.

Here are a few that have resurfaced over the past month, and I doubt I’ll ever get to them, or perhaps some other enterprising person has already done a similar thing. Of course most are for my own personal/professional gratification, but input from others of “great idea, when do we see it” could sway my interests.

INFORMATION_SCHEMA for MySQL Version 4

Why?

Well, quering the INFORMATION_SCHEMA is very cool, and long overdue for information gathering including statistics, schema definitions, schema version comparision tools etc. Of course there are concerns regarding the performance of using the INFORMATION_SCHEMA, and any design should significantly consider this limitation. The INFORMATION_SCHEMA is not physically represented as a normal MySQL database with tables and real data files, and then can be optimised appropiately, queries reference various sources of information and return the results at query time.

How would I achieve this? Here is an approach. The biggest issue would be what programming technology and non-Linux support.

  • Generate a physical INFORMATION_SCHEMA create SQL script from the most current database version. This would allow you to create a real database with real tables.
  • Population. Well at least some tables will be empty, but some work would be required to the population of other information. A first pass with some easy shell scripting and SHOW commands would easily populate information for SCHEMATA (SHOW DATABASES), TABLES (SHOW TABLES), COLUMNS (DESC [table]). Some are covered by the mysql schema. You get the picture. Of course some information will be more more difficult to populate easily.
  • Syncing. Populating is one thing, but if the information is not current then it becomes useless. Depending on your organisations management, in a controlled environment, simply re-running the population process when schema information is changed, or run nightly may work. A more brut force approach may be to look at filesystem timestamps of the database table definitions and trigger when changes occur, but I suspect this would a poor approach.

Storage Engines in use

I’ve often wondered, do people use X storage engine.

I would love to see plastered on the MySQL Home Page stats of 99,999,999 Downloads, 99,999,999 Installations reporting in, and then some breakdowns of database versions, storage engine usage, even demographics. This type of statistical information can be great, but can also be dangerous to competitors, or can be too detailed to become useful. Consider the objective it a general census.

I recently was introduced to Fotolog, and in the top left corner is Members, Photos, and Photos today. Of course now they have volume the numbers are impressive. You blink, do a refresh and they have changed.

Having a more general information goal approach, rather then detailed would be my guidelines. Here are some points.

  • Each MySQL installation is registered providing it a unique id. This is used for all collection and communication, and there is never any real reference back to organisation/name etc. How this works correctly is a little difficult, but would need to involve some checksuming of information like hostname,os,mysql version,maybe ip.
  • A script runs to generate basic statisical information, including unique installation id, present GMT date/time, MySQL version, master/slave, installation date, number of databases, breakdown of storage engines used per database/installation, total number of tables per database/installation.
  • The Information needs to be recorded, so it can be transmitted, and also used for historical comparision, best option would a flatfile on the filesystem.
  • Having an automated approach to then say email this information to a known server. There would need to be some means to have some authentication, and feedback to confirm success. Using email is not an ideal approach, but is a more readily available medium. Another easy means would be a webservices approach, but would require a more direct internet connection, while mail could overcome this means.
  • More advanced ideas would be to record information in and XML format, have a XML storage engine, and enable reports to be run against this collated information, so the System Adminstrator has at there finger tips historical details that are gathered. The benefit of an XML approach is this can be more easily collated with clever approaches with XSLT for example. The fallback would could be a CSV file, but the information being sent, may well need to be more structured for various reasons.
  • Of course if it was limited to Version 5 up, the benefits of stored procedures, events and UDF’s a lot of this could be implemented in MySQL specific technologies, but what about the bulk of installations pre Version 5, you would definitely what to include these.
  • My concerns would definitely be SPAM abuse, or false figures, so these points alone may nullify the process even being worth it.

This is just a broad stroke overview, I’ve got more detailed analysis of pro’s, con’s, example file layouts, and a means of collating the information and providing dynamic reporting on the server.

Of course there are a few ideas I have implemented, and I’m still passionate about and working on in that “spare time” we all never have.

ShowCase MySQL

A showcase of MySQL (via existing popular Open Source apps). I’ve done this via a VMWare Appliance called UltimateLAMP.

I did this for two reasons, first of course I wanted to easily show applications that could be used without any technical requirements (indeed the hardest part is installing the free VMware player), and second, if there was more penetration of possible MySQL options in my present home city of Brisbane, it may provide some MySQL work opportunities. Of course, this an application approach to demonstrating MySQL, rather then a MySQL features approach, and is dependent then on the community and these applications may be less then ideal in a large scale deployment or may not be of a quality to be placed with MySQL, but it’s one way to get leverage.

I would like some way to poll usage of MySQL within organsiations my home town, but it’s a very difficult approach, and I’m sure some organisations would not even know they are using MySQL. The process may also provide a means of providing a service, say quarterly updates of software (MySQL and open source products), customisations, but I think it would be a difficult sell. Having an army of people applying these ideals across a number of cities may provide a more marketable approach.

ACID Performance Benchmark

Leveraging the Apache project JMeter, which has all the infrastructure for handling theads, reporting, graphing, and volume load testing, I’ve written to enhancements for JMeter JDBC Sampler, the first was CALL support, enabling calls to stored procedures. The second is TRANSACTION support enabling a number of SQL statements to be execute in a true transaction, all pass or all fail.

These allow me to provide an application specific performance approach. Now MySQL has various means of testing, the mysql-test that comes with installations (I’ve got more on this for a later time), as well as mysqlslap (again, some more points on this at a later time). Both of these tools have benefits, but I wanted a more application specific approach. Using JMeter provides an overhead, but it provides a more realistic application approach, my goal is not to get the maximum thoughtput possible, being as close to the screws, but to provide a more realistic approach.

Of course, my objective is to use application specific SQL, the SQL you run, this would need to be gathered via various means. Information gathering including what types of transactions, the interactions, the volumes of use would be needed to create realistic approaches. The benefits of JMeter for example, is I can simulate various transactions, enable them to interact, enable them to have implied delays to simulate user use. These are some of the advantages over the current MySQL performance approaches.

Providing this level of simulation however is only half the requirement. This provides for a benchmark for performance, but what about validation. Sure the system can be used with 100 threads with X Insert Orders, and Y Select Orders, and Z Invoice Prints etc per thread, but what if the there were issues and errors.

Data Quality is a key interest and this level of benchmarking also has to have the extent of validating the results, and taking appropiate actions for failures. I’m planning also that this can be used by enabling Referential Integrity (via an appropiate storage engine) on an existing application and then confirming transactional state of queries within applications that are not presently transaction safe.

Presently, I using this approach to assist in the testing of the MySQL Community Storage Engine PBXT. I’m hoping in conjunction with the Sakila Sample Database and various data sets, that I can also publish comparision results with other Storage Engines.

Schema Designer Analyser

My background is database modelling and large systems design, and I’m writing a A MySQL Guide to Database Modelling, Design and Support (still under development). This is an online and very simple approach for open source development to improve it’s image in database modelling. Keeping things simple will help all those non-professionals out there.

To back up this article, which has a checklist for organisations to follow and then incoporate into standards, is a tool to do the analysis that the checklists requires.

I had hoped to initial develop this using the GRT portion of the MySQL Workbench, but I’ve decided to make that a secondary project for now, focusing on a tehnology of greater understanding, possibly Java. Of course the biggest issues is accurate gathering of schema information, so you can see why a INFORMATION_SCHEMA for MySQL version 4 would come in handy.

Conclusion

I’ve got a lot more ideas, and discussions with other MySQL peers rekind ideas I’ve thought of previously. I really should get to putting them all out there to see how many people think I’m crazy, or to find other crazy people to contribute, or even fund! One can dream.

How I work

by

My work life is really fragmented at present, so I’ve decided a split approach in answer to Dave Rosenberg’s How I Work–what I have learned so far .

What is your role?
Support Developer providing sole support an internal client web based system (Java, Oracle) with 1000+ users producing >$1m daily revenue. Independent Database Consultant. Specialising in Database Modelling, Large Systems Design and Web Development Technologies. Strong proponent of Agile Development Methodologies (specifically Extreme Programming – XP).
What is your computer setup?
Dell Optiplex GX280 (P4, 1GB, 17″ CRT) running Redhat FC4.
(When I started only 2 months ago I got lumped with a similar speced machine running Windows XP, and upteen windows apps (like 4 different versions of TOAD for example) . This OS was contry to my understanding provided in the interview process. 		Primary – Dell Inspiron 5150 (P4/3.2GHz, 1GB, 120GB, 15″ + 21″ CRT)
Secondary – Generic (P3/600MHz, 1GB, lots of HDD ~500GB, 17″ CRT)
Both with very impressive Saitek backlit keyboards (one red, one blue) Great at night. See image at bottom.
What desktop software applications do you use daily?
RedHat Fedora Core 4
Eclipse 3.1, FireFox 1.5, Thunderbird 1.5, WebSphere Application Server 5.1, J2DK 1.4.2, Oracle SQL Developer, Open Office 2, XMMS and SSH client (which I use most)
Due to legacy internal systems and support I also must run under Wine (Internet Exploder), and First Class (email client).
Not to stop there, I also must run under Citrix ICA Client apps (FocalPoint, Heat Call Logging, and Microsoft Word for internal forms that won’t work under OO2.) And before somebody suggests why not try VMWare or other clients, I have tried, but software like Focalpoint can’t install? 		CentOS 4.3
FireFox 1.5, ThunderBird 1.5, Gaim, SSH, Skype, Open Office 2.
Maybe not all of these every day, but some combination of each day –> MySQL 4.1, MySQL 5.0, MySQL 5.1, MySQL Workbench, Eclipse 3.1, J2DK 1.4.2, J2DK 5.0, Apache Tomcat 5.0.28, Apache Httpd 2.0.53, JMeter.

Presently also configuring a new laptop drive running Ubuntu Dapper 6.06 RC. For the Record, Beta 2, Flight 6 and Flight 7 all failed.
What websites do you visit every day?
Internal Wiki (all day)
At Lunch Google Sci-Tech News
PlanetMySQL 		Google Sci-Tech News
PlanetMySQL (if I haven’t already got to them).
What mobile device or cell phone do you use?
N/A NEC e606 3G phone (which I’ve had for probably 3 years now) operating on a 3G network with video calls, again for 3 years.
Do you use IM?
No. All access is blocked other then an internal Jabber server, that I use rarely, never for communication, just a cut/paste of command or syntax. Speaking of blocked, SSH access to my production servers is blocked, and even reading news like Skypes Call Out is blocked by WebSense. Extensively, however due to current employement policy, I’m very hamstrung unless before/after hours.
I use Gaim as I have AIM, MSN and Google Talk accounts, and Skype. My preference was always AIM, but as clients come and go, I’ve had to accumulate more accounts.
Do you use a VoIP phone?
No. Not at present, however for many years I worked with US clients and used Packet 8. Still have the hard phone somewhere.
I’ve also used Skype talk for one or one or conference calls. Of late in Australia to New Zealand and Singapore. Indeed, the quality to Singapore has been excellent, when living in the US, calls to Singapore on Skype were clearer then my Packet8 phone to US numbers.
Do you have a personal organization/time management theory?
Current contract employees a number of disjointed methods, which in observation just shows so much inefficency, it’s worth documenting just to highlight what not to do. We have daily team meetings (10 mins), each listing your top 2 daily tasks. Weekly we have to also submit weekly goals. Weekly combined meetings with another team where we again give weekly top 2 tasks. We use two seperate systems (with manual double entry) for work identification, one for call centre logging issues, and Bugzilla for software bugs, and enhancements. We also use XPlanner (again duplicating a lot of tasks) for time management.
With all this rigid structure, I am daily given either other work to do, or investigate, and in over 3 months, I would rarely end a week anywhere near where it was so described at the start of the week. 		With all the technology possible, I actually do not have any electronic management gadget, never had. I use a combination of notebook, plain paper (usually for daily notes etc, which I discard regularly), a diary, and normally a lot of emails which I normally send to/from home.
Given that email is used so much, I basically use Draft Emails for any electronic notes.
Anything else?
Perhaps there is merit in How I work now, and How I’d like to work now.

Saitek Keyboard

A better VNC

by

I’ve been using VNCViewer from RealVNC under Linux to remote connect to an older machine running windows. Two reasons, I don’t need yet another screen on my desk, and I need windows to adequately test and use the MySQL GUI products, in particular MySQL Workbench.

I never realised there was a better way, particularly over a local network, a command called rdesktop which is capable of natively speaking Remote Desktop Protocol (RDP).

$ su -
$ yum install rdesktop

This did only give me version 1.3.1. The current version 1.4.1 available from the rdesktop Website does provide greater commands including the syntax I use.

su -
cd /src
wget http://optusnet.dl.sourceforge.net/sourceforge/rdesktop/rdesktop-1.4.1.tar.gz
tar xvfz rdesktop-1.4.1.tar.gz
cd rdesktop-1.4.1
./configure
make
make install

Example Usage

$ rdesktop -u <username> -p <password> -K -N -z -a 16 -x b -P

Some additional common options include -d <domain> and -f for fullscreen.

Comand Syntax

$ rdesktop
rdesktop: A Remote Desktop Protocol client.
Version 1.4.1. Copyright (C) 1999-2005 Matt Chapman.
See http://www.rdesktop.org/ for more information.

Usage: rdesktop [options] server[:port]
   -u: user name
   -d: domain
   -s: shell
   -c: working directory
   -p: password (- to prompt)
   -n: client hostname
   -k: keyboard layout on server (en-us, de, sv, etc.)
   -g: desktop geometry (WxH)
   -f: full-screen mode
   -b: force bitmap updates
   -L: local codepage
   -B: use BackingStore of X-server (if available)
   -e: disable encryption (French TS)
   -E: disable encryption from client to server
   -m: do not send motion events
   -C: use private colour map
   -D: hide window manager decorations
   -K: keep window manager key bindings
   -S: caption button size (single application mode)
   -T: window title
   -N: enable numlock syncronization
   -X: embed into another window with a given id.
   -a: connection colour depth
   -z: enable rdp compression
   -x: RDP5 experience (m[odem 28.8], b[roadband], l[an] or hex nr.)
   -P: use persistent bitmap caching
   -r: enable specified device redirection (this flag can be repeated)
         '-r comport:COM1=/dev/ttyS0': enable serial redirection of /dev/ttyS0 to COM1
             or      COM1=/dev/ttyS0,COM2=/dev/ttyS1
         '-r disk:floppy=/mnt/floppy': enable redirection of /mnt/floppy to 'floppy' share
             or   'floppy=/mnt/floppy,cdrom=/mnt/cdrom'
         '-r clientname=': Set the client name displayed
             for redirected disks
         '-r lptport:LPT1=/dev/lp0': enable parallel redirection of /dev/lp0 to LPT1
             or      LPT1=/dev/lp0,LPT2=/dev/lp1
         '-r printer:mydeskjet': enable printer redirection
             or      mydeskjet="HP LaserJet IIIP" to enter server driver as well
         '-r sound:[local|off|remote]': enable sound redirection
                     remote would leave sound on server
   -0: attach to console
   -4: use RDP version 4
   -5: use RDP version 5 (default)

I haven’t reviewed the security implications but considering I’m only running in my own internal network, it’s not a major priority.

References

Improving Open Source Databases – WordPress

by

As part of both my UltimateLAMP side project, and my greater involvement with MySQL Workbench, I’ve been wanting to review and document the database schemas of Open Source products.

Indeed, as part of discussions with Mike Zinner of MySQL AB at the recent MySQL Users Conference, I suggested an idea used in the Java World, for example by Clover and Agitar, where to promote the usefullness of their respective tools (in this case code coverage), they provide results against Open Source Products. For an example check out Clover Examples.

With MySQL Workbench, to get some greater exposure of the use of graphical Data Modelling tool, I mentioned the idea of providing a respository of schemas, may help in exposure, usage and feedback of MySQL Workbench. Indeed my work is towards this being added to MySQL Forge, however I’ve found that at my first indepth work, using WordPress 2.0.2 has been a less then successful experience.

The schema would rate a 2 out of 5 on my scale of optimal database schemas. Here are some of my initial findings.

  • Inconsistent naming standards
    • ID uppercase for primary keys in some tables, but not in others.
    • different standards for primary keys, some id, some [table]_id
    • Inconsistent identification of foreign keys (e.g. post_parent). All keys should have _id.
    • Inconsistent datatypes for optional foreign keys (e.g. default of 0 rather then NULL)
    • Inconsistent column naming standards, some tables prefix columns with table name or table like name, (e.g. post_), but not all columns. Some tables has multiple prefix values (e.g. cat_, category_)
  • Poor usage of datatypes. Examples include BIGINT(20) for primary keys, TEXT for post title, INT(10) for columns which as values a TINYINT can support.
  • Inconsistent use of enums (e.g. Y/N for some values, and yes/no for others)
  • No Referential Integrity support (ie. Foreign Keys), limiting the benefits of a modelling tool showing table relationships
  • Different rules for primary keys (most tables have a suggorate key, but not all)

I am actually working on A MySQL Guide to Database Modelling, Design and Support, which covers a lot of these points and more. I’m some way from completing this document, but this initial research of the WordPress product highlights it’s importance to the open source community.

Of course we can just make noise about what we don’t like, or we can do something about it. I’m submitting my ideas, suggestions and code changes to the project. What is done about it is outside of my control, but I’ve at least given my input to this Open Source Product.

WordPress 2.0.2 Schema SQL (taken from a mysqldump after installation)

Here is a graphical representation of the Data Model from MySQL Workbench.
NOTE: My MySQL Workbench will not enable me to do a Export Image, so I’ll need to add this at a later time

Introducing Referential Integrity

Unfortunately this is no automated means of creating appropiate Referential Integrity for a schema. With more strict naming standards, it would be possible to reconstuct this information generally.

The approach to create these new rules were performed manually, but a tool could be developed to provide approximations for a number of these.

# Implement Foreign Keys. Changes include:
#  - Appropiate Storage Engine
#  - ensure column is of same definition as matching primary key (e.g. datatype + unsigned)
#  - create index for foreign key
#  - ensure optional keys use NULL

ALTER TABLE wp_categories ENGINE=InnoDB;
ALTER TABLE wp_comments ENGINE=InnoDB;
ALTER TABLE wp_linkcategories ENGINE=InnoDB;
ALTER TABLE wp_links ENGINE=InnoDB;
ALTER TABLE wp_options ENGINE=InnoDB;
ALTER TABLE wp_post2cat ENGINE=InnoDB;
ALTER TABLE wp_postmeta ENGINE=InnoDB;
ALTER TABLE wp_posts ENGINE=InnoDB;
ALTER TABLE wp_usermeta ENGINE=InnoDB;
ALTER TABLE wp_users ENGINE=InnoDB;

ALTER TABLE wp_usermeta MODIFY user_id  bigint(20) UNSIGNED NOT NULL DEFAULT '0';
ALTER TABLE wp_usermeta ADD FOREIGN KEY usermeta_users_fk (user_id) REFERENCES wp_users(ID) ON DELETE CASCADE;

ALTER TABLE wp_posts MODIFY post_author BIGINT(20) UNSIGNED NOT NULL;
CREATE INDEX post_author ON wp_posts(post_author);
ALTER TABLE wp_posts ADD FOREIGN KEY posts_users_fk (post_author) REFERENCES wp_users(ID) ON DELETE CASCADE;

ALTER TABLE wp_posts MODIFY post_parent BIGINT(20) UNSIGNED NULL DEFAULT NULL;
UPDATE wp_posts SET post_parent = NULL WHERE post_parent = 0;
CREATE INDEX post_parent ON wp_posts(post_parent);
ALTER TABLE wp_posts ADD FOREIGN KEY posts_posts_fk (post_parent) REFERENCES wp_posts(ID);

ALTER TABLE wp_postmeta MODIFY post_id  bigint(20) UNSIGNED NOT NULL DEFAULT '0';
ALTER TABLE wp_postmeta ADD FOREIGN KEY postmeta_posts_fk (post_id) REFERENCES wp_posts(ID) ON DELETE CASCADE;

ALTER TABLE wp_categories MODIFY cat_ID BIGINT(20) UNSIGNED NULL DEFAULT NULL AUTO_INCREMENT;
ALTER TABLE wp_categories MODIFY category_parent BIGINT(20) UNSIGNED NULL DEFAULT NULL;
UPDATE wp_categories SET category_parent = NULL WHERE category_parent = 0;
ALTER TABLE wp_categories ADD FOREIGN KEY categories_categories_fk (category_parent) REFERENCES wp_categories(cat_ID);

ALTER TABLE wp_post2cat MODIFY rel_id BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT;
ALTER TABLE wp_post2cat MODIFY post_id BIGINT(20) UNSIGNED NOT NULL;
ALTER TABLE wp_post2cat MODIFY category_id BIGINT(20) UNSIGNED NOT NULL;
ALTER TABLE wp_post2cat ADD FOREIGN KEY post2cat_posts_fk (post_id) REFERENCES wp_posts(ID) ON DELETE CASCADE;
ALTER TABLE wp_post2cat ADD FOREIGN KEY post2cat_categories_fk (category_id) REFERENCES wp_categories(cat_ID) ON DELETE CASCADE;
ALTER TABLE wp_comments MODIFY user_id BIGINT(20) UNSIGNED NULL DEFAULT NULL;
UPDATE wp_comments SET user_id = NULL WHERE user_id = 0;
CREATE INDEX user_id ON wp_comments(user_id);
ALTER TABLE wp_comments ADD FOREIGN KEY comments_users_fk (user_id) REFERENCES wp_users(id) ON DELETE CASCADE;
ALTER TABLE wp_comments MODIFY comment_post_ID BIGINT(20) UNSIGNED NULL DEFAULT NULL;
ALTER TABLE wp_comments ADD FOREIGN KEY comments_posts_fk (comment_post_ID) REFERENCES wp_posts(ID) ON DELETE CASCADE;

ALTER TABLE wp_comments MODIFY comment_parent BIGINT(20) UNSIGNED NULL DEFAULT NULL;
UPDATE wp_comments SET comment_parent = NULL WHERE comment_parent = 0;
CREATE INDEX comment_parent ON wp_comments(comment_parent);
ALTER TABLE wp_comments ADD FOREIGN KEY comments_comments_fk (comment_parent) REFERENCES wp_comments(comment_id);

ALTER TABLE wp_linkcategories MODIFY cat_id BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT;
ALTER TABLE wp_links MODIFY link_category BIGINT(20) UNSIGNED NOT NULL;
ALTER TABLE wp_links ADD FOREIGN KEY links_category_fk (link_category) REFERENCES wp_linkcategories(cat_id);

While there are no column name changes, by default the WordPress code should operate with this revised schema. Issues would include a poor delete strategy that still voilates the liberal constraints now enforced. Special handling of 0 for optional columns when the value is now NULL may also be a problem.

Revised Data Model in MySQL Workbench.
NOTE: My MySQL Workbench will not enable me to do a Export Image, so I’ll need to add this at a later time

Introducing improved Standards

Of course, I’d like to see the schema improve, here are my first draft ideas for improvements of the schema.

# Naming Improvements ===================================================================================
# Rename to lowercase id
ALTER TABLE wp_users CHANGE ID id BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT;
ALTER TABLE wp_posts CHANGE ID id BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT;
ALTER TABLE wp_categories CHANGE cat_ID cat_id BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT;
ALTER TABLE wp_comments CHANGE comment_ID comment_id BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT;
ALTER TABLE wp_comments CHANGE comment_post_ID comment_post_id BIGINT(20) UNSIGNED NOT NULL;

# Include _id for all Foreign Keys
ALTER TABLE wp_posts CHANGE post_author post_author_id BIGINT(20) UNSIGNED NOT NULL;
ALTER TABLE wp_posts CHANGE post_parent post_parent_id BIGINT(20) UNSIGNED NOT NULL;
ALTER TABLE wp_categories CHANGE category_parent category_parent_id BIGINT(20) UNSIGNED NOT NULL;
ALTER TABLE wp_comments CHANGE comment_parent comment_parent_id BIGINT(20) UNSIGNED NULL DEFAULT NULL;
ALTER TABLE wp_links CHANGE link_category link_category_id BIGINT(20) UNSIGNED NULL DEFAULT NULL;

# Primary Key Name Standardisation
ALTER TABLE wp_posts CHANGE id post_id BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT;
ALTER TABLE wp_users CHANGE id user_id BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT;

# Foreign Key Standardisation
ALTER TABLE wp_categories MODIFY category_parent_id parent_id BIGINT(20) UNSIGNED NOT NULL;
ALTER TABLE wp_comments CHANGE comment_parent_id parent_id BIGINT(20) UNSIGNED NULL DEFAULT NULL;

# Other Column Standaisations
ALTER TABLE wp_categories MODIFY cat_name category_name VARCHAR(55) NOT NULL;

# Column width improvements
ALTER TABLE wp_categories MODIFY category_count MEDIUMINT(5) NOT NULL DEFAULT '0';
ALTER TABLE wp_posts MODIFY post_title VARCHAR(255) NOT NULL;
ALTER TABLE wp_linkcategories MODIFY cat_name VARCHAR(255) NOT NULL;
ALTER TABLE wp_linkcategories MODIFY cat_id TINYINT(3) UNSIGNED NOT NULL AUTO_INCREMENT;
ALTER TABLE wp_options MODIFY autoload ENUM('Y','N') NOT NULL DEFAULT 'Y';

# Obvious Index Improvements
# Make login a unique index
DROP INDEX user_login_key ON wp_users;
CREATE UNIQUE INDEX  user_login_key  ON wp_users(user_login);

These will require code changes and adequate testing, something I’m not hopeful would happen unless I drove it, but you never know, the next time I review the WordPress Schema I might be impressed.

Conclusion

This is just a broadstroke first review of the schema. As part of all good design, testing of the code, further review and refinement based on better understanding of the product, and also peer review are all important steps not yet undertaken.

Migrating an MyISAM schema to use Referential Integrity

by

Here are some steps involved. Using the current MySQL defacto engine InnoDB. Of course, Falcon, PBXT and others will enable alternative engines to be used.

Convert Table Storage Engine Types

$ mysql -u[user] -p[password] [database] -e "SHOW TABLES" | grep -v "Tables_in" | sed -e "s/^/ALTER TABLE /" | sed -e "s/$/ ENGINE=InnoDB;/" > upgrade.sql
$ mysql -u[user] -p[password] [database] < upgrade.sql

NOTE: This may not work for all tables, for example those with FULLTEXT indexes will fail.

For the introduction of Referential Integrity we need to ensure the following.

  • Each Foreign Key column should have an index. Prior to 4.1 I think this was a requirement, however it's a good general practice regardless for SQL performance.
  • The datatype must match between Primary Key and Foreign Keys. The most obvious oversight is normally UNSIGNED, however you also for example have INT and INT, and not INT and BIGINT as datatypes.
  • Optional Foreign Keys must contain NULL values and not the normal practice of having a default of 0.

A Sample Foreign Key Syntax.

mysql> ALTER TABLE wp_usermeta ADD FOREIGN KEY usermeta_user_fk (user_id) REFERENCES wp_user(id);
ERROR 1005 (HY000): Can't create table './wordpress/#sql-cd9_10.frm' (errno: 150)

A closer investigation of what this error really is:

$ perror 150
MySQL error code 150: Foreign key constraint is incorrectly formed

A confirmation of table definitions.

mysql> desc wp_usermeta;
+------------+---------------------+------+-----+---------+----------------+
| Field      | Type                | Null | Key | Default | Extra          |
+------------+---------------------+------+-----+---------+----------------+
| umeta_id   | bigint(20)          |      | PRI | NULL    | auto_increment |
| user_id    | bigint(20) unsigned |      | MUL | 0       |                |
| meta_key   | varchar(255)        | YES  | MUL | NULL    |                |
| meta_value | longtext            | YES  |     | NULL    |                |
+------------+---------------------+------+-----+---------+----------------+
4 rows in set (0.00 sec)

 mysql> desc wp_users;
+---------------------+---------------------+------+-----+---------------------+----------------+
| Field               | Type                | Null | Key | Default             | Extra          |
+---------------------+---------------------+------+-----+---------------------+----------------+
| id                  | bigint(20) unsigned |      | PRI | NULL                | auto_increment |
| user_login          | varchar(60)         |      | UNI |                     |                |
| user_pass           | varchar(64)         |      |     |                     |                |
| user_nicename       | varchar(50)         |      |     |                     |                |
| user_email          | varchar(100)        |      |     |                     |                |
| user_url            | varchar(100)        |      |     |                     |                |
| user_registered     | datetime            |      |     | 0000-00-00 00:00:00 |                |
| user_activation_key | varchar(60)         |      |     |                     |                |
| user_status         | int(11)             |      |     | 0                   |                |
| display_name        | varchar(250)        |      |     |                     |                |
+---------------------+---------------------+------+-----+---------------------+----------------+
10 rows in set (0.00 sec)

A second pair of eyes (thanks Jon), showed that I actually spelt a table name wrong. Did you spot it. Of course it would have been nice if the error message actually told me this. This rather bland message Can’t create table could actually mean.

  • missing index (pre 4.1)
  • incompatible columns data types
  • Invalid Table.

I’m sure if I tried to break it I’d find more examples, but just a trap for unsuspecting people.

Now migrating an existing schema to using Referential Integrity provides some initial benefits (row level locking, misplaced key updates/deletes) but it does not provide true integrity unless your application has been written to support transactions. Chances are it hasn’t, but this is at least the first step.

Restyling a Mediwiki Installation – Lesson 1

by

Following my implementation of UltimateLAMP, read heaps more at this thread, I undertook to provide customizations of a MediaWiki Installation. Here is the first lesson that you can undertake if you wish to beautify the default MediaWiki Installation.

For the purposes of this demonstration, I am going to help out Jay & Colin and propose a restyle the MySQL forge to fit in with the default Home Page. Hopefully you will see it there soon!

Lesson 1 – Updating the default Monobook.css

There are several different ways to make style changes, the simplest is to customize the system wide Monobook.css, and this Lesson will focus on this.

By accessing the link [http://my.wiki.site/]index.php/MediaWiki:Monobook.css you will be able to make the following changes.

The best way to approach this, like any good programming style, make small changes, testing and confirmation and continue.

Note: For all screen prints, click for a larger image

1. Cleanup Backgrounds

body        { background-image: none; background-color: #FFFFFF;}
.portlet    { background-color: #FFFFFF; }

The MediaWiki Page is made up of three sections, these are the represented by styles .portlet, #content, and #footer. For the purposes of our first example, the content section and the footer section are already white.

==>

At this point I should recommend that you use FireFox for future work. You should then install the Web Developer Add-on. What results from this plugin is invaluable information about the internals of a web page. The two options most useful for this exercise is:

  • Information | Display Id and Class Details.
  • Information | Display Element Information (move the cursor around to get information)
==>

2. Cleanup Borders

I don’t feel that borders around things are warranted. I’m more a clean look kinda guy. Remove all borders, say one to separate the footer from the page.

#content    { border-width: 0px; }
.portlet .pBody
            { border-width: 0px; }
#footer     { border-top: 1px solid #888888; border-bottom-width: 0px; }
==>

Maybe, that’s a little too clean. Add some separators on left side options.

#p-navigation,
#p-search,
#p-tb       { border-top: 1px dotted #888888; }
==>

3. Links

Using the Forge Styles http://forge.mysql.com/css/shared.css we can adjust the links accordingly.

a:link      { color: #10688E; text-decoration: none; }
a:hover     { color: #003366; text-decoration: underline; }
a:visited   { color: #106887; text-decoration: none; }

a.new:link  { color: #AA0000; text-decoration: none; }
a.new:hover { color: #AA0000; text-decoration: underline; }
==>

4. Content

Ok, we have played around a little, now to move into some more serious changes. Looking at the general page look and feel that you see in most page content.


html,body,p,td,a,li
            { font: 12px/19px Verdana, "Lucida Grande", "Lucida Sans Unicode", Tahoma, Arial, sans-serif; }

h1          { font: bold 24px Helvetica, Arial, sans-serif; color: #EB694A; letter-spacing: -1px;
              margin: 1.5em 0 0.25em 0;
              border-bottom: 1px dotted #888888; line-height: 1.1em; padding-bottom: 0.2em; }

h2          { font: bold 18px Helvetica, Arial, sans-serif;  color: #EB694A; letter-spacing: -1px;
              margin: 2em 0 0 0;
              border-bottom: 1px dotted  #888888; line-height: 1.1em; padding-bottom: 0.2em; }

h3          { font-size: 12px; color: #6F90B5; }

h4          { font-size: 12px; color: #6F90B5; }

At this time, I’ve created an Example Page to better demonstration of the look and feel, as the default MediaWiki Main page has limited content.

Some Text Content at the start of the page.

= A First Level Heading =
The first paragraph text.
[[Main Page | Internal Link to existing Page]],
[[Nonexistent Page | Internal Link to nonexistent Page]],
[http://forge.mysql.com  External Link],
http://www.mysql.com  (plain text link)

== A Second Level Heading ==
The first paragraph text.

== A Second Level Heading ==
The first paragraph text.
* List Item Line 1
* List Item Line 2
* List Item Line 3
=== A Third Level Heading ===
The first paragraph text.
* List Item Line 1
** Sub Item 1
** Sub Item 1
** Sub Item 1
*** Sub Item 1
*** Sub Item 2
*** Sub Item 3
* List Item Line 2
* List Item Line 3
=== A Third Level Heading ===
The first paragraph text.
# List Item Line 1
## Sub Item 1
## Sub Item 1
## Sub Item 1
### Sub Item 1
### Sub Item 2
### Sub Item 3
# List Item Line 2
# List Item Line 3
== A Second Level Heading ==
The first paragraph text.

== A Second Level Heading ==
The first paragraph text.
==>

5. Table of Contents

By default, the Table of Contents shows at the top of a page when a given amount of sections or content is present (not sure what the trigger is). The issue is, for larger pages, it means you need to scroll down before you can see any page content. You can disable this with the __NOTOC__ option, but a better solution is to position the Table of Contents so as to not interfere with initial content.

#toc        { float: right; margin: 0 0 1em 1em; border: solid 1px #888888; text-color: #EFEFEF; color: #333333; }
#toc td     { padding: 0.5em; }
#toc .tocindent
            { margin-left: 1em; }
#toc .tocline
            { margin-bottom: 0px; }
#toc p      { margin: 0; }
#toc .editsection
            { margin-top: 0.7em;}
==>

Ok, we are about half way there.

6. Menu Options

In order to get a look and feel like the Forge Home Page., we now have to work on the rest of the navigation options at the top of the page above the content. Let’s start with Second Line of Menu Options (I’ll explain more later why).

#p-cactions { padding-right: 0px; margin-right: 0px; background-color: #006486; width: 100%; top: 77px; }
#p-cactions ul
            { margin: 0; padding: 0; list-style: none;  font-size: 85%; margin-left: 10px; }
#p-cactions li
            { float:left; margin:0; padding:0; text-indent:0; border-width: 0px; }
#p-cactions li a
            { display:block;  color:#F7F7F7;  font-weight: bold;
              background-color: #666666;  border:solid 1px #DDDDDD;
              border-width: 0px;  border-left-width:1px;  text-decoration:none; white-space:nowrap;}
#p-cactions li a:hover
            { background-color: #FFBC2F; color: #666666; }
#p-cactions li.selected a
            { background: #EA7F07;  border:none;}
#p-cactions li.selected  a:hover
            { color: #000000; }
==>

7. Top Menu Options

#p-personal .pBody
            { background: #FFFFFF url(http://forge.mysql.com/img/bggradient.jpg) no-repeat top right; }

#p-personal li a,
#p-personal li a.new
            { color: #FFFFFF; text-decoration: none; font-weight: bold; }
#p-personal li a:hover
            { color: #E97B00; background-color: transparent; text-decoration: underline; }
==>

The down side is it should be the same height at the Forge Page. This required a little more work, and other sections had to be adjusted accordingly, hence why I left this to last. (The size is based on the later mention logo height + margins)

#p-personal { height: 62px; }
#p-personal .pBody
            { height: 62px; }
#p-cactions { top: 62px; }
#content    { margin-top: 84px; }
==>

8. Some Miscellaneous Things

  • Deemphasis the text in the footer a little
  • Remove the annoying arrow that occurs after external links
  • Remove the logo from the login link
  • Hide the My Talk link
#footer     { color: #888888; }
#bodyContent a[href ^="http://"]
            { background:inherit !important; padding-right:inherit !important}
li#pt-userpage
            { background: none; }
li#pt-mytalk
            { display: none; }
==>

The Badging

Let us not forget the final step, the logo badging.

This requires a change to a MediaWiki filesystem file

LocalSettings.php.

$wgLogo             = "http://forge.mysql.com/img/mysqllogo.gif";

And the following Style changes.

#p-logo, #p-logo a, #p-logo a:hover
            { width:100px; height: 52px; }
#p-logo     { margin-left: 10px; margin-top: 5px; margin-bottom: 5px; }

#p-cactions { left: 0px; }
#p-cactions ul
            { margin-left: 180px; }

In order to overcome the top options bleeding to white text on white background, I’ve increased the right side fill of the default bggradient image, replacing the appropriate ULR with the following.

#p-personal .pBody
            { background: #FFFFFF url(/images/bggradient.png) no-repeat top right; }
==>

Conclusion

It’s not quite perfect yet, but this shows how it can be done. Some minor things are left, but I’ve run out of time for the few hours I allocated to this.
The end result of monobook.css for this lesson is:

/* edit this file to customize the monobook skin for the entire site */

/* Background Display */

body        { background-image: none; background-color: #FFFFFF;}
.portlet    { background-color: #FFFFFF; }

/* Borders */

#content    { border-width: 0px; }
.portlet .pBody
            { border-width: 0px; }
#footer     { border-top: 1px solid #888888; border-bottom-width: 0px; }

#p-navigation,
#p-search,
#p-tb       { border-top: 1px dotted #888888; }

/* Links */
a:link      { color: #10688E; text-decoration: none; }
a:hover     { color: #003366; text-decoration: underline; }
a:visited   { color: #106887; text-decoration: none; }

a.new:link  { color: #AA0000; text-decoration: none; }
a.new:hover { color: #AA0000; text-decoration: underline; }

a.external:link {color: #000000; text-decoration: none; }
a.external:hover { color: #AA0000; text-decoration: underline; }


/* Page Look & Feel */
html,body,p,td,a,li
            { font: 12px/19px Verdana, "Lucida Grande", "Lucida Sans Unicode", Tahoma, Arial, sans-serif; }

h1          { font: bold 24px Helvetica, Arial, sans-serif; color: #EB694A; letter-spacing: -1px;
              margin: 1.5em 0 0.25em 0;
              border-bottom: 1px dotted #888888; line-height: 1.1em; padding-bottom: 0.2em; }

h2          { font: bold 18px Helvetica, Arial, sans-serif;  color: #EB694A; letter-spacing: -1px;
              margin: 2em 0 0 0;
              border-bottom: 1px dotted  #888888; line-height: 1.1em; padding-bottom: 0.2em; }

h3          { font-size: 12px; color: #6F90B5; }

h4          { font-size: 12px; color: #6F90B5; }

/* Table of Contents */
#toc        { float: right; margin: 0 0 1em 1em; border: solid 1px #888888; #EFEFEF; color: #333333; }
#toc td     { padding: 0.5em; }
#toc .tocindent
            { margin-left: 1em; }
#toc .tocline
            { margin-bottom: 0px; }
#toc p      { margin: 0; }
#toc .editsection
            { margin-top: 0.7em;}


/* Second Line Top Menu Options */

#p-cactions { padding-right: 0px; margin-right: 0px; background-color: #006486; width: 100%; }

#p-cactions ul
            { margin: 0; padding: 0; list-style: none;  font-size: 85%; margin-left: 10px; }

#p-cactions li
            { float:left; margin:0; padding:0; text-indent:0; border-width: 0px; }

#p-cactions li a
            { display:block;  color:#F7F7F7;  font-weight: bold;
              background-color: #666666;  border:solid 1px #DDDDDD;
              border-width: 0px;  border-left-width:1px;  text-decoration:none; white-space:nowrap;}

#p-cactions li a:hover
            { background-color: #FFBC2F; color: #66666; }

#p-cactions li.selected a
            { background: #EA7F07;  border:none;}

#p-cactions li.selected  a:hover
            { color: #000000; }

/* Top Menu Options */
#p-personal .pBody
            { background: #FFFFFF url(/images/bggradient.png) no-repeat top right; }

#p-personal li a,
#p-personal li a.new
            { color: #FFFFFF; text-decoration: none; font-weight: bold; }
#p-personal li a:hover
            { color: #E97B00; background-color: transparent; text-decoration: underline; }

/* Top Menu Height Adjustments */
#p-personal { height: 62px; }
#p-personal .pBody
            { height: 62px; }
#p-cactions { top: 62px; }
#content    { margin-top: 84px; }

/* Minor Things */
#footer     { color: #888888; }
#bodyContent a[href ^="http://"]
            { background:inherit !important; padding-right:inherit !important}
li#pt-userpage
            { background: none; }
li#pt-mytalk
            { display: none; }

/* Badging */
#p-logo, #p-logo a, #p-logo a:hover
            { width:100px; height: 52px; }
#p-logo     { margin-left: 10px; margin-top: 5px; margin-bottom: 5px; }

#p-cactions { left: 0px; }
#p-cactions ul
            { margin-left: 180px; }

References

User Styles Alternative
Gallery of user styles
Skins
LocalSettings.php Style Information
Navigation Bar
User Rights
Wikipedia Monobook
MediaWiki Skin HowTo

UltimateLAMP

by

This product is no longer available. This was created over 6 years ago and software is too out of date.


As I discussed earlier in A picture can tell a thousand words, I outlined briefly what the intention of UltimateLAMP was for. Let me spill the beans so to speak.

What is UltimateLAMP?

UltimateLAMP is a fully functional environment allowing you to easily try and evaluate a number of LAMP stack software products without requiring any specific setup or configuration of these products. UltimateLAMP runs as a Virtual Machine with VMware Player (FREE). This demonstration package also enables the recording of all user entered information for later reference, indeed you will find a wealth of information already available within a number of the Product Recommendations starting with the supplied Documentation.

My executive punch line with the “right” buzz words is:


You have heard of all the hype in Open Source with lowering Total Cost of
Ownership (TCO) or Total Economic Impact (TEI)? Evaluate Open Source now
in your organistion at no cost or risk with this software package.

What are the uses for UltimateLAMP?







Well, in a nutshell UltimateLAMP allows for instant exposure of Open Source products that use MySQL. You have to remember my goal here is to promote MySQL. Unfortunately, as with any database the features alone will not get the new sale, it’s the applicability to an organisation, and with Open Source the wealth of existing and developing applications that exist can. I don’t necessarily like this approach. Indeed many open source products are poorly designed and can be poor choices in a large scale enterprise solution, but the flip side is, where else can you start.

The benefits of selling a MediaWiki for example due to the success and scalability with Wikipedia is great. So it’s important that the product list includes proven products and currently developing products (rather then stale ones). This is something that the community can definitely provide valuable feedback on to help in this selection.

Other then becoming a CD used as a drink coaster, I feel the potential is here already to provide a copy to people, even install it on a managers computer. You can’t break the software, so why not install it for your non-computer user friends/family. The goal is to move up to executive management however I feel the exposure to the general community first will greatly help.

How it came about

There is some history to this idea. Here are some of the highlights.

  • Late 2005, several planning sessions with Jon and Morgan about a more practical Open Source Contribution user group lead to obviously LAMP stack products for simplicity and exposure. This lead to exposure of LAMP stack products to more of the general person and split from the original intended goal, but was a great idea.
  • Early 2006, futher discussions of how MySQL could get exposure and traction into organisations. For myself professional, how could I promote in industry sectors that I work in.
  • In April 2006, the MySQL Users Conference with discussions of this idea with others and the positive feedback
  • In May 2006, the VMware Appliance Challenge was the possible exposure and deadline needed for me to “Just Do It”. Originally the idea was intended as a Live CD, but in some ways a virtual machine is just as good.

Where to from now!

Well, supply and demand. I don’t know if anybody else has a particular use, or will even download to use it, or market it.

I could see the potential for MySQL User Groups to get behind my idea, and enable members to filter this into known organisations. I could see for targeted opportunities/events, CD’s or information could be distributed. With the support and backing of MySQL AB, I could see the opportunity for even a breakfast CEO/CTI/CIO introduction or other format of meeting the ultimate intended audience.

I could see the potential that an organisation or entity could provide free hosting (30 days) to an organisation that pointed a 3rd level domain to the provider (e.g. wiki.acme.com). It’s like the honeypot, if it takes off after 30 days, the company will either want to pay for hosting, or what to move it. And that’s also fine, organisation provides a MySQL dump, and links to documented installation instructions, or perhaps a sale of services for initial installation/customisation/training can be made.

What can I do?

There is always a list of things that can be done. For now the greatest thing I can ask for is feedback. The good, the bad and the ugly. It’s find to get the comments to say, “That’s great”, or “Good job” or “I can use this”, it’s just as important to get the comments that are proactive in what’s not good. I would value any feedback. Please feel free to Download UltimateLAMP

On my immediate ToDo List or even partially complete is:

  • Document VMPlayer installation instructions for Windows/Linux (partial).
  • Add more product sample content.
  • Add more mediawiki content about the product, like customisation options, references to specific documentation, or other online working examples.
  • Documenting the installation/creation instructions for individual products.
  • Figure out a better way for users to contribute content that get’s rolled back into the Virtual Machines. For the mediawiki, I could see a public online copy, but for other products it could become harder
  • Optimise VM image (removing unnecessary OS stuff), removing product language support (not ideal), but my goal is to provide a 2 CD pack. The first CD has VMplayer in Win/Linux/RPM formats and the default VMware BrowserAppliance (All software from VMWare). The second CD is UltimateLAMP. Combined in a DVD 2x case along with perhaps a small booklet of a few pages, would enable this product to potential move to a commercial state.
  • Investigating other products

Should anybody wishing to help, leave a comment, that way I can see somebody is reading this and so can others.

Related Post:

UltimateLAMP Passwords

A picture can tell a thousand words

by

I’m a keen advocate of MySQL. However, while I use it and promote it within my limited IT circles, I often wonder how MySQL can get better traction and exposure, especially within both the industry sectors and physical locations where I am presently.

This presents a dilemma, it’s almost like the term that has been used in Venture Capital, and in the well named book, Crossing the Chasm. I see and believe that MySQL already has good penetration within certain industry sectors, specifically Internet Based, Startup Based, or Small Based segments. However, I’m sure within other commercial sectors, MySQL has either a token exposure or little to no exposure at all, at least in the circles I mix with.

So how can MySQL the product and MySQL AB the company get both better exposure and penetration? Ultimately it’s great for the community, both in dollars driving product features, product support and from my interest, more jobs. I figure there are many different approaches, and they all target organisations, and the individuals making decisions within the organisations differently. You have the 24×7 support with MySQL Network and certified installations that can satisfy management, you have the speed, flexibility, performance and capability that can appeal to a DBA. But this I doubt is the ultimate hurdle.

How do you do the active sell in 60 seconds to a potential cold contact, or how do you as an MySQL Open Source advocate within an organisation do the active or passive sell to a CEO/CTO/CIO/IT Manager etc.

Well I have an idea, it’s called UltimateLAMP. It’s not the ideal solution, it’s not the great sell, but I figure it’s one approach that I can contribute to and promote.

Stay tuned, more to come very soon.

MySQL GUI Products

by

I’ve started now to actively use more of the MySQL Workbench, MySQL Administrator and MySQL Query Browser and MySQL Migration Toolkit. I am traditionally a very command line person, and it’s important that these skills are never put on the back burner. For all those budding Developers and DBA’s you need these skills, expecially for any MySQL Certification.

To indicate my indent, I’ve even created a new blog category specifically for MySQL GUI Products.
As I’ve mentioned previously, Mike Zinner from MySQL AB really impressed me with where the products were going in development. I’m still rather disappointed about the stability and compatibility when attempting to run under Linux (which is my desktop), but I’ve put that aside and configured a suitable environment to best use and test these products under Windoze! I’ll also be able to trial products under both environments and provide valuable feedback to Workbench Bugs.

To overcome this, as I use Windows reluctantly in a VMware environment, I’ve decided to install all MySQL GUI Products on an older laptop (a Dell Inspiron 5000, PIII 600MHz), which I use for media playing (DivX .avi’s) on my TV. Yes, I hate to resort back to windows, but I had difficulties getting the TV Out working properly under Linux, and I only have so much free time. I would have liked to nut it out, but I find Windows superior in only 2 things, driver compatibility and running Photoshop, so I’ll leave it at that.

So being about to VNC to this machine gives me a spread of processing usage which is acceptable for now, and I’m working from a single desktop. (Time to dream some more about that Dell 24″ UltarSharp Widescreen LCD Flat Panel I’d like).

Well again, with just starting MySQL Workbench, and for the first ever time launching MySQL Administrator I came across Windows functionality I’d not seen before. Within MySQL Administrator under the Windows menu option I was able to switch to MySQL Workbench, a completely different running Windows Program. Now, I’d not seen this before in any multiple running products in Windows. Obviously the power of the GRE enables this, what a nice feature. Well back to now looking at this product some more.

WordPress Blog Upgrade Time

by

Time to Upgrade my WordPress Blog software from Version 1.5.2 to Version 2.0.2 after my latest spam attacks and Combating Blog Spam attempt.

Here is what I did.

cd /home/arabx/www
tar cvfz blog.backup.20060520.tar.gz wordpress-1.5.2
mysqldump -uroot -p arabx_blog > blog.db.20060520.sql
mv blog.*20060520* /u01/backup   # Not in my WWW
scp blog.*20060520* to@someplacesave:/backup
wget http://wordpress.org/latest.tar.gz
mv latest.tar.gz    wordpress-2.0.2.tar.gz # I really hate unversioned files

Disable Plugins. You had to do this manually from the Admin interface (I’m sure it’s just a SQL statement).
Goto http://blog.arabx.com.au/wp-admin/plugins.php

Now some more work.

tar xvfz wordpress-2.0.2.tar.gz  # creates a wordpress directory
mv wordpress wordpress-2.0.2
ln -s wordpress-2.0.2 blog
cp wordpress-1.5.2/wp-config.php wordpress-2.0.2

Back to the Web page again. So much for unattended upgrades?

http://blog.arabx.com.au/wp-admin/upgrade.php

Follow the Steps (step actually). That all worked, now to restore all the additions?

cp -r wordpress-1.5.2/wp-content/themes/arabx wordpress-2.0.2/wp-content/themes/
cp -r wordpress-1.5.2/wp-content/plugins/eventcalendar3* wordpress-2.0.2/wp-content/plugins/

Reactivate my plugins at http://blog.arabx.com.au/wp-admin/plugins.php
Change to my Theme at http://blog.arabx.com.au/wp-admin/themes.php

So my blog page http://blog.arabx.com.au/ looks much like it did previously. Time now to review some of the new features and plugins, expecially steps to combat spam.
References
WordPress Official Upgrade Page
Upgrade to WordPress 2.0
What’s New in WordPress 2.0?

MySQL :: Developer Zone Quick Polls

by

I don’t get to the MySQL Developer Zone main page often enough. In thinking about what pages I view everyday or regularly, it doesn’t rate as high as Planet MySQL, MySQL Forums or even the MySQL Forge.

I was most dissappointed in the results of a recent poll What did you think of the 2006 Users Conference?. The top response was I had no idea there was a Users Conference. That’s not good to see this.

An interesting poll What are you most looking forward to at the MySQL Users Conference (April 24-27)?, the clear winner was Drinking beer with MySQL gurus. What does this say about the attendees. Either they are all alcho’s or the just want to be around guru’s in a less technical way.

I see this page also has a live feed of Planet MySQL. Perhaps we should get some more stuff down the right side of PlanetMySQL like the current Quick Poll itself and a feed of the current developer articles at the Developer Zone.

Installing a new USB External Drive

by

I’ve decide to move from CentOS to Ubuntu as the Linux Distro on my laptop, so it was time to do a full backup to a new USB external drive.

I got a new Maxtor 300G 7200rpm ATA133 (16MB cache) HDD from Umart for AUD$149. I already had an unused USB 2.0 3.5″ external casing.

My first issue was, I couldn’t detect the new drive from my laptop. Switching to another desktop running CentOS, I got the following errors.

$ tail -f /var/log/messages
May  6 12:36:53 marvin kernel: usb 1-1: new full speed USB device using address 4
May  6 12:36:57 marvin kernel: SCSI subsystem initialized
May  6 12:36:57 marvin kernel: Initializing USB Mass Storage driver...
May  6 12:36:57 marvin kernel: scsi0 : SCSI emulation for USB Mass Storage devices
May  6 12:36:57 marvin kernel:   Vendor: Genesys   Model: USB to IDE Disk   Rev: 0033
May  6 12:36:57 marvin kernel:   Type:   Direct-Access                      ANSI SCSI revision: 02
May  6 12:36:57 marvin kernel: usbcore: registered new driver usb-storage
May  6 12:36:57 marvin kernel: USB Mass Storage support registered.
May  6 12:36:58 marvin scsi.agent[3573]: disk at /devices/pci0000:00/0000:00:07.2/usb1/1-1/1-1:1.0/host0/target0:0:0/0:0:0:0
May  6 12:36:59 marvin kernel: usb 1-1: USB disconnect, address 4
May  6 12:36:59 marvin kernel: sda : READ CAPACITY failed.
May  6 12:36:59 marvin kernel: sda : status=0, message=00, host=1, driver=00
May  6 12:36:59 marvin kernel: sda : sense not available.
May  6 12:36:59 marvin kernel: sda: Write Protect is off
May  6 12:36:59 marvin kernel: sda: assuming drive cache: write through
May  6 12:37:00 marvin kernel: sda : READ CAPACITY failed.
May  6 12:37:00 marvin kernel: sda : status=0, message=00, host=1, driver=00
May  6 12:37:00 marvin kernel: sda : sense not available.
May  6 12:37:00 marvin kernel: sda: Write Protect is off
May  6 12:37:00 marvin kernel: sda: assuming drive cache: write through
May  6 12:37:00 marvin kernel: sda : READ CAPACITY failed.
May  6 12:37:00 marvin kernel: sda : status=0, message=00, host=1, driver=00
May  6 12:37:00 marvin kernel: sda : sense not available.
May  6 12:37:00 marvin kernel: sda: Write Protect is off
May  6 12:37:00 marvin kernel: sda: assuming drive cache: write through
May  6 12:37:00 marvin kernel:  sda:<3>scsi0 (0:0): rejecting I/O to device being removed
May  6 12:37:00 marvin kernel: Buffer I/O error on device sda, logical block 0
May  6 12:37:00 marvin kernel: scsi0 (0:0): rejecting I/O to device being removed
May  6 12:37:00 marvin kernel: Buffer I/O error on device sda, logical block 0
May  6 12:37:01 marvin kernel: scsi0 (0:0): rejecting I/O to device being removed
May  6 12:37:01 marvin kernel: Buffer I/O error on device sda, logical block 262143
May  6 12:37:01 marvin kernel: scsi0 (0:0): rejecting I/O to device being removed
May  6 12:37:01 marvin kernel: Buffer I/O error on device sda, logical block 262143
May  6 12:37:01 marvin kernel: scsi0 (0:0): rejecting I/O to device being removed
May  6 12:37:01 marvin kernel: Buffer I/O error on device sda, logical block 0
May  6 12:37:01 marvin kernel:  unable to read partition table
May  6 12:37:01 marvin kernel:  sda:<3>scsi0 (0:0): rejecting I/O to device being removed
May  6 12:37:01 marvin kernel: Buffer I/O error on device sda, logical block 0
May  6 12:37:01 marvin kernel: scsi0 (0:0): rejecting I/O to device being removed
May  6 12:37:01 marvin kernel: Buffer I/O error on device sda, logical block 262143
May  6 12:37:01 marvin kernel: scsi0 (0:0): rejecting I/O to device being removed
May  6 12:37:01 marvin kernel: Buffer I/O error on device sda, logical block 0
May  6 12:37:01 marvin kernel:  unable to read partition table
May  6 12:37:01 marvin kernel: Attached scsi removable disk sda at scsi0, channel 0, id 0, lun 0

So, thinking it may be the pin switches, I unscrewed everything to change the pin selector from Cable Select to Master. Trying again, with the bare disk and just the connectors, I confirmed this was the issue. This is what one should expect.

May  6 13:00:28 marvin kernel: usb 1-1: new full speed USB device using address 6
May  6 13:00:29 marvin kernel: scsi2 : SCSI emulation for USB Mass Storage devices
May  6 13:00:29 marvin kernel:   Vendor: Genesys   Model: USB to IDE Disk   Rev: 0033
May  6 13:00:29 marvin kernel:   Type:   Direct-Access                      ANSI SCSI revision: 02
May  6 13:00:29 marvin kernel: SCSI device sda: 586114704 512-byte hdwr sectors (300091 MB)
May  6 13:00:29 marvin kernel: sda: test WP failed, assume Write Enabled
May  6 13:00:29 marvin kernel: sda: assuming drive cache: write through
May  6 13:00:29 marvin kernel:  sda: unknown partition table
May  6 13:00:29 marvin kernel: Attached scsi removable disk sda at scsi2, channel 0, id 0, lun 0
May  6 13:00:30 marvin scsi.agent[4172]: disk at /devices/pci0000:00/0000:00:07.2/usb1/1-1/1-1:1.0/host2/target2:0:0/2:0:0:0

So working with /dev/sda which was expected, sd being for the USB connections, and a for the first.

Double check nothing on drive.

$ fdisk -l /dev/sda

Disk /dev/sda: 300.0 GB, 300090728448 bytes
255 heads, 63 sectors/track, 36483 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sda doesn't contain a valid partition table

Partitioning

Partition the drive into 3 x 100GB partitions.

$ fdisk /dev/sda
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 36483.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

Command (m for help): p

Disk /dev/sda: 300.0 GB, 300090728448 bytes
255 heads, 63 sectors/track, 36483 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-36483, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-36483, default 36483): +100000M

Command (m for help): p

Disk /dev/sda: 300.0 GB, 300090728448 bytes
255 heads, 63 sectors/track, 36483 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1       12159    97667136   83  Linux


Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (12160-36483, default 12160):
Using default value 12160
Last cylinder or +size or +sizeM or +sizeK (12160-36483, default 36483): +100000M

Command (m for help): p

Disk /dev/sda: 300.0 GB, 300090728448 bytes
255 heads, 63 sectors/track, 36483 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1       12159    97667136   83  Linux
/dev/sda2           12160       24318    97667167+  83  Linux

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (24319-36483, default 24319):
Using default value 24319
Last cylinder or +size or +sizeM or +sizeK (24319-36483, default 36483):
Using default value 36483

Command (m for help): p

Disk /dev/sda: 300.0 GB, 300090728448 bytes
255 heads, 63 sectors/track, 36483 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1       12159    97667136   83  Linux
/dev/sda2           12160       24318    97667167+  83  Linux
/dev/sda3           24319       36483    97715362+  83  Linux

Command (m for help): v
15371 unallocated sectors

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Confirm.

$ fdisk -l /dev/sda

Disk /dev/sda: 300.0 GB, 300090728448 bytes
255 heads, 63 sectors/track, 36483 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1       12159    97667136   83  Linux
/dev/sda2           12160       24318    97667167+  83  Linux
/dev/sda3           24319       36483    97715362+  83  Linux

Format

Formating the 3 new Partitions. (Note: this takes a while)

$ mkfs.ext3 -v /dev/sda1
$ mkfs.ext3 -v /dev/sda2
$ mkfs.ext3 -v /dev/sda3

Confirm

Create some mount points and test mount.

$ mkdir /u11
$ mkdir /u12
$ mkdir /u13
$ mount -t ext3 /dev/sda1 /u11
$ mount -t ext3 /dev/sda2 /u12
$ mount -t ext3 /dev/sda3 /u13

Add hard disk to fstab

A Post MySQL Conference review. The 4 F's

by

Finally back home after some R&R at Yosemite before leaving the US. In conclusion, to sum up my experience of the 4th Annual MySQL Users Conference “Excellent”.
Here’s my take. Friends, Functionality & New Features, the Future.

Friends

I’ve used MySQL now for over 6 years, and full time for a number of years, yet I’ve only become active in the MySQL community, particularly Planet MySQL in the past 6 months. Over that time, I’ve read a lot from members, and heard from many people. It was great at the conference to meet many of these people for the first time. The list includes: Community Members – Frank Mash,Mike Kruckenberg, Markus Popp, Roland Bouman, Giuseppe Maxia and Paul McCullagh. MySQL Employees – Mike Hillyer, Colin Charles, Jay Pipes, Mike Zinner, and New Contacts – Kristian Köhntopp, Jeremy Cole, Sheeri Kritzer, Taneli Otala, Laura Thompson just to start the list.

Functionality

Not only was there plenty of discussion on Server Functionality, there was plenty of MySQL Client functionality including the MySQL Workbench, MySQL Migration Toolkit and the other MySQL GUI products.
There were a number of discussions on uses and implementations of MySQL in large web deployments. It would be great to see some more white papers here.

New Features

A few months ago I wrote an article A call to arms!. In some part, I was just giving my opinion and hoping to gee up some support and feedback from the community. Well, the MySQL 5 Pluggable Storage Architecture got a great boost with announcements of transactional storage engines Falcon by Jim Starkey, Solid and PBXT. Add details of InnoDB New Features, MyISAM additions, and indications of other wonderful if not entirely practical options. I’m sure there is much more in stall to come this year that wasn’t discussed.

A number of talks featured Cluster including Monday’s tutorial, and with 5.1 and beyond I can see next year there will be more discussion on successful Cluster implementations. There was a lot of talks about Scaling out. I’d like to see more practical examples, perhaps a detailed tutorial.

The Future

What does the future hold for MySQL? The MySQL Server and Storage Engine Roadmap provided an insight of the upcoming planned features and releases over the next 2 years. Of course, the marketplace can change quickly, and MySQL is in a great position to react to the needs of the community quickly.

And before your know it, the 5th Annual MySQL Conference will be in play.

Conference Feedback

One thing I had a chance to discuss with Jay Pipes after the conference, I wasn’t the first to mention, and plans are already in motion, was a number of talks just needed more time. Moving the schedule to 55 minute talks gives that extra time, even if it is open question time from the floor, but it also makes knowing when sessions are on much easier, if they always start at the top of the hour.

In Conclusion

Frank (a.k.a Farhan Mashraqi) asked me what session I liked the most? Hard to say. Agile Database Techniques: Data Doesn’t Have to be a Four-Letter Word Anymore rated very highly, as the content was close to heart and my expertise. HackFest B: Creating New SHOW Commands by Brian Aker, showed just how easy it was to get into the MySQL source. Of course the internals are much more complex then this, but it was a good introduction. My favourite keynote was The Ubuntu Project: Improving Collaboration in the Free Software World. There were a number of talks I was disappointed in, as well as a number I didn’t get to due to 8 sessions in parallel.

I would have to say, that what impressed me most was no one single talk, but the functionality of the GRT Shell that Mike Zinner and his team have built into the GUI product line. I was very impressed, and I could see this providing extensive functionality and not just MySQL specific centric tools. This will be area I’ll be focussing on my contributions in the near future.

MySQL Stored Procedures Performance

by

Another one of the sessions at the MySQL Users Conference I attended was Tuning MySQL5 SQL and Stored Procedures by Guy Harrison from Quest Software. A global company with 6000+ customers.

Guy has written a number of Oracle Performance Books in the past. His work now is on the “Spotlight” product family – Database diagnositic tools converting data to graphical representations. For these products, MySQL 5 and InnoDB only is necessary, simply due to accessing the right internal information for presentation. There are Freeware MySQL product downloads.

In this presentation he stated, nothing he was talking about specifically was relatively new. He did make quite a funny comment, “He is now seeking refugee status in the MySQL Community”.

Guy is author of O’Reilly “MySQL Stored Procedure Programming” Book. I managed to get for free at the conference from the MySQL Quiz night, in addition to a shirt and cap for stumping a Guru.

His talk were on tools and techniques for tuning MySQL.

  • Explain Command – reveals what the optimizer intends to do
  • Explain Extended
mysql> explain extended select ...;
mysql> show warnings G
Shows what the optimizer actually did. In this example, An IN was converted to EXISTS

There were 4 ways to provide optimizer hints.

  1. STRAIGHT_JOIN
  2. USE INDEX(…)
  3. FORCE INDEX(…)
  4. IGNORE INDEX(…)

In addition to the Show Query Log, there are Innodb specific commands, two in particular.

show status like 'innodb%'
* innodb_buffer_pool_read_requests
* innodb_data_read

Indexing and the optimizer

  • In MySQL Index is the best tool to improve performance, however sometimes it’s better to access the entire table.
  • Indexes generally effective when between 5% and 20% of rows are accessed.
  • Subqueries need to be satisified by an index or performance will be quite inefficent.
  • Overloading indexes with additional columns when key queries only use a few columns can enable improved performance.

Not all indexes are created equal. In the following examples, each advancement improved performance.

  • No indexes ()
  • Single Index (customer)
  • multiple indexes (customer, product)
  • concatenated indexes (customer + product)
  • covering index (including required columns, customer+product+qty)

Examples of SQL that can’t benefit from Indexes.

  • Derived tables – SELECT table in a from clause, creates a temporary table and will never get an index.
  • Views with UNIONS/GROUP BY

A comment from the audience was that derived tables can be of a benefit to a correlated sub-query in specific examples.

Stored Procedures provided a mixed blessing for performance.

  • Can improved perfomrance when high network overhead.
  • Some improvement on parsing.
  • Breaking up complex queries may provide benefits.
  • SQL is highly optimized for SET operations.
  • SP is not optimized for number crunching. Computionally not a fast language.

A written routine to calculate prime numbers provided the following performance (from most expensive to least) MySQL SP, Oracle SP,PHP,Perl,Java ,VB.NET ,C (gcc). This showed an example that was excessively inefficient. On the other hand, if the program is network dependent (e.g. access a million rows, perform some statisical aggretation). Comparatively the same between Java and SP locally, but much better in a remote host mode.

Performance of SQL in a SP will dominate overall performance. Where SQL is tuned, goto tried and proven traditional optimisation techniques.

  • Optimize iterations
  • Optimize Logic/Testing
  • Avoid recursion

Loop Management

  • Only perform necessary code within iterations
  • LEAVE or CONTINUE when possible in loops
  • Test the most likely IF/THEN statements first
  • extract if comparisions duplicated to produce nested if’s (within reason)

Some guidelines for Triggers.

  • Triggers will have a non-trival overhead for even the simplest trigger.
  • Due to FOR EACH ROW only, don’t have expensive SQL in any trigger.
  • Very carefully tune SQL in triggers.
  • Empty trigger produced 12% overhead.

For more information check out www.quest.com/mysql

Mark Shuttleworth

by

Thursday’s Keynote speaker at the MySQL Conference was Mark Shuttleworth talking on The Ubuntu Project: Improving Collaboration in the Free Software World.

Ubuntu from Canonical Ltd is fastly becoming one of the most popular Linux Distributions, based on Debian. Here were some notes from his presentation. Mark had some really powerful one or two line slides that typified both his presentation intention and the goal of Ubuntu.

Delivering on the Promise of Free Software

  • MySQL levelled the playing field, individuals could use MySQL to build applications that could now compete in the industry.
  • Apache, Python and Linux made it all possible.
  • This has provided the Opportunity and passion to build software that can serve a purpose.
  • Ubuntu is the continuation of levelling the playing field. Making always available at no cost.
  • Whatever your vision, you have the opportunity without financial limitation.
  • To be a pioneer in new fields, open source freedom enables the opportunity to anybody.

The New Deal of Free software

* Different Economics

  • Ubuntu is world class and free of charge.
  • Important to recognise the governence.
  • We don’t build 99%, it’s built by communities. We want to maintain good healthy and open relationships with the communities of software developers and contributors.

* Different culture

  • Transparency. You can participate, you get to see how the software is built. Difficult for co-ordination of people worldwide, the advantage is you get to be people in the public space everywhere.
  • To be able to try, experience to express their own inspiration.

Our vision is to build a complete community-driven distribution that can grow and substain itself through support and services.

Self substaining at a professional level. Community Driven.

  • The Ubuntu Proposition
  • Genuine Freedom and Flexibility
  • Zero Licensing Fees
  • Certification: Hardware, Softare, Skills
  • Superb Commercial Technical Support
  • Standards Compliance
  • Internatiolzation and customization
  • Collobaration to the Core

We do the best work when we focus on extreme challenges.

Mark made a great comment about the Chinese Version, What was in English on any pages, for Mark was the saving grace, but for the Chinese user, that was the wall, so a committed emphasis on Internationalisation was important.

Ubuntu as you know it today is widely rated #1 desktop Linux with millons of users globally, with complete office and home desktop distributions. We started on the desktop, we are continuing to expand into different variants including the enterprise. The next release in June 2006 – Dappy – 6.06 will includ LTS (Long Term Support) 3 years for Desktop, 5 years for server.

We maintain relationships with the community ecosystem.

  • Looking the the User Relationships.
  • Prime focus is simplicity. Also Comprehensive and Community, Current.
  • For the business relationships, wanting to offering Tiered Partnership Program and Extensive Global Solutions Provider Network.

Free Software Relationships

  • Close to upstream (e.g. MySQL, Apache, Firefox etc)
  • Close also to other distributions (both debian based and not). We want the best of their work to also be part of what’s available to all.
  • Need to ramp up the collaboration of projects especially between projects. Check out launchpad.net
    Site – Much easy to colloborate across projects
  • With MySQL and Ubuntu you have freedom to deploy without asking permission, and support when and where you need it
  • A key is make packaging ubuntu and mysql the best getting contributions from the community. Expresses most powerfully what MySQL can do.

Beating history into submission

  • We judge ourselves by history. We should look at our strengths. we should ask “What futures are enabled.”

Free software enables a different economic paradigm.
Embrace it.

A question from the audience was where does the name Ubuntu come from. The Answer was.
Ubuntu is common to many African languages. It means “human-esse”. The Ubuntu root is common to a lot of words. Some means included “The way we look after other people defines who we are. ”

Kubuntu – Has a meaning “For Humanity”.



My MySQL Conference Presentation

by


My talk at the MySQL Users Conference went well. MySQL For Oracle Developers was part of two talks, the second was by Mark Leith on MySQL For Oracle DBA’s.

I had a number of positive comments from attendees, including Ken Jacobs of Oracle who also contributed some valuable information in comparision of UPDATE/DELETE ORDER BY/LIMIT statements I was unaware of.

You can download my paper here.

My slides, like many talks just scratched the surface in the alloted. I’ve been working on additional reference material, a work in progress is available at MySQL For Oracle Developers. My continuatation on this will be dependent on feedback from the community of it’s intended value.

Hacking MySQL Source improvements

by

Further to my earlier post Hacking MySQL Source (in a good way) in which I was having a compilation problem when re-compiling MySQL source (i.e. tt worked the first build, but failed on subsequent re-compiles, even with no changes).
I’ve been able to solve my re-compiling issue, with special thanks to Jan Kenschke of MySQL AB, who was near during the MySQL Quiz Show.

As suspected, there is no need to re-run the ./BUILD/compile-pentium-debug command as this cleans, a simple make command is sufficient (I needed to confirm no other command args where needed). By doing this, it also better highlighted the actual error, where previously is wasn’t. I didn’t keep a copy of the error, but in effect I needed to do the following in order to successfully recompile.

su - mysqldev
cd mysql-5.1.10-beta-nightly-20060413
touch mysql-test/std_data/client-key.pem
touch mysql-test/std_data/client-cert.pem
touch mysql-test/std_data/cacert.pem
touch mysql-test/std_data/server-cert.pem
touch mysql-test/std_data/server-key.pem
make
# Success

I can’t answer why I had to do this manually (create these cert permissions), and why some level of make doesn’t (or does but I don’t know the command), but it works, and right now that’s important. Hopefully a MySQL guru can enlightment me.

From here, I was able to test as I’d written previously. I just need to confirm my changes work before reporting back, as well as providing some guidelines for debugging with gdb.