Downgrading a MySQL schema from 5 to 4

Why oh why would you want to do this. Well it my case, I’ve committed to developing a web application using MySQL 5 features, knowing that I had to upgrade my production server from 4.0

Well as part of doing this, I hit a stumbling block. My current production web server runs RedHat 7.3, and even with all the latest rpm updates, it does not have glib 2.3 which is required for MySQL 5. I’m no guru, but trying to upgrade from Redhat 7.3 to 9, to at least get these rpm is not an easy process. I’m not confident to try to compile glib 2.3 and all it’s dependancies on a production server, nor is it possible to recompile MySQL down (I suspect not). All just too many variables. It appears the time has come to scrap it and work with a more current RedHat Enterprise Linux version. Down side, the 25 web sites are not going to be too happy.

Anyway, as an interim to at least move forward as much as possible I dowgraded the provided schema.sql to at least run under 4.0. We all know about new features in 5.0, so this is just really the same list, but it was worthwhile documenting.

  • Remove views
  • Remove triggers
  • Remove DELIMITER syntax
  • Change ENGINE= to TYPE=
  • Change DROP SCHEMA to DROP DATABASE
  • Change CREATE SCHEMA to CREATE DATABASE
  • Remove DEFAULT CHARSET=utf8 from table definition
  • Remove DEFAULT TIMESTAMP from TIMESTAMP columns (NOTE: Need to verify the column is the first TIMESTAMP column)
  • Change BOOLEAN to TINYINT(1)

Update:
Friend and collegue Morgan points out that if I was dumping a schema with 5.1 you could use the syntax mysqldump --compatible=name. Unfortunately in my case the schema was coming from a third party, however now I know this syntax, I can always ask for it.

Update 2:
I know in earlier versions that you could get a MySQL product installation, and also get one compiled with glib23. I just figured that over time the older glib22 were dropped, and were not available with 5.1. It seems domestic blindness has now hit me in computer software. I did review the MySQL 5.1 downloads page, and did not see lower in the page mixed in the 30 to 40 downloads and option to download a glibc22 version. Thanks Arjen for pointing that out.


Domestic Blindness
“The inability to find a common object that is right in front of your face.”

The challenges of compiling non working Open Source (Part 2)?

Did I push to much in my last post? I don’t think so, but I guess it’s a fragile balance sometimes in Open Source between those keen end users, and the developers that do give so much towards their own creations (I understand, I’m in that category myself).

I was very proud of my work yesterday, it took a whole day of my time (I do have better things to do, like finish my own Open Source project HTMLtags, while will allow me to build my sample application, which I can then use for my MySQL Users Conference presentation). I learnt to dig around the net a lot, go on the wild goose chase several times, understand some more under the hoods of compiling, libraries and dependencies in the GTK world I would have otherwise not really cared about. But as I said, I got to a brick wall by the end, and it was dishearting.

It seemed my Bug Report #16604 on MySQL Workbench compiling listing a clear number of bugs was not well received by the development team. Maybe I should have slept on it, but about 1am in the morning I made another plea for assistance.

Well I woke this morning, and as I mentioned in my opening statement was I too passionate about this pursuit. Perhaps not. Overnight, a positive response to my #16604 provided feedback that my hard work was indeed incoporated, and the next clue of compilation (which could just be in a simple INSTALL file hopefully in the future) was presented.

So eagerly ignoring breakfast, and the increasing pile of dirty dishes, I jump in with this new intel. It ws just one line.

you need to configure gui-common with: ./configure –enable-grt –enable-canvas to enable the components needed by Workbench.


$ cd /src/mysql-workbench-1.0.0/mysql-gui-common
$ ./configure --enable-grt --enable-canvas
$ make
make[3]: Entering directory `/src/mysql-workbench-1.0.0/mysql-gui-common/library_grt/source'
if gcc -DHAVE_CONFIG_H -I. -I. -I../.. -I../../library_grt/include -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/libxml2 -I../../library_util/include -I../../library_util/shared_include -I../../library_grt/newt -I/opt/mysql/include -I/usr/include/pcre -I/usr/include/python2.3 -DENABLE_JAVA_MODULES -DENABLE_PYTHON_MODULES -DLUA_TEXT_DIALOGS -g -O2 -MT lua_dialogs.o -MD -MP -MF ".deps/lua_dialogs.Tpo" -c -o lua_dialogs.o lua_dialogs.c;
then mv -f ".deps/lua_dialogs.Tpo" ".deps/lua_dialogs.Po"; else rm -f ".deps/lua_dialogs.Tpo"; exit 1; fi
lua_dialogs.c:20:17: lua.h: No such file or directory
lua_dialogs.c:21:21: lauxlib.h: No such file or directory
....
make[3]: *** [lua_dialogs.o] Error 1
make[3]: Leaving directory `/src/mysql-workbench-1.0.0/mysql-gui-common/library_grt/source'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/src/mysql-workbench-1.0.0/mysql-gui-common/library_grt'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/src/mysql-workbench-1.0.0/mysql-gui-common'

More googling, hopefully http://www.lua.org is what this is???


$ cd /src
$ wget http://www.lua.org/ftp/lua-5.0.2.tar.gz
$ tar xvfz lua-5.0.2.tar.gz
$cd lua-5.0.2
$ ./configure
$ vi config
# (change Line 151: INSTALL_ROOT from /usr/local to /usr)
$ ./configure
$ make
$ make install


$ cd /src/mysql-workbench-1.0.0/mysql-gui-common
$ make
make[3]: Entering directory `/src/mysql-workbench-1.0.0/mysql-gui-common/source/linux'
if g++ -DHAVE_CONFIG_H -I. -I. -I../.. -DXTHREADS -D_REENTRANT -DXUSE_MTSAFE_API -I/usr/include/libglade-2.0 -I/usr/include/gtk-2.0 -I/usr/include/libxml2 -I/usr/lib/gtk-2.0/include -I/usr/X11R6/include -I/usr/include/atk-1.0 -I/usr/include/pango-1.0 -I/usr/include/freetype2 -I/usr/include/freetype2/config -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/gtkmm-2.4 -I/usr/lib/gtkmm-2.4/include -I/usr/include/glibmm-2.4 -I/usr/lib/glibmm-2.4/include -I/usr/include/gdkmm-2.4 -I/usr/lib/gdkmm-2.4/include -I/usr/include/pangomm-1.4 -I/usr/include/atkmm-1.6 -I/usr/include/sigc++-2.0 -I/usr/lib/sigc++-2.0/include -I/opt/mysql/include -I/usr/include/pcre -DENABLE_JAVA_MODULES -DENABLE_PYTHON_MODULES -I/usr/include/freetype2 -I../../library/include -I../../library_util/include -I../../library_util/shared_include -I../../library_grt/include -I../../library_grt_modules/include -I../../library_grt_workbench/include -I../../library_gc/include -I../../library_gc/ftgl/include -I.. -DDATADIRNAME=""share"" -DCOMMONDIRNAME="""" -g -O2 -MT MGRT.o -MD -MP -MF ".deps/MGRT.Tpo" -c -o MGRT.o `test -f 'MySQLGRT/MGRT.cc' || echo './'`MySQLGRT/MGRT.cc;
then mv -f ".deps/MGRT.Tpo" ".deps/MGRT.Po"; else rm -f ".deps/MGRT.Tpo"; exit 1; fi
MySQLGRT/MGRT.cc: In member function `void MGRT::init_thread(const std::string&)':
MySQLGRT/MGRT.cc:115: error: `myx_grt_shell_print_welcome' undeclared (first use this function)
MySQLGRT/MGRT.cc:115: error: (Each undeclared identifier is reported only once for each function it appears in.)
MySQLGRT/MGRT.cc:170: error: `myx_lua_init_loader' undeclared (first use this function)
MySQLGRT/MGRT.cc:194: error: `myx_grt_init_lua_shell' undeclared (first use this function)
MySQLGRT/MGRT.cc: In member function `void MGRT::perform_shell_command(const Glib::ustring&)':
MySQLGRT/MGRT.cc:275: error: `myx_grt_lua_shell_execute' undeclared (first use this function)
MySQLGRT/MGRT.cc: In member function `Glib::ustring MGRT::shell_prompt()':
MySQLGRT/MGRT.cc:283: error: `myx_grt_lua_shell_get_prompt' undeclared (first use this function)
make[3]: *** [MGRT.o] Error 1
make[3]: Leaving directory `/src/mysql-workbench-1.0.0/mysql-gui-common/source/linux'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/src/mysql-workbench-1.0.0/mysql-gui-common/source'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/src/mysql-workbench-1.0.0/mysql-gui-common'
make: *** [all] Error 2

This seems defined in library_grt/source/myx_grt_lua_shell.c which has compiled to object code. So it’s probably clearly related with the lua dependency in
some way, but I’m not a C++ developer so apart from googling and greping it’s a good 3 levels over my head.

A very strong dead end again, clearly very C++ and MySQL specific. Back to my new buddy within MySQL AB, for the next clues.

The challenges of compiling non working Open Source?

One of the great benefits of Open Source, it’s Free, and you can get great support, sometimes even from the developers directly (rather then 5 levels of paid customer support for a commercial product). One of the greatest banes of Open Source, if you have a problem, and nobody has experienced and documented in a forum etc the problem you have with the same OS, libraries etc, you could be totally up the creek without a paddle, boat and for that matter water. (luckily you still have oxgyen)

Well, I’m having this problem with MySQL Workbench. A product promising so much, but if you can’t get the binary working on Linux to even start, where do you go.

You will see via the Forums, I’m not the only person. This is the current Bugs List.

Wanting to make a difference, even just for myself, and those others that also seem lost, I set out to pursue this to the bitter end. Long story short, some 6-7 hrs later I’m so close, and the response in a related Bug at the point I’ve now finally reached is:

at this moment we do not process bugs of mysql workbench, because it’s still in the intensive development, and we provided snapshot in order to give people first implression of WB.
I am changing status to ‘Analyzing’ and return to this bugreport when WB will be issued officially.

Well, my only statement here at 11:30pm at night, is I’m annoyed and frustated. The only reason why I’m compiling source is the snapshot doesn’t work, there is no information of when it will be issued officially, infact as I’ve mentioned previously, there are 3 different versions for 3 different OS’s at present. (More Info in Forums). It’s complicated as it seems to be all one way communication, people reporting problems, but no information feedback in return. Perhaps I should have worked on my own Open Source project?

My simple question in response to this comment is. “Please provide the source that built the binary, so at least we can work with the same apples. The released source and snapshot binary are not consistent.”

My only pursuit now is to publish my findings. I’m made signficant progress that will help others, but only so far. I’ve also uncovered 3 configuration errors, and library inconsistency as per minimum requirements (which isn’t documented, it’s hit and miss, trial and error), and a fatal compilation error stopping you in your tracks. My current logged bug report of this Bug #16604

The road already covered

My environment is CentOS 4.2 (a.k.a RedHat Enterprise Linux RHEL 4.2) recompiled and free.

The first pot hole.

Downloaded Linux Binary 1.0.0-alpha from ftp://ftp.mysql.com/pub/mysql/download/mysql-workbench-1.0.0-alpha-linux-i386.tar.gz (announced on forums MySQL Workbench 1.0.0alpha for Linux (November 24, 2005)

Errors on startup.
./mysql-workbench-bin: /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.5' not found (required by ./mysql-workbench-bin)
./mysql-workbench-bin: /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.4' not found (required by ./mysql-workbench-bin)

Refer to my Initial findings published at forums (December 16, 2005) over 1 month ago with details of my environment and installed libraries.

Looking at the hole, now it’s a trench.

Diving right in.

$ su -
$ cd /src
$ wget ftp://ftp.mysql.com/pub/mysql/download/mysql-workbench-1.0.0-alpha-linux.tar.gz
$ tar xvfz mysql-workbench-1.0.0-alpha-linux.tar.gz
$ cd /src/mysql-workbench-1.0.0/mysql-gui-common
$ ./configure

Errors
No package ‘gtkmm-2.4′ found
configure: error: Library requirements (libglade-2.0 gtkmm-2.4) not met; consider adjusting the PKG_CONFIG_PATH environment variable if your libraries are in a nonstandard prefix so pkg-config can find them.
Found http://www.gtkmm.org – gtkmm is the official C++ interface for the popular GUI library GTK+. First problem, current version is 2.8, previous version (as on home page links is 2.2).

More digging, clicking the home page documentation link, and some docs recommend binary installs (why am I not surprised). gtkmm 2.4 documentation. What the! The home page has 2.8 and 2.2, and the docs are 2.4. I’m a little confused, but I trundle on.

Found http://www.gtkmm.orgNo RHEL binaries, tried for Fedora Core 4. Attempt to Add FC4 extras to yum settings and do yum install gtkmm24-docs failed. (this was the docs recommendation)

So track it down on a mirror. Now that is now gtkmm24 Version 2.6???? I’m sure there’s a reason but I’m a GTK lay person and it’s confusing me.


$ wget http://public.planetmirror.com/pub/fedora/linux/extras/4/i386/gtkmm24-2.6.2-2.i386.rpm
$ wget http://public.planetmirror.com/pub/fedora/linux/extras/4/i386/gtkmm24-devel-2.6.2-2.i386.rpm
$ wget http://public.planetmirror.com/pub/fedora/linux/extras/4/i386/gtkmm24-docs-2.6.2-2.i386.rpm

$ rpm -ivh gtkmm24-2.6.2-2.i386.rpm
warning: gtkmm24-2.6.2-2.i386.rpm: V3 DSA signature: NOKEY, key ID 1ac70ce6
error: Failed dependencies:
libglibmm-2.4.so.1 is needed by gtkmm24-2.6.2-2.i386
libsigc-2.0.so.0 is needed by gtkmm24-2.6.2-2.i386
libstdc++.so.6(GLIBCXX_3.4.4) is needed by gtkmm24-2.6.2-2.i386

I figured it wasn’t going to be that easy. However, found an interesting note, (GLIBCXX_3.4.4), hmmm, see that error before. Is that sunlight I see looking up out of the trench.

It doesn’t ran it pours.

Won’t focus here.

libsigc Not found in CentOS RPMS Found at http://libsigc.sourceforge.net

$ cd /src
$ wget http://ftp.gnome.org/pub/GNOME/sources/libsigc++/2.0/libsigc++-2.0.17.tar.gz
$ tar xvfz libsigc++-2.0.17.tar.gz
$ cd /src/libsigc++-2.0.17
$ ./configure
$ make
$ make install
$ rpm -ivh gtkmm24-2.6.2-2.i386.rpm

Still fails with same dependancies including libsigc-2.0.so.0 which is what I just compiled.

Using a trench digger now, shovelling was too much work.

I won’t bore you with the iterative details of my approach, needless to say, basically I had to work backwards from this resultant coding, a number of times. But this worked. (Compiling the dependancies that is)


$ cd /src
$ wget http://ftp.gnome.org/pub/GNOME/sources/libsigc++/2.0/libsigc++-2.0.17.tar.gz
$ tar xvfz libsigc++-2.0.17.tar.gz
$ cd /src/libsigc++-2.0.17
$ ./configure --prefix=/usr
$ make
$ make install

One Down, note the –prefix=/usr is significant.. There must be some funky way to use LD_LIBRARY_PATH, LIBDIR, just need somebody to explain why simple evironment variable doesn’t work downstream.


# configure: error: Package requirements (sigc++-2.0 >= 2.0.0 glib-2.0 >= 2.4.0 gobject-2.0 >= 2.4.0 gmodule-2.0 >= 2.4.0) were not met.
$ cd /src
$ wget http://ftp.gnome.org/pub/GNOME/sources/glibmm/2.4/glibmm-2.4.8.tar.gz
$ tar xvfz glibmm-2.4.8.tar.gz
$ cd /src/glibmm-2.4.8
$ ./configure --prefix=/usr
$ make
$ make install

Two down.


# Requires checking for glibmm-2.4 >= 2.4.0 atk >= 1.6.0... Package glibmm-2.4 was not found in the pkg-config search path.
$ cd /src
$ wget http://ftp.gnome.org/pub/GNOME/sources/gtkmm/2.4/gtkmm-2.4.11.tar.gz
$ tar xvfz gtkmm-2.4.11.tar.gz
$ cd /src/gtkmm-2.4.11
$ ./configure --prefix=/usr
$ make
$ make install

Three Down.


$ cd /src/mysql-workbench-1.0.0/mysql-gui-common
$ ./configure
checking for pcre-config... no
configure: error: Could not find pcre-config script. Make sure the pcre libraries are installed

What is pcre-config? Not much luck finding that, try pcre, come across http://www.pcre.org/ – Perl Compatible Regular Expressions. Well could be, no info about pcre-config, nothing on the Perl regular expressions man page. Well, no pain no gain, it’s in a yum list so.


$ yum install pcre

Already installed, well that doesn’t help. Some more digging around, more time digging a bigger hole, there’s a pcre-devel, let’s try that.


$ yum install pcre-devel

Voila!, I have a pcre-config script


$ cd /src/mysql-workbench-1.0.0/mysql-gui-common
$ ./configure

./configure: line 3488: -f: command not found

Are a problem, lucky already documented in Forums (at least somebody has been this far).

Replace Line 3488
if ! -f po ; then
with
if test ! -f po ; then

Try Again.


$ cd /src/mysql-workbench-1.0.0/mysql-gui-common
$ ./configure

./configure: line 7368: syntax error near unexpected token `else'

Haven’t see this documented.

Replace Line 7466 (note 2 lines earlier)
if test "${ac_cv_prog_PCRE_LIBS+set}" = set; theN
with
if test "${ac_cv_prog_PCRE_LIBS+set}" = set; then

Try Again.


$ cd /src/mysql-workbench-1.0.0/mysql-gui-common
$ ./configure
$ make
../../library_util/include/myx_util_functions.h:30:18: pcre.h: No such file or directory

Patience is running short, a find locates file in /usr/include/prce directory.


# I'm now no longer amused with this, I'm not a C developer, but basic code should compile.
$ cp cd /usr/include /pcre/* /usr/include
$ ./configure
$ make
MGTableEditor.cc: In constructor `MGTableEditor::MGTableEditor(bool)':
MGTableEditor.cc:229: error: 'class Gtk::ComboBoxEntry' has no member named 'get_entry'
MGTableEditor.cc:265: error: 'class Gtk::ComboBoxEntry' has no member named 'get_entry'

Well, why am I not surprised. More reading, now into the bowes of GTK. API Docs at http://www.gtkmm.org/docs/gtkmm-2.4/docs/reference/html/classGtk_1_1ComboBoxEntry.html
shows that get_entry is valid for 2.4.

Hmmm, more though, now is that 2.4, or is that 2.6. Well this trench digger just fell into the hole is was digging.

A bigger hammer, now the full blown Hydraulic Excavator

I won’t linger here, been down this path before, however more plumbing was required so it was an iterative process again.

cd /src
wget ftp://ftp.gtk.org/pub/gtk/v2.6/glib-2.6.6.tar.gz
ftp://ftp.gtk.org/pub/gtk/v2.6/glib-2.6.6.tar.gz
cd /src/glib-2.6.6
./configure --prefix=/usr


# checking for sigc++-2.0 >= 2.0.0 glib-2.0 >= 2.6.0 gobject-2.0 >= 2.6.0 gmodule-2.0 >= 2.6.0... Requested 'glib-2.0 >= 2.6.0' but version of GLib is 2.4.7
cd /src
wget http://ftp.acc.umu.se/pub/GNOME/sources/glibmm/2.6/glibmm-2.6.1.tar.gz
tar xvfz glibmm-2.6.1.tar.gz
cd /src/glibmm-2.6.1
./configure --prefix=/usr
make
make install


cd /src
wget ftp://ftp.gtk.org/pub/gtk/v2.6/atk-1.9.0.tar.bz2
bunzip2 atk-1.9.0.tar.bz2
tar xvf atk-1.9.0.tar
cd /src/atk-1.9.0
./configure --prefix=/usr
make
make install


cd /src
wget ftp://ftp.gtk.org/pub/gtk/v2.6/pango-1.8.2.tar.gz
tar xvfz pango-1.8.2.tar.gz
cd /src/pango-1.8.2
./configure --prefix=/usr
make
make install


# Requested 'pango >= 1.8.0' but version of Pango is 1.6.0
cd /src
wget ftp://ftp.gtk.org/pub/gtk/v2.6/gtk+-2.6.9.tar.gz
tar xvfz gtk+-2.6.9.tar.gz
cd /src/gtk+-2.6.9
./configure --prefix=/usr
make
make install


#pango-1.8.2.tar.gz
# checking for ATKMM... Requested 'atk >= 1.9.0' but version of Atk is 1.8.0
# checking for GDKMM... Requested 'gtk+-2.0 >= 2.6.0' but version of GTK+ is 2.4.13
cd /src
wget http://ftp.acc.umu.se/pub/GNOME/sources/gtkmm/2.6/gtkmm-2.6.5.tar.gz
tar xvfz gtkmm-2.6.5.tar.gz
cd /src/gtkmm-2.6.5
./configure --prefix=/usr
make
make install

Now the test.


$ cd /src/mysql-workbench-1.0.0/mysql-gui-common
$ make clean
$ ./configure
$ make
$ make install

WOOOOHOOO! no errors here!

One down, one to go.


$ cd /src/mysql-workbench-1.0.0/mysql-workbench-1.0.0
$ make clean
$ ./configure
$ make

make[3]: Entering directory
`/src/mysql-workbench-1.0.0/mysql-workbench/source/linux'
make[3]: *** No rule to make target
`../../../mysql-gui-common/source/linux/libwbcommongui.a', needed by `mysql-workbench-bin'. Stop.

Investigation

$ cd /src/mysql-workbench-1.0.0/mysql-workbench/source/linux # (current make directory)
$ ls -l ../../../mysql-gui-common/source/linux/lib*
-rw-r--r-- 1 root root 7410114 Jan 18 22:43 ../../../mysql-gui-common/source/linux/libmacommongui.a
-rw-r--r-- 1 root root 8006062 Jan 18 22:44 ../../../mysql-gui-common/source/linux/libqbcommongui.a

Similar library name exists, but not libwbcommongui.a Well that’s it, it’s finally now looks not a environement problem, more a software development problem. I’ve logged my findings at Bug #16604. I guess we will wait for a good response.

Was it all worth it! Well 90 mins of documenting, the longest blog I’ll ever write. If the next runner can move this forward to a getting a binary from compilation, that then starts, then it was worth it. But only in a reasonable time.

Database Modelling Software for MySQL

I’m stuck between a rock and a hard place. I’ve been using DBDesigner 4 from FabForce, an open source visual design tool, and apart from working around a number of bugs, I’ve found it practical to design from scratch. The big plus, it works under Linux.

With the announcement that this was being incorporated into MySQL, called MySQL Workbench, I was looking forward to getting my hands on it. I guess that was about 8 months ago. Finally about 6 weeks ago, Version 1.0.0-alpha was released for Linux. Unfortunately it didn’t work, would not even start for me. Logged as Bug #15421, which got marked as a duplicate of Bug #15218 (I could have sworn I did a search first). Anyway, this got promptly closed as Unable to reproduce, but I see it’s finally been reopened again.

Windows is at 1.0.2, Mac OS/X is at 1.0.3-alpha, and yet linux is still at 1.0.0-alpha. What gives? There seems to be a clear lack of communication as to what’s going on.

The problem is, I’m preparing presentatations for the Brisbane MySQL Users Group, and the MySQL User Conference, and I want to import an existing MySQL database into a modeling tool to generate diagrams. It seems that DBDesigner4 is not MySQL 5 friendly, and given it’s not supported anymore where do you turn.

In DBDesigner, I get the error. “dbExpress error: Invalid Username/password”. Upteen double checks and then searching on the web finally leads to
http://forums.mysql.com/read.php?113,32121,57081#msg-57081, and the trick SET PASSWORD FOR ‘some_user’@’some_host’ = OLD_PASSWORD(‘newpwd’);

So now I can get a connection to the database, there ain’t no model to import. It seems I’m now forced to go look elsewhere for a open-source and free modelling tool that runs under Linux.

Ultimately I can’t really complain, this is the primary problem with Open Source, on one side, it’s Free, the other side support, documentation and help can be a hit and miss affair, and there are no commitments to a release schedule or feedback as there is no money involved.

Still if after all this time, I’m forced to go out and buy a product, it will be a great shame.

How many installations, and just what are they doing?

Would it not be great if on the MySQL website there was a page of stats (updated daily) that provided statistics like number of installations, a breakdown of versions registered (not certain I like that exact word) , OS’s, countries etc. More specifically, some useful stats on the engine types in practical use, avg number of tables per database etc. Of course the types of stats could be limitless, but with the success of MySQL as well as other open source projects, more imperial figures on installations other then just downloads I think would definitely benefit given the current momentum. (Availability of information to competitors could be both a good and bad thing.) Perhaps figures can be shown in percentages, not actual numbers.

Anyway, nice idea you say, we can all come up with ideas, but how could you implement something like this.

I made a post yesterday and mentioned the thought of an XML storage engine. It seemed to stir some feedback. Those notes were actually taken from more detailed notes that I’ve never published so I thought it would be good to explain some of the background.

Many months ago at the Brisbane MySQL Users Group our informal discussions following the evening presentation turned to a more detailed analysis of the different engine types available within MySQL. A throw away comment by somebody like ‘We have no idea if BDB is really used?’ prompted me to consider the possibility of why can’t this question be answered in the future.

How would it work? Well for each installation of MySQL on a server (given you can have multiple installations of varying versions per machine) the option to provide a feedback loop is made available post installation, it could even be part of the install processs (probably not in .rpm). The feedback loop configures the following basic settings.

  • Frequency (Monthly, Weekly, Daily, Once Off)
  • Granularity (Summary, Verbose)

What would I have this feedback loop submit:

  • MySQL Version
  • OS Version
  • Country/Locale
  • Number of User Databases (excluding mysql & test)
  • Number of tables per storage Engine per database

More detailed information, not necessary for stats, but perhaps useful for MySQL internals would be output from the SHOW set of commands. (variables, status, table status,engines, innodb status). This alone is another topic, allowing collection of information that can be used to evaluate usage, possible tunings etc). Additional information such as number of triggers, stored procedures etc, would identify if this level of functionality is actually really being used out there.

By registering with MySQL to provide a feedback loop from your server installation to MySQL, you are provided with a unique Id, which enables clear tracking for MySQL, but provides clear anonymity for the customer to ensure we are not taking any propriety information from them.

We never want to hide the data of an installation that is provided to MySQL, it needs to be in the open. Why not make it an open standard of data exchange (using XML), and why not then enable the installation itself to use these statistics internally for some level of reporting.

XML allows for the files to be reused outside of MySQL directly. Using XSLT you could take these XML files for better presentation of statistical information. This would probably go against the ethos of MySQL with now information stored in XML instead of the database. So why not create an XML Storage Engine.

It would for lack of a better word be grossly inefficient, however it was more a Read/Only type structure, more you just write out a chunk of data in one go, and you always read the entire file. You don’t do partial updates (you could, but you rewrite the entire file), it leads itself to statistically information, written once only. Of course you would have to satisfy the search

What other possible extensions for an XML storage engine, you could store and read RSS feeds, even OpenDoc standards as implemented by Open Office for Documents.

Now, regardless of all the associated complexities like, how do you post add this functionality to older versions, how do you automate the scheduling of the feedback look across different OS’s, blah, blah, blah, the purpose is just to throw the idea around for a minute and see what falls out.

Additional statistics more useful for internal MySQL usage could pinpoint numbers of upgrades, from which version to which versions (more specifically if not current GA version). The biggest downside is as part of Open Source you can’t enforce gathering any of this information, so when the website says 1 million installations (it’s 1 million of people that have taken the time to notify MySQL, and that could be 10% or 50%, you just don’t know).

Could there perhaps be space here for a commercial product? I doubt it as this is very tightly coupled with MySQL, and unless there was seamless integration I doubt very few people would go to any trouble after installing MySQL.

It’s just an idea, but it’s nice to have ideas at least some of the time.

MySQL 5.1 is gaining some momentum

It wasn’t that long ago that MySQL released the GA Release of Version 5.0 with major new features (Oct 24 2005). It still took 5.0 about a year to go from alpha to GA, however I’d suspect a much shorter turnaround this time.

Version 5.1 is already at alpha, and the largest public functionality mentioned has been Partitioning. It is also anticipated that Storage Engines (a very handy MySQL feature in comparison to other RDBS products), will be a hot-pluggable API instead of a source re-compilation. Now I’ve never even looked at the Storage Engine code, but it’s been talked about a few times, particularly the CSV Storage Engine in general discussion.

There are some new features being documented now, but not generally available in an alpha build including Events Management (5.1.6) and XPath support within SQL (5.1.5).

Well I could use the events management immediately with some refactoring of code into a stored procedure. My current CRM allows for a system wide (ie all users) Phone Batch to be created for phone donations. Batches are locked into daily, so a new batch must be manually created each day. The ability to both automatically create a daily open Phone Batch, as well as automatically close the previous day’s while only a small task, removes a currently repetitive task from somebody’s day. Of course at present, this is planned, but via a cron job and wrapper script, now I could do it all in the Database.

XPath expressions, interesting, I use XPath Explorer a great standalone Java app, and Eclipse Plugin, but I guess now I can just do it at a SQL Prompt. What I’d really love to see is an XML Storage Engine.

XML Storage Engine

What about an XML Storage Engine? It would for lack of a better word be grossly inefficient, however it was more a Read/Only type structure, more you just write out a chunk of data in one go, and you always read the entire file. You don’t do partial updates (you could, but you rewrite the entire file), it leads itself to statistically information, written once only. So would it provide any benefit. Really only you have any sources of XML data and you just want to keep this information in an XML format, effectively the MySQL data file is an actual XML file that can be easily copied.

XML would allow for the files to be reused outside of MySQL directly. Using XSLT you could take these XML files for better presentation of statistical information. What other possible extensions for an XML storage engine could be possible. You could store and read RSS feeds, even OpenDoc standards as implemented by Open Office for Documents.

Would it serve any real purpose? Probably not. But with MySQL Storage Engines at least it’s easily possible.

Unit Testing A Database

In a recent job interview I was asked the question regarding Unit Testing/Automated Testing of a Database? An interesting question and indeed an interesting problem. I thought it was a good topic to describe what I’ve done in the past, and where I would go for a more complete testing environment given the opportunity of a entire XP project.

This is the approach I have implemented successfully in the past. It’s not a complete solution, however at the time with the client it provided appropriate coverage.

I don’t use a framework such as dbUnit to load data via XML, or specifically test data. XML is ugly to store data, and also with maintenance and comparison. I start with a pre-configured database of representative sample data, refer to my notes later on this, and then I use the tests of the application to perform the necessary data manipulation. This ensures that you are as close as possible to testing actual situations, and ensuring that any issues the application does (such as enter bad data, or RI failures) are caught appropriately.

Within this process an automated build test would first reset the database to a known set of data. I’ve also found that this helps as you can also recreate the schema if necessary. As part of Schema Design in an XP Development, I have two ways to create the database schema. You can create it from scratch (so there is always appropriate SQL to create the current version of the Schema, lets say BUILD_102. Alternatively, you can always upgrade between releases, for example between BUILD_101 and BUILD_102 with the appropriate upgrade scripts. Upgrade or patch scripts only move one version to the next version. It’s not possible in a production environment to simply recreate your schema for each release, however for testing and training you can. It’s also pointless after 50 releases to have to perform 50 patch releases from the original source schema for every automated build test.

This does lead to two paths necessary for creating a schema, but this can also be tested adequately in an automated way.

I also split my application tests suites that use the sample data into two buckets. Destructive and Non-Destructive. The reason is the Non-Destructive tests (i.e., non DML statements) can be re-run as many times as necessary. The Destructive tests (i.e., DML statements) can only be run once, before the database must be restored. Of course you can have the approach of setUp() and tearDown() within JUnit however it’s cleaner if you can extract this somewhat to a higher level, making the Unit Tests easier. By also running tests that don’t continually use the some data, or builds the data though test execution, you get a better coverage of different data sets. To give a few examples, You could create a Test that created a row of data, then edited the same row, then deleted the row. These are indeed valid, but if the first test fails, how do you know if the update and delete tests are also broken, they are by dependant by default and will fail, but did they really. If with your sample data, you created a new row, edited a different pre-configured row, and also deleted a different pre-configured row, you could eliminate the need to dependencies.

Now of course, there are situations where data must be specifically checked at the database level, for example it may never be displayed in your application, it may be intermediate information that is then summarised for display, or internal audit information held against data (for example Create User, Last Updated User), or data created by procedures or triggers. There are also situations when even within an application testing that can verify the data in a User Interface, you want to verify this at the source.

To this end I have a custom written JUnit extension that can perform specific SQL statements and comparison. I’ll need to write about this and provide this at a later time. (when I can dig it up)

Sample Data

Sample Data in the database is pre-configured, not in XML files, but so it can be managed by more primitive means, either by a database GUI interface or via SQL flat files or even text files. Why have pre-configured data this way? A few reasons.

  • It’s not coupled to your tests in any way, so it can be reused, for example as Training Data.
  • You can use database specific tools more easily say in loading the data in a relational way.
  • You can use the same database specific tools to export the data easily, if say you use an application to modify certain information.
  • You could more easily incorporate legacy data that is also being migrated if you use the same database specific tools.

Granted XML is universal in it’s data representation, it’s more self descriptive, but it’s a really pain to edit manually, and it’s very verbose when there are simply more primitive methods of this type of data management.

So we are creating a pre-configured data set, and an extensive one when possible. As I mentioned, the re-use capabilities for training or demonstrations really works.

Training Data

I have successfully with a number of systems, specifically CRM implementations used a Cartoon Environment for the sample data. There are a few reasons for this. First, most people I’ve ever met can related in some way to some set of the data. If they can’t, then there can read info online, or watch a movie etc, and get an appreciation from the representative data set, effectively I’m leveraging of the time and effort of others here, much better then a non-descript set of data.

You have the cartoon characters (e.g. Mickey Mouse, Donald Duck, Daffy Duck, Marvin the Martian, The Simpsons, The Flintstones, The Jetsons), use all the streets and rides as Disneyland as addresses, the animators as the users (e.g. Walt Disney, Chuck Jones, Stan Lee), you can use the different studios (e.g. Warner Bros, Disney, Pixar) for different states or countries, you can use shows or movies (e.g. Toy Story, Shrek) to group characters in other ways.

With this type of data, common attributes such as birth date, family units, nick names, people deceased etc, are all part of the available data. It’s surprising how much information you can find when using The Simpsons for example, of full names, addresses, interests etc.

It’s impressive when the CEO of a company is showing the application to overseas business partners, when his knowledge of the application (from his management perspective) is sufficient, there is no knowledge of the data really necessary to use or explain as it’s commonly used and generally understood.

At this point I would like to ensure that I correctly acknowledge the registered trademarks of Disney, Warner Bros, Hanna Barbera, Pixax, Dreamworks and that I am not using the names for any profit.

Summary

So this is what I’ve used in the past. What would I do in the future if I was charged with bullet proof testing of a database, even independent of an application, effectively 100% test coverage of the data. Well, this is an unproven approach, but I’d relish the opportunity to give it a full blooded test one day.

How to test the database with an automated test approach.

I’d consider the breaking down of testing into 3 areas. These being:

  1. Schema
  2. Data
  3. Business Logic/Referential Integrity

Each of this is effectively built on the the preceding points.

Schema

This would be quite straight forward it’s a flat comparison between schema’s, which could be managed via the appropriate products data dictionary tables using SQL. You could even simply compare 2 schema’s in a few simple SQL statements. You could also use the approach of export the schema definition, and then compare flat files. You will find some downsides to these approaches, ordering is a big thing, columns within a database table, or the order of the tables that are exported may not be guaranteed. However given appropriate standards are defined used of tools this comparison could easily occur.

Being able to verify patches between releases, and full installed schema’s are also possible. The schema is the easy part.

Data

Data could be tested in varying means. Counts, sums and sample comparisons, but it’s also just data, why not md5sum the entire data. Why not even dump the data to flat files, and use basic difference tools for comparison. One simple approach. Especially if you are loading data, using or manipulating it, then you can export and compare at a file level. This would work very well for data considered Read Only for the life of testing.
This format may allow you to compare data between two different database products, e.g. Oracle use for your transactional online processing, and MySQL use for Web Data or Management Summary Reporting application.

In order to test the data you need the schema, but how can you test the data without the business logic and Referential Integrity. Within MySQL you can easily disable foreign key constraints, or easily adjust the table type to a structure to ignore this syntax. This could allow you to run tests with and without Referential Integrity to determine the strength of your application.

Of course this is a static version of the data. Performing separate testing of DML statements directly against the database could prove a waste of time, unless your application was written in such a way, that your application database layer was a complete API. Still you would be simulating what your application is ultimately doing, so it could be overkill. You could apply the techniques of comparison with know results after a successful running of automated build testing.

Business Logic/Referential Integrity

The hard stuff. Well you could be half way there with adequate schema and data testing. At least you are then confident that core integrity exists.

The problem is also to do with application integrity verses database integrity.

Let’s take a percentage, it goes from 0 to 100. Now using MySQL for example, you would define this as TINYINT(3) UNSIGNED, giving you a valid range of 0-255, and by default a display characteristic of 3 characters (the (3) in this example is just beautification).

You application logic restricts the value into this column to 0 to 100. But do you enforce this at the database? Depends on your needs. If the application is the only way to insert and maintain data, then you could get away with it, if data can be managed from other external systems, you may have APIs that also need to manage it. What if you grant SQL access to DBA’s, could they accidentially mess it up. That song “It’s a fine line between pleasure and pain” comes to mind at this moment. I guess what I’m getting at here, in solely database testing you could easily insert 255 into a percent column and pass a number of data specific tests. I’d assume it would fail some as well otherwise your tests aren’t complete, but when using the application you could never test 255, as the client would never allow it.

There are a lot more issues in RI testing, Cascading UPDATE, DELETE rules for etc. And then when you work all that out, you have to start with Triggers and Stored procedures. I’m not going to spend any time here at this time.

Unforeseen Side Effects

Data is a strange beast. It’s the source of information, so I always like to go back to the data for comparison, however the lack of good data (most notably Legacy Systems Migration) can drive you mad. What good is it to have a new system but not be able to enforce an adequate level of Referential Integrity or Business logic due to incomplete historical data. In essence this has proven in the time I’ve also supported large systems, a good portion of the development cost in support, it’s bad data and/or the need for a simple application to have more complex rules to cater for so called incomplete data.

I’ll give you a trivial example. Gender. In the new system an organisation will always ensure they get the gender of a customer (let’s not wonder how they do it, it’s just an example). So the application is designed to support Male/Female, Reference Data may exist to translate M & F to Male and Female respectively for data storage efficiency. Check constraints, enumeration data types (which I don’t like) may exist. Reports may do side by side comparisons.

Now the company buys a competitor, and then gets their database of 500,000 customers, but they don’t record the Gender. Do you then relax all your great integrity? Do you introduce a gender of Unknown? But that’s only for display, the maintenance screens can’t allow you to select it, so you then need a different level of Reference Data management. Do you make an educated guess and correct the data? Does the customer do an expensive mailout campaign and data collection process to correct this information? So what’s the big deal anyway the customer asks? Well if it’s an organisation that sells hygiene products, you don’t want to sending out material on Shaving Cream to Women on your list and Body Wax to Men. However you can’t use that explanation to describe solely a database driven reason to the customer for the cost of introducing this data. How do you show business value to the customer, when they simply what the data available?

I know this is a trivial example, but if I had a dollar for every trivial problem that customers spent months on,verses the really hard problems, I be writing this from a much more comfortable and relaxing resort haven. (A ski resort rather then a beach resort)

Conclusion

So can you Unit Test a Database solely without an application? Yes. Would you want to? Maybe, to a certain degree. Depending on your type of data. If all your information is highly visible in data entry and data retrieval, you should couple testing more closely with your application. If your data is very generated and collated, rarely user entered but bulk loaded, such as Sensis information or GIS information, then dedicated testing all aspects of the database decoupled from the application could indeed make your application test easier, because it’s easier to identify bad data that the application creates.

Database Modelling within an XP Methodology

In an eXtreme Programming (XP) Agile Methodology approach towards software development the absence of adequate database design, or the scant regard of it, with the assumption that a framework and persistence infrastructure will take care of that can be a disaster in a larger enterprise solution. In essence it’s a scaling effect. The smaller the system, normally the smaller the number of users, amount of functionality and volume of data does not show the inefficiencies in database design as they can be masked by acceptable performance. But scaling up the system, or designing a large enterprise system the effect will become multiplied quickly. Of course using solid XP practices, the ability to make changes and integrate will be easier, but the amount and complexity of changes may be significant.

A more pragmatic approach is necessary in Database Modelling and Design, especially in a larger enterprise solution when using XP or an Agile approach. Assuming that the choice of Relational Database/s has been chosen, greater care is necessary and advanced preparation and planning required. Purists could argue YAGNI, but ultimately the customer will be distraught if the system is perfect in functionality and user interface but can’t handle the performance a production load of users or gradual growth of users when all is well in testing, demonstrations and customer training. The other catch is the need for additional disk space added monthly due to unknown requirements.

Additional considerations such as legacy systems data migration, database sizing, database growth, performance requirements, number of users etc can’t conform to a traditional XP approach. These tasks require varying lead times, for example the purchase and configuration of hardware and software need to be augmented with the XP approach.

XP is not for every project, and in the number of instances I’ve been involved with, certain considerations due to the environment, customer and usually management are necessary to be adjusted or tailored for the specific project. I’m not stating that an XP approach can’t apply to large scale enterprise Database Modelling, more that some adjustments particularly within the Planning Process are needed to integrate a more balanced solution.

The database is the foundation, I draw an analogy when discussing with friends to building a house. If you don’t have the core foundation correct, the slab, the essential primary fittings of plumbing, power etc, and a floor plan of key, important and known things, you will forever when building your house be spending additional time and resources to prop this up, taking away time, money and energy from the significant part of building the house so it can be ultimately used. (Thought: I wonder how you would build a house XP style. What’s the most important part for the customer. Something to ponder one day).

Are foundations perfect? No. Do they change and adapt? Yes, They do. However the cost is significantly higher, and the investment in getting it right the first time is invaluable.

I can’t count the number of times I’ve come onto an existing project, and the Database Designer for lack of a better word is a novice. It makes me shutter. It’s an expertise I’ve specialised in, however I’ve had to broaden my skill set, as this task is not used throughout a traditional SDLC project of 12-24 months, and it really saddens me when simple 101 mistakes are made and the downstream impact is significant, and management don’t know or realise the impact. Even worse is when I tell them what’s needed for a correction, and they look at the impact, and the decision is that this fixed known cost to correct is worse then any projected unknown down stream costs of maintenance and future development.

Ultimately I’ve got my way on projects more easily when I’ve also focused on Application Performance Tuning in projects, where I look solely at the application needs, not just the hard core DBA or System Administration tuning. A DBA doesn’t really care about the application and end user impact, they care about the figures in the database. In this situation, low level structural changes and associated costs are weighted against application performance ultimately necessary for the end user. I remember one project that took the development/user/testing/release team 3 months of work (probably ~ 200 man weeks of work) to implement and deploy a key structural change that I had identified and proposed as part of longer performance analysis period of this system. Of course when you had 6 full-time DBA’s alone, 33 remote distributed systems and 1000’s of users, the resultant impact and the future improvements possible were worth the investment. The need not to upgrade the hardware alone was worth it. This project was also over 10 years ago and a lot of techniques have changed since then. Anyway, back to the point of the discussion.

Let me give you an example situation where you use a traditional XP approach to software development, and how to weave in a more structured database design and modelling approach.

User Stories

As part of the gathering of initial User Stories from the customer, the Database Designer should be reviewing these and beginning to build a high level Logical Data Model. The Database Designer is not needed in any initial interaction with the Customer, however early reviews may clearly indicate gaps in stories due to historical experience that can then be feed back to the customer for considerations. Initially it should just be on paper, or a whiteboard, as initially this should correspond to the high level comparison with the User Stories and also the fluid nature of changing user stories. However it should highlight immediately known key entities, key relationships between entities, key integrations to external systems and areas that involve early input of volume estimates of data, perhaps with comparison to existing legacy systems.

For example, a rates billing system I was involved in had ~400,000 clients billed quarterly (4 times a year), and each bill had on average 10 line items (just looking a small sample, and getting a figure from the legacy system). Now it doesn’t take much to consider the size of tables which would record billing history (400,000x4x10) for each cycle, or GL reconcilation over 5 years (400,000x4x10x2x5 that’s 160 Million). Disk storage alone without any hint of the average row length, or indexes can be guestimated when a number of key entities are identified. Of course it could be out by a factor of 2, 5, or 10 times at this early stage, but not a factor of 100 times.

In a situation like this with any entity, even at the first mud map, I would be recording a number of indicators when possible. These would be:

  • Initial number of rows
  • Annual growth in rows
  • Projected rows after ‘x’ years

In addition I’d also flag each entity roughly for performance considerations by access, and also volume of transactions. This helps in identification of key performance considerations which may impact the Database Design in areas such as indexes (i.e. Disk Space) and schema optimisations after normalisation.

  • OLTP
  • Batch

For example, the billing process that creates quarterly bills and inserts millions of rows is both a Batch Process and a frequency of once every three months, as apposed to new accounts which is OLTP, happens at a regular quantity per day every day, and during the day.

Now, in the mud map these figures don’t have to be accurate, in fact in the billing system example, if the table didn’t have 10,000 rows it didn’t rate a mention, and 10,000 to < 400,000 was considered small. The limit here being the volume of the most key entity. Simply because a number of tables will be a factor larger based on this key figure, and this will swamp any insignificant tables. Again, this is week 1 of a potential 52 week project, it can afford to be vague in areas.

Spike

How do you term this in a XP environment? Call it a Spike. The purpose of a Spike is to explore options and evaluate risk. While in general the practice would be to throw away the results, in this case these Spikes should be kept and built on progressively.

Release Planning

As part of the Customer’s User Stories being estimated and prioritised within Release Planning, more refined Logical Modelling and even Physical Modelling is necessary. Of course until the customer refines priorities based on estimates, the actual order of implementation can’t be confirmed.

It is possible however a number of stories within an iteration may not relate together within the Database Model. How do you address this? Regardless of the actual attributes of these logical entities that are not available, a physical model representation can be built on necessary tables and relationships to ensure practical use of functionality.

Should the need for missing relationships within a Database Model impact the priorities of User Stories? No. Estimates should accurately reflect if additional work is needed for any given User Story, hence the Database Designer is necessary in the estimation process.

Iteration Planning

As part of the Iteration Planning it is key that the physical Database Requirements are in place to enable Writing Unit Tests and Writing Code for each Development Story for the Iteration. This should occur at the start of the Iteration, but is not before.

Daily Stand Up Meeting

This is the best opportunity during the Daily Stand Up Meeting to advise of structure impacts, or developers to request more involvement by the Database Designer in their specific task.

No system is perfect, and there will be times that the Database Model does not reflect the requirements by a coder for a task. At this time the developer is responsible to adjust their instance of the Database model to complete the coding task. The issue is; does the coder have the ability to modify the database schema when checking in? No. The responsibility for the Database Model is for the Database Designer, a difference to the normal coding practice where any coder can modify any code. The next section describes this reason in detail. At this point there is an inconsistency within the repository, and any automated testing for a build will fail. And this is what should happen. From the developers perspective their instance of the Code Ownership is correct, but for the entire iteration it is not.

One could argue this is not valid however, the cycle of Test, Code, Refactor, CheckIn while applying to the individual task, should also apply to the Database Design, however it should be at an iteration level not at a task level. To have your continuous builds breaking will however raise a lot of red flags, and I’m sure this approach will ensure that the tests which are broken as a result, will be corrected by the schema definition (the Code), there will be some refactoring if for example the Developer didn’t serve in the best interests of the bigger picture, the schema is checked in, and the continuous build bar is now green. Yipee. The bar is green, the code is clean, we can all go home now.

The Database Designer

The Database Designer should be dedicated resource in the development team for this task. For example in a team to 6-10 developers you would have 1 or 2 people. Even in a team of 2 your would have 1 person responsible. Unlike coding where any of the team pair up and work on tasks within the Iteration, the Database designer team is responsible for Database Design, like the Development Team is responsible for Coding. There are a few reasons for this:

  • They are ultimately responsible for the structure, the foundation, and this is the bigger picture that includes visibility and scope of further iterations and releases.
  • The skills are more specific.
  • They should also be a part/time developer of the team, so as to best understand the dynamics and also be part of the team.

Correspondingly, the end of the iteration should include an addition code review, that is schema related. A difference report of the schema definition at the start and end of the iteration should be compared, to ensure best practices in the larger picture as well as standards, optimisations, future performance considerations (e.g. indexes), future disk requirements (e.g. adding a new index to a 160 million GL table will take 4GB of disk space).

Other Factors

In fact as part of my upcoming conference presentation Overcoming the Challenges of Establishing Service and Support Channels I spend some time discussing Data Quality. Quite often this is the bane of support due to the complexities of software development to cater for data exceptions, or most commonly data anomalies due to historical data not meeting minimum RI and data specifications in your new system.

Ruby

Being a little despondent regarding Spring, a framework I’ve chosen to skill up in Read More, I’ve changed tack to investigate further Ruby. I was in a training demonstration of Ruby late last year, I’ve had other colleagues talk about it, and in a number of readings of late, Ruby has been making an impact, so time to delve in. I’ve got my working notes on my Wiki, and it took all of a few minutes to be operational. There appears a good wealth of reference material available including at least 2 online books. You can check out these in my Ruby References section.

Here is a comment from one of the current books I’m reading now. Beyond Java.

My partner and I decided to implement a small part of the application in Ruby On Rails, a highly productive web-based programming framework. We did this not to satisfy our customer, but to satisfy a little intellectual curiosity. The results astounded us:

  • For the rewrite, we programmed faster. Mush faster. It took Justin, my lead programmer, four nights to build what it had taken four months to build in Java (RB: They were using Hibernate,Spring and Web Work). We estimated that we were between 5 and 10 times more productive.
  • We generated one-fourth the lines of code; one-fifth if you consider configuration files
  • The productivity gains held up after we moved beyond the rewrite.
  • The Ruby on Rails version of the application performed faster. This is probably not true of all possible use cases, but for our application, the ROR active record persistence strategy trumped Hibernates’s Object Relational Mapping (ORM), at least with minimal tuning.
  • The customer cared much more about productivity then being on a safe Java Foundation. (RB: Highlighted)

As you can well imagine, this shook my world view down to the foundation.”

From Beyond Java. by Bruce. A. Tate pg 3-4.

Linux Format Reader Awards 2006

The Linux Format magazine is having it’s annual reader awards in a number of categories.

These include (I’ve include my picks after each category):

You can nominate at http://www.linuxformat.co.uk/nominate/. Nomimations close Friday 10 Feb 2006.

Adding to the Library Collection

I took the chance today to order some books from Amazon today to add to the library. Of course I’m still reading 2 current books Spring in Action and the MySQL Certification Study Guide in order to site the second MySQL Professional Certification Exam.

As with most things, you start off looking or reading on the web for something and you end up completely somewhere else. In this case, it was looking at Linux Software Labs (Australia) at the price of their Linux Distribution CD’s, which lead me to the book Beyond Java listed on their site. Called my local computer book store, but not being open (Boxing Day Public Holiday), lead me to go, “well I’ve been meaning to order some books from Amazon”, what were they again. So this lead me to coming up with a whole new list, and I figured for the cost of freight to Australia, I may as well order a few. So here is what I got.

Better, Faster, Lighter Java, MySQL (3rd Edition) (Developer’s Library) , High Performance MySQL,and of course Beyond Java.

The hard part now being the waiting 6-10 days.

XP Group in Brisbane

Brisbane has another XP Group. Just found out about it. Info can be found at http://groups.google.com/group/Brisbane-XP. I’ve been involved in some part in 2 previous groups in Brisbane.

I’m thinking about some ideas myself, I’ve got all the XP skills, however I’m now skilling up in Spring (a full-stack Java/J2EE application framework) and Hibernate (a powerful, ultra-high performance object/relational persistence and query service for Java). And I’ve got 2 other friends in similar positions.

Wouldn’t it be great if for 6 to 8 weeks, a few hours a week we could work on a project honning traditional XP as well as having some experience people in technologies helping others. Of course in comes back to some giving all to others, but I’m sure it doesn’t have to be that way.

Speaking at MySQL Users Group

I’m preparing to speak at the next MySQL Brisbane Users Group in Febraury 2006. My topic will be Know your competitor – A MySQL Developers Guide to Using Oracle Express Edition. You can get a full copy of my presentation slides at my Articles Page.

Having a strong background in Oracle, and having been using MySQL for the past 5 years, the release of Oracle Database 10g Express Edition (XE) as a Free offering (with limitations of 1 CPU, 1GB Ram, and 4GB disk) is an interesting move by Oracle.

I’ve written a number of recent comments on various Oracle/MySQL things including Responses to some Oracle v’s MySQL Questions, How can Oracle 10g Express Edition target MySQL?, Oracle 10g Express Edition Target Audience. Is it MySQL?, Oracle 10g Express, Free v’s Open Source and OFA.

The question could be posed, what relevence does this have to MySQL developers? Well, in some respects very little, but in others, both knowing about your competitor a little more, being able to see their offering, and in particular in comparision to MySQL can help in a level of understanding in Database differences. I am hoping that from the discussions, people will consider some approaches to design and development that is more “database compatible”, regardless of which database.

Will Oracle 10g Express Edition take off, well difficult question, there are many target markets, will it compete with MySQL in Open Source, hopefully my talk will sporn some discussion of peoples experiences in the various organisations and businesses represented in our meetings.

Upcoming Open Source Conference Presentation

I’ve been working recently on a paper I’m presenting to a conference in February 2006 titled Implementing Open Source for Optimal Business Performance. I’ve got the final glossy brochure yesterday so I now have something to show everybody. View Here (Be warned it’s a little bright)

The topic I have been asked to speak on is Overcoming the Challenges of Establishing Service and Support Channels. My notes are still in the early stage, but are available at http://wiki.arabx.com.au/index.php/ARABX_Articles:Overcoming_Service_and_Support_Channels

Conference Details on Ark Group Web Site

Review of Database Magazine Article – "The Usual Suspects"

In the “Australian Technology and Business Magazine” – December 2005 edition there was an article on comparing database products. Here are my comments, which I also plan to forward to the editor.

BTW: I’ve since also found this articles content on another site here. It seems that most if not all is the same.

In response to your cover story article “The Usual Suspects Four databases we suspect your business could be quite interested in.” which appeared in the December 2005 edition, I would have to sum up your article in one word “Disappointing”. Let me provide some feedback from my perspective.

You start by defining a scenario, which is the only approach you can take for a suitable comparision of database products due to diversity of features available in todays products. A good start, necessary to limit the discussion of features and functionality. However, you then specify some additional business requirements, for example “relatively small e-commerce” and “cost of the initial server and database software is certainly an issue.” Now, having worked for a number of small internet and e-commerce companies, you don’t have the budget for a Dell Quad Xeon processor machine, nor then the requirements for co-located hosting or dedicate networking bandwidth for your fancy new hardware as well as the additional staffing support costs. So immediately your scenario is more unrealistic.

The major sticking point I have is your 4 processor requirement. The most efficient and cost effective initial implementation is to lease dedicated servers, there are numerous reasons including cost savings, better hardware support, larger bandwidth capacity and easy growth path to start. You can also easily monitor growth and more quickly change needs then having a large initial hardware cost. I could continue regarding hosting, however this alone changes the requirements to using single or dual processor machines given your scenario. With this in mind the playing field is now completely different but a better reflection of the scenario. Your argument for “scale up to a small server farm”, also does not hold, because you can easily get economy of scale in splitting application server and database server, splitting OLTP and batch database requirements and other common practices, not to mention additional benefits such as redundancy.

My final comments on your hardware, specifically in relation to MySQL (including using Version 3), you can get significant performance from hardware given your small size requirements and even with modest growth on single and dual processor equipment. Other then opening remarks your article makes no further references to performance requirements or indeed any level of performance analysis of the products reviewed in this article.

You make scant reference to other database products, mentioning only one ‘Sybase’ in half a sentence in your opening and once again. In 8 pages, surely rounding the article to give a clearer perspective of the marketplace with even one paragraph to mention that there are many different database products both commercial and open source that service differing business needs. Other major products not compared at this time include Sybase, Informix, Ingres, PostgreSQL, MaxDB and Berley DB as well as many more.

Your choice of products is also not consistent or reflective of your scenerio. I’ll provide a few specific reasons. Firstly, you compare beta products against production products, if your criteria was current production products you should have compared using MS SQL Server 2000, however that would clearly provide a poor reflection in Microsoft due to it’s clearly dated product. If you allowed one beta product, why then did you not use MySQL 5.0 beta which was available at the time. While you have taken the effort to adjust your article to include references to MySQL 5.0, and you in turn choose MySQL as your editor’s choice, you should have been consistent throughout the article as you give mixed comparising referencing two versions of one product. Futher to this, you choose to use a dated Oracle product in 10g Release 1. 10g Release 2 has been available for a number of months. I would also question your decision to choose the more expensive Standard Edition over Standard Edition One, but this again could be soley due to your overspecd hardware.

If your rationale for including beta was cost based, then you did Oracle a clear injustice. You make again, only a half sentence reference to Oracle’s new released free product in the opening section. You mix more recent MySQL 5.0 information within your review of MySQL 4.1.14, yet you mention nothing of Oracle 10g Express Edition, for example it’s a free product much like Microsoft SQL Server Express, but also has similar limitation in 1 CPU, 1GB RAM and 4GB of disk but all the power and functionality of other Oracle Products, as well as default inclusion of web based administration tools with HTMLDB.

Your quick product summary (4 columns of information) suffers from a number of already mentioned points, however in relation to the only commercial product with a free offering, MS SQL Server Express, you clearly gloss over the limitations. 1 CPU, 1GB Ram and 4GB of disk is critical information, this should have been included in the product summary, you only go part the way. Regarding MySQL, should have clearly stated reference to $0 under GPL license. On that note, and mentioned in your detailed review, there are limitations in the distribution of MySQL within a commercial product and this is not in your summary.

Your article places no emphasis on performance or efficiency. Given your need to mention your testing on quad processor hardware, you make references to various limits of CPU across products, memory and hardware requirements as well as some generic maximum sizings, but nothing on performance, throughput and then growth potential, as this was part of your opening scenario.

It’s not possible to clearly date when this review of products was performed, granted the marketplace has changed rapidly in recent months, the fact that your article references Oracle 10g Express edition clearly includes changes were possible to the article in early November.

In an 8 page article, as mentioned you could have allocated one column to mention that the Database marketplace contains many more products. In particular considering you have included an Open Source product, and you selected this as product of choice, I feel this gives even more justification to at least giving credit to the emerging Open Source Database market. You actually place I recall only one mention to “Open Source” which is signifcant in the context of your choice. Other products would include PostgreSQL, Berkley DB, Apache Derby and even Ingres. While your article should clearly not need to analyse these at this time, by leading into this topic you provide clear opportunity for further discussion.

At the end of the day, while you provided a concise one page breakdown of features and certain limits, this technical information does not provide a clear benefit to an IT manager, or even a technical person.

Web 2.0 Design Patterns

In his book, “A Pattern Language”, Christopher Alexander prescribes a format for the concise description of the solution to architectural problems. He writes: “Each pattern describes a problem that occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice.”

1. The Long Tail
Small sites make up the bulk of the internet’s content; narrow niches make up the bulk of internet’s the possible applications. Therefore: Leverage customer-self service and algorithmic data management to reach out to the entire web, to the edges and not just the center, to the long tail and not just the head.
2. Data is the Next Intel Inside
Applications are increasingly data-driven. Therefore: For competitive advantage, seek to own a unique, hard-to-recreate source of data.
3. Users Add Value
The key to competitive advantage in internet applications is the extent to which users add their own data to that which you provide. Therefore: Don’t restrict your “architecture of participation” to software development. Involve your users both implicitly and explicitly in adding value to your application.
4. Network Effects by Default
Only a small percentage of users will go to the trouble of adding value to your application. Therefore: Set inclusive defaults for aggregating user data as a side-effect of their use of the application.
5. Some Rights Reserved.
Intellectual property protection limits re-use and prevents experimentation. Therefore: When benefits come from collective adoption, not private restriction, make sure that barriers to adoption are low. Follow existing standards, and use licenses with as few restrictions as possible. Design for “hackability” and “remixability.”
6. The Perpetual Beta
When devices and programs are connected to the internet, applications are no longer software artifacts, they are ongoing services. Therefore: Don’t package up new features into monolithic releases, but instead add them on a regular basis as part of the normal user experience. Engage your users as real-time testers, and instrument the service so that you know how people use the new features.
7. Cooperate, Don’t Control
Web 2.0 applications are built of a network of cooperating data services. Therefore: Offer web services interfaces and content syndication, and re-use the data services of others. Support lightweight programming models that allow for loosely-coupled systems.
8. Software Above the Level of a Single Device
The PC is no longer the only access device for internet applications, and applications that are limited to a single device are less valuable than those that are connected. Therefore: Design your application from the get-go to integrate services across handheld devices, PCs, and internet servers.

What Is Web 2.0?

In his article What Is Web 2.0 – Design Patterns and Business Models for the Next Generation of Software Tim O’Reilly gives a very detailed description of these seven principles.

1. The Web As Platform
2. Harnessing Collective Intelligence
3. Data is the Next Intel Inside
4. End of the Software Release Cycle
5. Lightweight Programming Models
6. Software Above the Level of a Single Device
7. Rich User Experiences

Core Competencies of Web 2.0 Companies

In exploring the seven principles above, we’ve highlighted some of the principal features of Web 2.0. Each of the examples we’ve explored demonstrates one or more of those key principles, but may miss others. Let’s close, therefore, by summarizing what we believe to be the core competencies of Web 2.0 companies:

* Services, not packaged software, with cost-effective scalability
* Control over unique, hard-to-recreate data sources that get richer as more people use them
* Trusting users as co-developers
* Harnessing collective intelligence
* Leveraging the long tail through customer self-service
* Software above the level of a single device
* Lightweight user interfaces, development models, AND business models

The next time a company claims that it’s “Web 2.0,” test their features against the list above. The more points they score, the more they are worthy of the name. Remember, though, that excellence in one area may be more telling than some small steps in all seven.

Some of the information provided is very interesting, I will be waiting with interest to see if this term “Web 2.0″ becomes something, or not.

Myths Open Source Developers Tell Ourselves

Some interesting points from this ONLamp article on Myths Open Source Developers Tell Ourselves

Publishing your Code Will Attract Many Skilled and Frequent Contributors
Myth: Publicly releasing open source code will attract flurries of patches and new contributors.
Reality: You’ll be lucky to hear from people merely using your code, much less those interested in modifying it.

Feature Freezes Help Stability
Myth: Stopping new development for weeks or months to fix bugs is the best way to produce stable, polished software.
Reality: Stopping new development for awhile to find and fix unknown bugs is fine. That’s only a part of writing good software.

The Best Way to Learn a Project is to Fix its Bugs and Read its Code
Myth: New developers interested in the project will best learn the project by fixing bugs and reading the source code.
Reality: Reading code is difficult. Fixing bugs is difficult and probably something you don’t want to do anyway. While giving someone unglamorous work is a good way to test his dedication, it relies on unstructured learning by osmosis.

Packaging Doesn’t Matter
Myth: Installation and configuration aren’t as important as making the source available.
Reality: If it takes too much work just to get the software working, many people will silently quit.

It’s Better to Start from Scratch
Myth: Bad or unappealing code or projects should be thrown away completely.
Reality: Solving the same simple problems again and again wastes time that could be applied to solving new, larger problems.

Programs Suck; Frameworks Rule!
Myth: It’s better to provide a framework for lots of people to solve lots of problems than to solve only one problem well.
Reality: It’s really hard to write a good framework unless you’re already using it to solve at least one real problem.

I’ll Do it Right *This* Time
Myth: Even though your previous code was buggy, undocumented, hard to maintain, or slow, your next attempt will be perfect.
Reality: If you weren’t disciplined then, why would you be disciplined now?

Warnings Are OK
Myth: Warnings are just warnings. They’re not errors and no one really cares about them.
Reality: Warnings can hide real problems, especially if you get used to them.

End Users Love Tracking CVS
Myth: Users don’t mind upgrading to the latest version from CVS for a bugfix or a long-awaited feature.
Reality: If it’s difficult for you to provide important bugfixes for previous releases, your CVS tree probably isn’t very stable.

Web 2.0. Not to be confused with Internet2

What is Web 2.0? Well the definitions out there aren’t clear and precise. Tim O’Reilly from O’Reilly Publishing has a detailed description at http://www.oreillynet.com/lpt/a/6228. (More notes from this below) His compact description is:

“Web 2.0 is the network as platform, spanning all connected devices; Web 2.0 applications are those that make the most of the intrinsic advantages of that platform: delivering software as a continually-updated service that gets better the more people use it, consuming and remixing data from multiple sources, including individual users, while providing their own data and services in a form that allows remixing by others, creating network effects through an “architecture of participation,” and going beyond the page metaphor of Web 1.0 to deliver rich user experiences.”

The Web 2 Conference (www.web2con.com) with the theme “Revving the Web” has some interesting content on the site.

Let me be clear, I didn’t know what Web 2.0 was 2 hours ago, I stumbled across an article Web 2.0 Principles applied by Yellowikis while research IT Outsourcing jobs in India/China etc (go figure). Anyway, the following summation prompted me to read about this topic a little more.

* Web-based (of course) and uses wiki technology; the same MediaWiki software that powers Wikipedia.
* Any user can both read and write content – adding business listings and editing them. To put it in ‘Web 2.0 wanker’ terms, it harnesses collective intelligence.
* Requires a significant amount of ‘trust’ in the users.
* Can be deployed via the Web in countries all over the world (see Emily Chang’s interview with Paul Youlten for more details on this aspect).
* Developed and is maintained by a small team (just Paul and his 14-year old daughter – both working part-time).
* Has fast, lightweight and inexpensive development cycles.
* Uses Open Source LAMP technologies (Linux, Apache, MySQL and PHP) – meaning it is very cheap to run.
* The content has no copyright and is freely licensed under the GNU Free Documentation License 1.2.
* Can and will hook into other Web systems, e.g. Google Maps. Indeed if it introduces its own APIs, then it will be able to be remixed by other developers.
* Relies on word-of-mouth and other ‘viral’ marketing.
* Requires network effects to kick in order to be successful (at least at the scale of disrupting the Yellow Pages industry).
* Yellowikis will get better the more people use it. The Wikipedia is an excellent example of this.

Taking a few lines from Tim O’Reillys detailed description as a quick taste for you to read more.

  • Wikipedia, an online encyclopedia based on the unlikely notion that an entry can be added by any web user, and edited by any other, is a radical experiment in trust, applying Eric Raymond’s dictum (originally coined in the context of open source software) that “with enough eyeballs, all bugs are shallow,” to content creation. Wikipedia is already in the top 100 websites, and many think it will be in the top ten before long. This is a profound change in the dynamics of content creation!
  • It is a truism that the greatest internet success stories don’t advertise their products. Their adoption is driven by “viral marketing”–that is, recommendations propagating directly from one user to another. You can almost make the case that if a site or product relies on advertising to get the word out, it isn’t Web 2.0.
  • 4. End of the Software Release Cycle – As noted above in the discussion of Google vs. Netscape, one of the defining characteristics of internet era software is that it is delivered as a service, not as a product. This fact leads to a number of fundamental changes in the business model of such a company:
  • One of the key lessons of the Web 2.0 era is this: Users add value. But only a small percentage of users will go to the trouble of adding value to your application via explicit means. Therefore, Web 2.0 companies set inclusive defaults for aggregating user data and building value as a side-effect of ordinary use of the application. As noted above, they build systems that get better the more people use them.
  • Contrast, however, the position of Amazon.com. Like competitors such as Barnesandnoble.com, its original database came from ISBN registry provider R.R. Bowker. But unlike MapQuest, Amazon relentlessly enhanced the data, adding publisher-supplied data such as cover images, table of contents, index, and sample material. Even more importantly, they harnessed their users to annotate the data, such that after ten years, Amazon, not Bowker, is the primary source for bibliographic data on books, a reference source for scholars and librarians as well as consumers. Amazon also introduced their own proprietary identifier, the ASIN, which corresponds to the ISBN where one is present, and creates an equivalent namespace for products without one. Effectively, Amazon “embraced and extended” their data suppliers.
  • Users must be treated as co-developers, in a reflection of open source development practices (even if the software in question is unlikely to be released under an open source license.) The open source dictum, “release early and release often” in fact has morphed into an even more radical position, “the perpetual beta,” in which the product is developed in the open, with new features slipstreamed in on a monthly, weekly, or even daily basis. It’s no accident that services such as Gmail, Google Maps, Flickr, del.icio.us, and the like may be expected to bear a “Beta” logo for years at a time.
  • …Support lightweight programming models that allow for loosely coupled systems…. …Think syndication, not coordination… … Design for “hackability” and remixability…

I could go on.

Other References
ZDNET Web2Con

Quotes from Web 2.0 Conference Web Site

I’m writing something about Web 2.0, but I got distracted by the random header quotes that appear on the website at www.web2con.com. Never being a Simpon’s fan, but it reminds me of those sites out there with all Bart’s blackboard quotes.

  • “Web 1.0 was making the Internet for people, Web 2.0 is making the Internet better for companies.” – Jess Bezos
  • “I personally use the web as an Intelligence Amplifier” – Bran Ferren of Disney
  • “Truly great companies aren’t built by the greedy, but by the passionate” – William Gurley
  • “Never underestimate the Internet. Manipulate it. Respect it. But don’t try to dominate it.” – Jerry Yang
  • “Operate as if you are in perpetual beta.” – Tim O’Reilly
  • “The value of your product is in inverse proportion to the cost of customer aquisition.” – Shelby Bonnie
  • “Most people think money is the key to reducing risk. Prepartion is.” – Mark Cuban
  • “The internet is the most underutilized advertising medium that’s out there.” – Mary Meeker
  • “In the era of Internet television, it will be as simple and cost-effective to create a microchannel as it is to create a Web site.” – Jeremy Allaire
  • “It used to be that Internet was considered a secondary market. Now it is the primary market.” – Sky Dayton
  • “Innovation is not the exclusive province of New Economy companies.” – John McKinley
  • “I’d rather do something interesting, solve an interesting problem, then do something boring and get rich.” – Louis Monier
  • “There’s always a curve ball! But that’s when the interesting stuff happens.” – Mark Fletcher

A better approach to using China for software development

India and China are the next powerhouses of software development, simply due to the numbers, but I’ve never heard a good report (maybe I have to dig deeper). My recent experiences are with Australian companies placing call centres in these countries, and almost always the language barrier is a clear limit.

As part of an upcoming conference paper I’m giving I have been looking more closer at the software options available, and I came across an interesting concept that has the background funding to get off the ground (a common problem in startups), and addresses a number of issues including the language barrier (which is less prominent with code).

Sinocode (www.sinocode.com) is the new generation in Offshore Development Centres (ODCs), delivering high value developer expertise from China. Our service offers:

  • Strong economic value in robust software solutions;
  • Proven western style management expertise; and,
  • Highly talented staff.

All of these drivers underpin our proven capability to execute. Execution, on time and on budget, is our key attraction.

(This is their sales pitch, not mine)

Some more reading:
Article in the Australian on 4th October 2005 Ernst places faith in China
An ABC radio interview on 30th August 2005 IT entreprenuer Lloyd Ernst

"JS Debugger Service is not installed" – Firefox debugger message

Attempting to run the Venkman JavaScript Debugger in FireFox 1.07 doesn’t work under CentOS 4.2.

It keeps launching an error window with “JS Debugger Service is not installed”. This does not appear to be an isolated issue from a Google search, however finding the solution proved a little harder to track down.

There is a thread at the Mozilla Bug Tracking site that sheds light on the problem.

1. You should first ensure you have the latest version of the debugger? (0.9.85, as far as I’m aware) (Available from http://hacksrus.com/~ginda/venkman/)

2. You need in the location bar on the browser enter about:buildconfig and see if –disable-jsd is specified?

Should you see that, you have your problem, the Linux distro has disabled this.
The solution is to download the tar directly from FireFox, however if your OS has previously installed by rpm, as in my case, you have to consider now running a different non rpm version.

Responses to some Oracle v's MySQL Questions

I was asked a few questions by a reporter thru a collegue, here is an extract of the discussion.

1) Based on your initial experience with Oracle Database Express Edition, what are your initial thoughts on the product in terms of meeting developer needs?

Installing Oracle 10g Express Edition was a breeze. (Article). In the past Oracle products have been more difficult to install, however this has gradually improved with the more recent version releases of 8i, 9i and 10g. This easy installation via rpm under Linux, in particular the inclusion of HTMLDB provides an ideal database environment that is functional in just a few minutes. This is an important first step in gaining initial developer support.

With the database installation, the Web Based HTMLDB Interface and a sound amount of developer articles online at the Oracle Technology Network ( http://otn.oracle.com), there is a lot of information to meet a developers need. In particular, HTMLDB which offers Database Administration, general SQL tools, easy data management and a complete web forms development environment. There is a learning curve in understanding this functionality, however it is an extensive and powerful tool for developers, and requires a minimum of syntax specific knowledge. With the release of Raptor in December, Oracle will have a visual tool for development, which will complement this offering in meeting developer needs.

2) As a mySQL developer, do you think it is on par with proprietary database offerings?

MySQL celebrated it’s 10 anniversary this year, and with the Release of MySQL 5.0, gave the open source community a product to match the strengths of others products with new functionality including views, stored procedures, triggers and a data dictionary. Oracle provided these features with Oracle Version 7, released in 1992, long before MySQL was even a company. In a commercial world, MySQL now offers the strengths of an Oracle Database Product, however Oracle has grown extensively and offers not only a commercial database product but an entire suite of products. MySQL is on par in a great number of areas. Granted they are catching up quickly, and the Open Source community offers features that commercial companies cannot. This includes the ability for open source companies to change, adapt and correct more quickly, bringing features and functionality to the marketplace more often, and more open support and knowledge bases enabling users to gain easier and open access. These two points offer downsides that commercial product offerings in turn offer back, including more commercial strength functionality, and more consistent and managed knowledge management, particularly in support.

3) Do you think the efforts of vendors like Microsoft and Oracle to gain mindshare with entry level versions of their products would entice mySQL developers like you to consider their higher-end database products?

I think that Oracle 10g Express Edition caters to different markets. For existing users and developers with Oracle, the opportunity to provide their products for resale at a reduced cost, with a zero cost Oracle license is a definite market and opportunity for these Oracle Partners. I think the education sector, including universities is another potential market, where a genuine free product can be used in teaching, and then practical application, and lead into usage within the professional employment. The third sector, the competitor market, which includes MySQL and Microsoft is more complicated. Oracle will need to go further to bridge the gap in convenience for MySQL developers. To sight a few examples, MySQL is generally available now in online hosting packages, on Linux Distros. It’s quick, easy and convinient to access, and is gaining significant popularity in the open source community, a good barometer in success in a developer driven environment. When Oracle can provide it’s products within a Linux Distro and you can see Online Hosting providers offering packages including the Oracle 10g Express Edition database, gaining market share will at least be more convenient.
One other strength, is MySQL being part of LAMP, and the significant wealth of open source products developed with MySQL, makes it hard for Oracle to ever gain ground in these continued product offerings. Perhaps if Oracle considered a bounty system, much like Ubuntu, and enticed developers to ensure Open Source products were Oracle compatible, would further entice MySQL developers. (Article)
The downside of these two examples, is there is no return in investment to Oracle, so it will depend on which markets are ultimately intended for Oracle 10g Express Edition. (Article)

4) Among the proprietary database vendors, who do you think has the best offering in your opinion, and where would mySQL fit in?

In this marketplace, there is a platitude of Database Offerings (can provide a list if you want), Oracle has always been the market leader in the commercial space, and there still is no other offering to match this. Historically, software development with databases was undertaken by IT professionals and there were a small number of commercial products available. Oracle gained a large market share in the past 20 years and has kept up with technology advancement. In the past 10 years, the Open Source community has changed the shape of the product offerings, and has also change the type of developers. There is a divide between the professional developers with qualifications and specific experience in software development in a structured development environment, and the new age of developers that have easy access to tools and information, and can easily contributed to an open source project that may have hundreds or thousands of developers worldwide, but they do at times lack solid grounding, and this can lead to bad habits. Depending on your requirements, a large commercial organisation, or a small webstore and blog, there are different needs, and you can’t therefore compare database products based solely on features. You need to include the customer requirements, as well as the ability for a database product to grow and adapt to your business in the ever changing environment.

I think in this day and age, there needs to be a tolerance between multiple database offerings within organisations. As the CIO or CTO of a large organisation, I would hope that different needs within the organisation would lead to two database products in use and synergy, both Oracle and MySQL.

How can Oracle 10g Express Edition target MySQL?

As I mentioned earlier, is MySQL a target of the new Oracle 10g Express Edition. Maybe not specifically, but let’s assume it’s on the radar screen. What can Oracle do to woe MySQL users and developers?

I see distinct marketing will be required for Oracle 10g Express Edition, marketing for example to existing partners must be different to “Competitor Marketing”. While I’m sure Oracle will now be able to get benefits from Parters trying to sell their products, now being able to reduce costs to customers (at least initially). Other partners that never considered selling developed applications due to the license cost may now reconsider.

But back onto MySQL. Oracle needs to target specific information to MySQL. OTN Technical Articles for Developers and Architects has nothing on MySQL. I can see the need for a Cookbook to attract MySQL users. The Oracle 10g Express Edition page has some information on developer docs, but not on Conversion, Migration and comparison. Some more opportunity here!

While on Migration, I attempted using the Oracle Migration Workbench to migrate my Version 5 database, without success. MySQL would be low on the agenda if a Version 5 migration was not available for a product that’s been in a pre-production release for most of 2005.

So, there is a clear need for some documentation, Quick Start Guides and How To’s for the MySQL community. Obviously being able to do a database migration also very high. But again this is not a practical application, it’s just the icing.
Until some Open Source projects actively engage in Oracle 10g Express Edition, it will always be a novelty to the MySQL community that won’t get off the ground.

I can see one way to jump start the process. Oracle could offer bounties, much like Ubuntu, to get some of the large open source projects Oracle 10g Express Edition compatible.
For example, http://www.phpbb.com, http://www.oscommerce.com, http://www.phpwebsite.appstate.edu, http://www.dotproject.net.

Now offering $200 per project, that would shake some things up!

Oracle 10g Express Edition Target Audience. Is it MySQL?

Just where is Oracle planning on targeting the new Oracle 10g Express Edition?
The obvious answer would be to counter the arch nemesis Microsoft, and the low end product offerings, like the MS SQL Server and the low end free engine MSDE. I didn’t realise to recently, that Microsoft have finally released the next version of MS SQL Server, being 2005, and at the same time provide a free cut down offering, strangely enough called “Express Edition”. It’s taken Microsoft 5 years. Makes you wonder if Database Technology is a high priority!

Is Oracle also now threatened in anyway by MySQL? I think not, however the continued growth of MySQL, it’s availability on any Linux server and in many distros, and now with the MySQL 5.0 GA release with features like Views, Triggers, Stored Procedures, Information Schema and ANSI SQL compliance allows MySQL to compare itself in features with Oracle. Certainly from our local MySQL Users Group, the inside MySQL news is it’s targeted against Microsoft.

Of course Oracle has been around for quite some time, and when I started working with Oracle 7 and Oracle Designer 1.2 back in 1996, these features were already available.

Oracle 10g Express Edition is a specialist database product from a database company, and with the wealth of knowledge that has lead the lasted version 10, everybody else has a long way to go to match this strength. From MySQL, the feature set even in Version 3 and 4, was adequate for application development in many areas, and the community is driving additional features to match other database product offerings.

I think it will be interesting to see what additional information Oracle releases targeted against the MySQL community. I’ve got a number of thoughts here, will post more details soon.

Oracle 10g Express, Free v's Open Source and OFA

In lunching with an old Oracle Friend, the topic turned to Oracle 10g Express Edition, and we discussed the pros and cons for organisations. The first thing he asked me was, “Have you tried loading the database larger then the 4G limit yet”. Some People?

In general the consensus seemed that it was viable for a small organisation, good for a startup company to get into a no cost entry solution, but appealing to those venture capitalists as you have a definite growth pattern.

I’m warming to more actively pursing Oracle 10g Express Edition, especially with HTMLDB, and at present while preparing to submit a paper for an upcoming Open Source conference which I hope is accepted, I’ve allocated some time on the significant topic of Open Source Verses Free. Stay tuned.

It dawned on me later in the day to more closely investigate the Oracle RPM installation and where exactly everything went in the FileSystem. I was surprised that it was not OFA compliant.

What if an existing Oracle customer with licenses was trialling Oracle 10g Express Edition and using the same staff.
Or more importantly, what if a new Oracle person picked up some bad habits from the Oracle 10g Express Edition, and when placed in a more commercial Oracle DBA environment got caught out?

I’m a Database Modeler, I’ve studied it, and I’ve practiced it, I’ve applied it and I’ve learnt some things from time to time of other “experts”, but nowadays, you put a database in somebody’s hands, like MySQL which is readily available, and all of a sudden you have people claiming to know and understand database design. Food for thought.

I hope from this new Oracle 10g Express product, also comes some good support from the Oracle Community to ensure a high standard in all things databases.

Oracle for FREE

Oracle software has always been free to develop with, however to deploy you have to buy a license and for web apps, it could never be limited to users, it was limited to CPU’s and that was expensive. Well now that have released
Oracle 10g Express Edition. A free little brother to Oracle 10g Enterprise. The software I understand is identical in the base sense, with a cut back set of additional features, and limitations to CPU/Memory/Disk usage. So providing your application, or applications all on one server never reach these limits you will be fine, but if overtime your product grows or database grows you may get caught out.

Still it’s not open source, but it is FREE, and for old Oracle people such as myself, a huge benefit as it keeps my skills a little more up to date.

How do I rate it. Well if installation is any guide, I am very, very impressed (and that’s a lot for me). After downloading the RPM here is what I had to do to get be operational.

# missing library found when running rpm of oracle
# CentOS 4.1 CD #3
$ rpm -ivh /media/cdrecorder/CentOS/RPMS/libaio-0.3.103-3.i386.rpm

$ rpm -ivh oracle-xe-10.2.0.1-0.1.i386.rpm

$ /etc/init.d/oracle-xe configure

Oracle Database 10g Express Edition Configuration
————————————————-
This will configure on-boot properties of Oracle Database 10g Express
Edition. The following questions will determine whether the database should
be starting upon system boot, the ports it will use, and the passwords that
will be used for database accounts. Press to accept the defaults.
Ctrl-C will abort.

Specify the HTTP port that will be used for HTML DB [8080]:

Specify a port that will be used for the database listener [1521]:

Specify a password to be used for database accounts. Note that the same
password will be used for SYS, SYSTEM and FLOWS_020100. Oracle recommends
the use of different passwords for each database account. This can be done
after initial configuration:
Confirm the password:

Do you want Oracle Database 10g Express Edition to be started on boot (y/n) [y]:y

Configuring Database…
Starting Oracle Net Listener.
Starting Oracle Database 10g Express Edition Instance.

HTMLDB interface accessible at http://localhost:8080/htmldb
I had to find a rogue Tomcat install running on my default IP and 8080, so I had to stop and restart using /etc/rc.d/init.d/oracle-xe

And it was that easy. Now for a person that used and Installed Oracle for a living it was more complicated, and at the best of times, you would need to allocate at least an hour for a basic install. This was complete and operational in under 10 mins.

Now I’ve never installed MySQL from RPM, but I’d like to compare how easy it is to install MySQL and also required tools for administration.