In some environments using a distro package management system may* provide benefits including handling dependencies and providing a simpler approach when there are no dedicated DBA or SA resources.

However, the incorrect use can result in pain and in this instance production downtime. Even with dedicated resources at an unnamed premium managed hosting provider, the simple mistake of assumption resulted in over 30 minutes of unplanned downtime during peak time.

One of the disadvantages of using a system such as rpm is the lack of control in managing the starting and stopping of your MySQL instance, and the second is unanticipated package dependency upgrades.

So what happened with this client. When attempting to use the MySQL client on the production server, I got the following error.

$ mysql -uxxx -p
error while loading shared libraries: libmysqlclient.so.10: cannot open shared object file: No such file or directory

The server was running MySQL 5.0.27 via an rpm install.

$ rpm -qa | grep -i mysql
MySQL-server-standard-5.0.27-0.rhel3
MySQL-shared-standard-5.0.27-0.rhel3
MySQL-devel-standard-5.0.27-0.rhel3
MySQL-shared-compat-5.0.27-0.rhel3
MySQL-client-standard-5.0.27-0.rhel3

With no access to this managed server the information was relayed to the hosting provider and some time later we found the production website down. Some 30 minutes later we found that to fix the rpm problem, a dependency upgrade has also caused an automatic upgrade from 5.0.27 to 5.0.88.

While upgrading is not necessarily a bad thing, the lack of planning including a backup, a scheduled window of downtime and any level of testing is simply a poor cowboy approach to DBA management.