You are here: Home / Technologies / Databases / The case against using rpm packaging for MySQL

The case against using rpm packaging for MySQL

by

In some environments using a distro package management system may* provide benefits including handling dependencies and providing a simpler approach when there are no dedicated DBA or SA resources.

However, the incorrect use can result in pain and in this instance production downtime. Even with dedicated resources at an unnamed premium managed hosting provider, the simple mistake of assumption resulted in over 30 minutes of unplanned downtime during peak time.

One of the disadvantages of using a system such as rpm is the lack of control in managing the starting and stopping of your MySQL instance, and the second is unanticipated package dependency upgrades.

So what happened with this client. When attempting to use the MySQL client on the production server, I got the following error.

$ mysql -uxxx -p
error while loading shared libraries: libmysqlclient.so.10: cannot open shared object file: No such file or directory

The server was running MySQL 5.0.27 via an rpm install.

$ rpm -qa | grep -i mysql
MySQL-server-standard-5.0.27-0.rhel3
MySQL-shared-standard-5.0.27-0.rhel3
MySQL-devel-standard-5.0.27-0.rhel3
MySQL-shared-compat-5.0.27-0.rhel3
MySQL-client-standard-5.0.27-0.rhel3

With no access to this managed server the information was relayed to the hosting provider and some time later we found the production website down. Some 30 minutes later we found that to fix the rpm problem, a dependency upgrade has also caused an automatic upgrade from 5.0.27 to 5.0.88.

While upgrading is not necessarily a bad thing, the lack of planning including a backup, a scheduled window of downtime and any level of testing is simply a poor cowboy approach to DBA management.

Comments

  1. says

    This really isn’t RPM’s fault. If you screw things up, they will screw up. This doesn’t depend on what packaging system you use.

    If you’re running a serious production infrastructure, packages should not simply “get upgraded”; you should be in full control of all the software repositories and add packages into them after they’ve been tried on a non-production system and when ready to do so.

    This is the real problem, not RPM.

  2. ronald says

    @Mark R Thanks for the comment. I never said it was RPM’s fault and that you can never use RPM’s, this is a description of a case when not to use RPM. I agree people should definitely try and confirm on a non-production systems, however to *assume* your expense hosting provider knows what they are doing is a risk to your business viability, as indicated in this real life example. Used appropriately RPM is a viable approach, however it does come with risks (and sometimes unexpected) when not managed correctly.

  3. says

    Uh, “The case against using rpm packaging for MySQL”?! This has nothing to do with RPM, and everything to do with poor choice of hosting provider. Maybe it should be “The case against using X hosting provider”.

  4. Oliver says

    Why were you running an upgrade in the first place during “peak time” anyway, and without a scheduled window of downtime? None of what you have said made sense. Problems can happen regardless of the way you deploy MySQL; not planning upgrades and not understanding the entirety of the system will cause downtime like this.

    Don’t blame the packaging tool when lack of foresight is a more likely cause.

  5. mikeh says

    Some of the packaged versions of MySQL do some annoying things. The RHEL/CentOS RPM versions of MySQL will start MySQL upon install, which always grates on me. The Debian/Ubuntu packages annoy me even more by logging to syslog and insisting on checking all MyISAM tables on startup.

  7. Bryan Ryan says

    30 minutes later sounds too long to figure how to fix a standard rpm problem. If it was really a standard rpm installation, then simply remove it and install the old MySQL rpm version.

  8. Tom Hanlon says

    What should be used then ?

    Me.. I like a binary tarball as my first choice, second choice build my own package.

    —
    Tom

  9. Rene Lauer says

    This has nothing to do with RPM packaging, it was just the hosting provider that failed here.

  10. hartmut says

    Doesn’t RPM provide a mechanism to keep certain packages at fixed versions like .deb does?

  11. apz28 says

    You are not using rpm correctly. Learn it first before blaming it! You can check with below options

    rpm -i -test ….

    Cheers

  12. says

    Sounds like the problem was an admin that didn’t understand how to properly use up2date so that MySQL doesn’t break. Also, as others have mentioned, why were packages being upgraded on a production server during business hours (not in a maintenance) without first verifying in staging that the package updates would be successful and not break the database?

    Sounds mostly like bad change management process and nothing at all to do with RPM package choice for MySQL. Personally I use RPMs for all of my several hundred production MySQL servers because it’s vastly simpler for management, installations, and upgrades. Also, using RPMs for MySQL doesn’t limit you to the RPMs that Oracle puts out – you can roll your own and have everything be custom, although I’m sure you know that already :)

Trackbacks