Speeding Up Movable Type

Pardon the dust around here recently. Have been growing increasingly frustrated with the time it took to post a new entry — as long as 80 seconds, and around 60 for guests to leave a new comment. The latter was becoming a genuine problem because people would get impatient and hit the Post button repeatedly, so I’d end up with duplicate or triplicate comments.

This isn’t the fastest CGI server in the world (trying to garner enough customers to warrant an upgrade), but I knew something wasn’t right. Post time had been increasing with the size of the database, which didn’t make sense since as a well-designed web app doesn’t care about the size of the database; the millionth entry should post as quickly as the first, pert near. Movable Type is exceptionally well designed, so I couldn’t believe it would contain a design flaw this glaring, and yet it was true; slower by the day, with just over a thousand entries. Other blogs with fewer entries on the very same server take only seconds to generate new entries. MT entry URLs are padded to ten million posts; one has to wonder what the post time would be like after a decade of blogging, even on a superfast server. Decided to get to the root of the problem.

I had read various theories about this in the MT forums. The most common suggestion is to not use MT Modules and includes. Instead, generate include files via Index Templates, then suck the output into your pages via PHP, SSI, or whatever you like. By putting re-usable content into index templates, custom MT tags still get properly parsed, but MT won’t need to generate the same data for every page each time the database is modified. Instead MT will crunch the include file once, and the http daemon will include it in the page at request time. I replaced all of my MT includes with PHP includes. Result: almost nothing. Post and comment time was virtually unchanged. MT modules were not at fault.

Next I tried removing everything fancy and stripping the templates down to “factory install.” No significant change.

A few times in the past, I’ve modified the timestamp of entries, which resulted in their numerical order not matching their chronological order. I’ve often wondered if this fact was causing MT to do more work. The only way I could think of to test this cleanly was to export all entries, create a new test blog, import all entries (thus re-assigning the post IDs to each entry), and testing post and comment time again. Some success! The “virgin” blog took “only” 62 seconds to generate a new post, and 40 seconds to leave a comment. Better, but still unacceptable. If I were to write a blogging system myself with PHP, post and comment time would be virtually nil, even if generating static pages like MT does. I was looking for something closer to five seconds.

Thinking about what one user had said about the <MTEntryArchive> tag being slow, I went into the Archiving section of the config and disabled Monthly and Category archives. Unfortunately, I now got a build error with my Archives include, since MT is no longer maintaining these archive types. So I made the Monthly and Category archive lists into static PHP includes and tried again. Whoa! Post time decreased to 18 seconds and comment time to 15 seconds. Not quite what I was after, but it was at least within the realm of the acceptable.

However, this means that new posts are no longer added to the Monthly or Category pages automatically. My first thought was to use the MTRebuild plugin to rebuild those archives once a night via cron, but of course that won’t work since I’ve disabled archiving for those two types. They can’t be rebuilt if MT isn’t aware of them. That means I’ll have to rebuild them manually from time to time by turning archiving on for those two, rebuilding, and turning it off again.

So. I’ve achieved a significant speedup (leaving comments should be less frustrating now) but at the expense of some of the cooler automated features. I still consider this a big flaw in MT’s design, and hope it will be worked out in MT Pro.

Music: Daevid Allen :: The Switch Doctor

17 Replies to “Speeding Up Movable Type”

  1. Pingback: thornography
  2. I have done research into this, myself. It seems that keeping archives, category or monthly, really slow down MT after a while. Especially category archives as they can get VERY big if you post to the same category frequently.

    What happens is every time someone updates a post in that category, the entire archive needs to be rebuilt.

  3. Hi Scot,

    The main performance issue in MT is the <MTArchives> tag. This is the tag used to generate the list of your archives (monthly, weekly, category, etc). We’re planning on optimizing this (as we’ve done in TypePad) so that it can be done in O(log N) instead of O(N), though it is very likely that the optimization will require using a relational database backend (eg MySQL, not Berkeley DB).

    For now, what I would recommend is to make sure that you only use the <MTArchives> tag in one index template, and then just include it everywhere else using SSI or PHP.

    Hope this helps,

    Ben

  4. Pingback: The Blog Herald
  5. Thanks for posting the results of your experiments. And that’s impressive that Ben came by with his feedback. Post something about Windows XP and see if Bill comes by… or maybe about the Iraq war to get George’s response. :-O

    Jeb

  6. I don’t have anything useful to say, so skip this if you’re in a hurry.

    I just wanted to say that since you’ve posted this, I’ve been making a conscious effort to post more comments to your site. Not that my postings have been spurious, but it’s just that since I saw this post, I’ve been pushing myself the extra tenth-of-a-mile to click the Comments link and post the responses your entries usually elicit, but that I don’t usually comment on because it seems like I’m always in a hurry. I figure it’d help me be a better blog-reader (good blog readers let the author know they’re out there and paying attention), and it’d help me understand the ramifications of what you’re saying a little better.

    So. Here’s my observation: yup, sure does take a long time to post. This is a problem. Or will be in about 12-24 months when all these other MovableType blogs are as ‘mature’ as yours is and those all start having real long Post times.

    Of course by then, I’m sure the problem will have been corrected. Because it was caught now.

    Looking forward to that log(N) patch now.

    Not looking forward to having to learn MySQL.

  7. Fortunately you don’t need MySQL to implement it with MT. At least not beyond the point of simply creating a database. The installer scripts take care of the rest.

  8. you shouldn’t have mt write all of your individual archives into one directory. break up the load into several directories by using an archive directory structure based on date.

    example:
    /archives/2003/11/02/001119.php

  9. Charles, why would this affect rebuild speed? The rebuilding process has no / needs no awareness of the number of files in teh archive directory (and besides, a thousands files in a dir is not large).

  10. Intersting approaches via the MTArchives tags– gotta look into it. I had some slow rebuilds a few weeks ago that was relieved by going into the mySQL databases and removing some overhead (using phpMyAdmin).

    In addition, much as I love MT, I loate how it tosses all posts to a single, never ending category archive… that is bound to gunk up the works. I have done a clumsy splitting approach to make my main category archives list only the last 15 posts in that archive, and then link it to a secondary archive that displays “all the rest”:
    http://jade.mcli.dist.maricopa.edu/alan/archives/000119.html

    Bring on the MTPro!

  11. Actually, I got around the multi-comment posting issue by tweaking some of the PeRL scripts so that I can gray out the Post buttons after they’re clicked.

    However, I too am beginning to get pissy about the rebuild times and sadly I don’t have the option for MySQL on server that I’m currently being hosted by. I have no idea what I’ll end up doing, give that the “fix” will only be supported on certain MT options.

  12. Hi, I’ve had the same problem in http://www.rollblog.it, a MT blog with over 11000 comments and 1700 entries.

    I solved the rebuild archives issues using the MTRebuild plug-in. As you wrote, you cannot use that plug-in for rebuilding categories if archives aren’t activated in the Archive Configuration page.
    What I’ve done?

    Since you can activate/deactivate that options using an HTTP/GET call with an url like this:

    http://www.yourdomain.com/mt-cgi/mt.cgi?username=theusername&password=thepassword&__mode=cfg_archives_save&blog_id=thelogID&_type=blog&archive_type=Category,Individual,Monthly

    I scheduled a ColdFusion script that does the following:

    – make an HTTP/GET call to activate the Category and Monthly archives
    – run the mtrebuild plug-in using the appropriate options
    – make an HTTP/GET call to de-activate the Category and Monthly archives

    I run it every night.

  13. Since I run a Movable Type installation with over 20,000 posts and probably well over 40,000 comments (haven’t counted for a while), I was wondering this myself. Earlier this year I wrote a basic templating system in Python. More flexible in some ways than MT, less powerful in others.

    It’s two orders of magnitude faster.

    There’s something wrong here. I’m going to spend some time profiling MT and see if I can work out where the time goes.

  14. Hi there,

    Came by your page entry at Google; I’m in the middle of moving my MT-run blog to a new hosting provider, and figured I’d take the time during the changeover to optimise performance. Whole-site rebuilds on my old server were not only taking forever, but also prompting irate emails from the hosting provider.

    I’ve been converting any “supporting” information (lists in left and/or right columns) into PHP include() functions calling pre-generated blocks using Index Templates. This has really helped rebuild times. Given that every new entry and/or comment prompts a rebuild of some index pages, it makes sense to make things like the list of monthly archives a one-time only build, rather than running the same loop on many pages. That’s already helping a lot.

    Another thing I’ve tried, which seems to help, is the management of comments. One of the first things I always do with MT is to get rid of the pop-up window (out of personal taste), preferring the comments to be displayed inline on the page. However, generation of this page can be slow.

    What I tried, and what seems to work, is to redeploy the now-defunct “comment listings” template that was used for the popup. I stripped it down to the bare bones; just the necessary tags and comment/trackback loops.

    Then, where I would normally put the comment loops within the individual archive page, I replaced them all with a call:

    <?php include(‘<$MTCGIPath$><$MTCommentScript$>?entry_id=<$MTEntryID$>’); ?>

    Now, because there’s no loop through the comments when rebuilding the page, rebuild time does seem to be improved. It does mean that the overhead on reading pages is increased, but for my seldom-read blog that’s more than acceptable.

  15. Scott –

    Moving to an includes system is definitely the best way to improve build times. You could, in fact, see MT’s include falls as a workaround for those who don’t have a technology like PHP to call on. Of course, by taking them “out of the system” those pages or includes lose the ability to have awareness of other MT modules, tags, etc.

    I get the impression that MT3 will improve this situation radically.

    Good tips, thanks for posting.

Leave a Reply

Your email address will not be published. Required fields are marked *