Joseph Piché » optimization

The Importance of EXPLAIN and SQL_NO_CACHE in Query Development

Joseph Piché — Thu, 08 Oct 2009 04:17:15 +0000

On average, I guess 35% of my over time at work involves MySQL query building, optimizing and re-writing. I find it an enjoyable challenge, and recently I re-worked an existing framework for a new site which put my focus almost solely on SQL, queries, and the schema.

With a normalized database schema, SQL optimization tends to focus on aspects outside the query, like making sure indexes get put in the currect locations. But I happen to be stuck with large and slightly-non-normalized tables. So, in order to write queries against these tables for a site that averages 30k hits per month, I need to be able to see what MySQL is doing with each query. For this I would be lost without EXPLAIN and SQL_NO_CACHE.

I don’t like writing tutorials, and I don’t really have time. But I love linking to great documentation. Most of what I know about MySQL I learned from their documentation, and I encourage even experienced query writers to make sure they know the importance of testing query times with SQL_NO_CACHE or using a development environment with caching completely disabled. When it comes to web applications, even microseconds can hurt in the long run.

Page Load Times Are Very Important

Joseph Piché — Sat, 22 Aug 2009 19:00:43 +0000

We’ve progressed pretty far since 56k dial-up days; no more waiting for pages to load in order to access information, almost everything is immediately accessible via multi-megabyte per second internet connections. But we would be naive to think that there have not been growing pains. In the IT industry, high turnover is both frequent and expected due to the constantly changing environment. (The exception is highly specialized fields.) Improving technologies changes the needs of the people and their expectations.

Reading Percona’s Web Scaling Blog, I saw a page on Front End Performance which outlined why web developers need to be conscious of users.

Web Site Speed Affects your bottom line. [...] Amazon’s experiment: Slowing pages by 100ms decreased sales by 1%

Highly-available internet access shifted our needs to security and privacy, but now we expect websites to be uber-fast. As a web developer, I need to listen to the needs of the user. The studies presented here show that those who do not develop sites up to user expectations will not get the traffic and revenue they desire.

This argument is very similar to the business scenario of the credit card purchase. Using a credit card is convenient to users, and even though it presents cost to a business to accept credit card purchases, if they do not accept them, they will lose business because customers expect it. Also, if a business abuses the power of the credit card by throwing in additional charges that are unnecessary (or unethical) customers will either not make purchases or not return for additional business.

So while producing a high-performance website—or trimming an existing one—may be costly, it is an important gesture, and will probably pay off.

“New” MySQL features

Joseph Piché — Thu, 06 Aug 2009 06:34:47 +0000

The more I use MySQL, the more I love it. I recently reviewed documentation for triggers when I ran into a pickle that needed to be solved in a short amount of time: ordering a query by TIMESTAMP.

I had a table with approx. 70k rows which had data continually inserted into it via cron job. I needed to query this table for the most recently inserted data given certain values. After re-organizing indexes and converting the table to InnoDB from MyISAM I was about to give up since (surprisingly) none of it gave any real improvement. The fundamental part of this query (the subquery in the SELECT expression which was ran over and over) was taking on average 0.022 seconds when ordering by a TIMESTAMP field which was automatically inserted by MySQL. When I used ORDER BY UNIX_TIMESTAMP(`field`) DESC LIMIT 1 instead, the query took 0.001 seconds on average. Since I didn’t wan’t MySQL converting it every time (really I don’t know what it does with times internally), I looked up some documentation and saw triggers. It saved my query and the table. I was able to make an additional integer field in the table and set a trigger on insert that did the conversion and stored the unix timestamp in the new field, then I could query the new unix timestamp field all I want and speed up this query even more.

It was an epiphany. Suddenly I saw SQL problems under a different light. I was even able to solve another complex problem by creating an event which would build rebuild a VIEW every hour. It was awesome until I discovered the production MySQL server was running 5.0 which doesn’t have events and I suspect handles VIEWs much more poorly than 5.1 which I was running locally. Of course I have no control over this server, and it handles uber-important data, so there is absolutely nothing I can do about it. So unfortunately I had to go back to the drawing board with that one.

Static vs Generated HTML and Content-Length

Joseph Piché — Mon, 27 Jul 2009 14:55:39 +0000

A frustrating aspect of writing in PHP is that almost all HTML is generated from PHP and is offered to the client after some processing is done. Under Ideal conditions, the amount of processing would be minimal, but in the case of the code I am currently working on at my job, it is not. I believe the root of the problem though is probably something most professional PHP programmers experience: the code they are working on is apart of a collaboration and being under a tight schedule, they cannot afford the resources to change what they would like.

The answer I most often get is “since it already works, let’s focus on something else.” With static HTML files on the server, apache can easily send the Content-Length, but even with PHP generated HTML, I could store all HTML in variables, then calculate the size and use apache’s Content-Length header to speed up page load times. Time is a valuable resource though, and being a good employee, I’m gonna (try to) do what I’m told.

Using LIMIT

Joseph Piché — Fri, 10 Apr 2009 18:08:37 +0000

The last couple weeks I’ve had to focus on building and optimizing MySQL select statements. It’s been an interesting journey as I’ve learned about choosing optimal field types when building tables, when to use a subquery over a left join, and how to use the information procedure analyse() gives. But the one thing I did not expect was the extreme performance increase LIMIT gives when used appropriately. Looking back, I should have guessed it, since when you limit a query to only retrieving a specified amount of rows instead of an indefinite amount, MySQL has to do a lot less work.

Today’s challenge though is a little different; I’m working with an existing table which has text fields as indexes along with integer indexes and primary key. This quarter million row table needs to be checked against for data, but I am only able to check against indexes on varchar(80) fields and not unique keys, so those queries are running orders of magnitude slower than the others. Unfortunately, this data is used in many other locations, so I am not able to do anything with it right now.