You can see a brief screencast covering this.

A cache serves as a logical middle layer between the application and the database.

A general pattern I’ve seen for caching:

KEY="data"
data = cache.get( KEY )
if (!data) {
data = db.query("slow")
cache.set(KEY, data, 30);
}


You’ll notice that the code first checks the cache to see if we get a “hit”. If not, we query the database and store the data back to the cache.

Now we know how to store and retrieve data in the cache.

What happens when a user sees the data, changes something, and then wonders why the data isn’t showing up?

Many cache services like Memcache and/or APC allow for a timeout to be specified so that we can make the data we put into the cache expire automatically.

Timeouts only go so far. Sometimes you need to lower the timeout to give the user fresh data, which marginalizes the benefit of caching. Ideally, your application knows exactly when the cache becomes invalid–when the user updates the object affected by the cache. A solid caching strategy also dirties the cache elements so the user never has knowledge of the cache!

We do lots of caching here to keep iContact running. We run a cluster of Memcache servers to store our sessions, and cache lots of data that’s frequently accessed to reduce the load on our database servers. On some pages, once the cache is warmed the number of queries goes down to 2-3 per page!

Agile teams tend to make the mistake of doing almost no up-front design before plowing into the work at hand.  Being agile doesn’t meant the work should be a complete mystery when you’re being asked to estimate and deliver it.  Here at iContact, we’re rolling out a checklist that helps us determine when we’re ready to move forward from an idea to an Epic and when a Story is ready to go into a sprint.

Epic Readiness – know what we need to build so we can design

1. Business Requirements
2. Technical Requirements
3. Documentation

Story Readiness – know how we’ll build it so we can estimate

1. Design
2. Stories
3. Documentation

We’ve developed some checklists to help us get to this level of readiness:

These tools are helping our teams plan backward from their grooming sessions to determine what level of technical vetting is necessary.

Must haves:
1. Request
2. Response
3. RequestHandler

These must haves provide an abstraction for the key elements of any web framework, whether it be MVC or otherwise.  Having objects to encapsulate the request, and send back a response allow for many tasks be simple.  Most importantly, they allow you to construct a test harness that doesn’t have to jump through hoops to process headers or handle sessions.

Extremely Helpful
4. Autoloading – Webs of require/require_once are no fun
5. Error/Exception handling – Displaying pretty output when errors occur
6. Logging – Log levels, different targets
7. Profiling
8 Security helpers – XSS/CSRF

Helpful, but probably belongs somewhere else
9. Database wrapper
10. Sessions and authentication

While these latter elements aren’t in the same class as 1-3, you’ll end up constructing most of these components to make tasks easier.  Wouldn’t you rather focus on building our application than building another framework.

How do your favorite web frameworks shape up?  Let us know in the comments!

A great example of approximate data shows up when you do any search through your favorite search engine.  Both the number of search results and the number of pages are approximations.  Sometimes you’ll find as you go to switch from page 3 of search results to page 4 that the search engine finds out there aren’t actually any more results to show.  Alternatively, as you page through the results the number of pages can decrease.  In both of these cases, the performance and accuracy of the information is tuned to the needs of the users.

Data shows that more than 50% of users only click on the first search result after conducting their search.  When you add up the entire first page of results accounts for >90% of clicks to results.  To more than 90% of users, the number of pages of further results are almost entirely irrelevant.  Search engines have realized this and prioritized the computation of these values.

At iContact, we just had an instance where we decided to change the user experience a bit to improve performance, however I doubt our users will ever notice.  When you click the “Create” tab inside our application we’d designed it so that if you had yet to create a message, we would hide the “Re-use a sent message” link.  The challenge was that this created a relatively expensive query that demanded a temp table and filesort.  When we went to optimize the query, we realized a few things:

1. The only people who wouldn’t see the link are people who just created their account.
2. We incur the cost of checking for this message to load an otherwise static page mainly used for navigation.
3. The cost to run the query gets more expensive the more messages you send and the less valuable.

We decided to make the link show up all the time.  This eliminated the query on one of the most common pages loaded in our application.  It’s reduced its page generation time by 100s of milliseconds on average.

What are ways you can get creative with your application to alleviate database hotspots?

1. Can data be approximated?
2. Can you update less frequently, perhaps even probabilistically?
3. Can you stop showing the data at all?
4. Can you email the data to the user so you have more time to process it?

What other suggestions to you have for changing application logic to create more performant applications?  Any case studies out there?

The query itself – Can it be improved?  Can you reduce the number of joins, return less fields, do less sorting, avoid pagination, avoid generating temporary tables, make better use of existing indexes?  MySQL’s EXPLAIN is your friend to help you see what you should improve about your queries.

Looking at an individual query, SELECTs can be first run through MySQL’s EXPLAIN.  The output might look something like:

*************************** 1. row ***************************
 id: 1
 select_type: SIMPLE
 table: pending
 type: ALL
 possible_keys: mailkey
 key: NULL
 key_len: NULL
 ref: NULL
 rows: 1396998
 Extra: Using where; Using temporary; Using filesort

Look for your possible_keys and the key that MySQL plans to use for executing your query.  Are there fields you could include in your where clause that might make it quicker to find the row (or rows) that you’re looking for?

In the query I’ve referenced, you’ll notice that no keys are being used (see the key column), and it’s using a temporary table, and a filesort (both noted in the Extra column).  No matter what anyone tells you, neither of these are intrinsically bad, but if you have too many queries using temporary tables and filesorts across your application, you’ll quickly need to get more disk I/O bandwidth to handle them or they become a Hotspot.

Are you asking the question in the right way?  Can you add an index? Can you eliminate joins?  Can you eliminate subqueries?  Can you delete data to improve performance?

Let’s walk through an elementary example optimization from start to finish.  We’ll start with a simple schema:

CREATE TABLE `test_opt` (
 `id` int(11) NOT NULL auto_increment,
 `owner` int(11) NOT NULL,
 `data` char(40) character set latin1 default NULL,
 PRIMARY KEY  (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

This table has three columns and you’ve been using it to store data for clients.  Each row has an auto-increment id, then a foreign key owner field, finally the data payload in the data field.

We’ll generate some test data using this small Perl snippet (test_opt.pl)

#!/usr/bin/perl

for ($i = 1; $i<=10000; $i++) {
 print $i%13 . "\t". "payload $i\n";
}

And the following commands:

$ perl test_opt.pl > test_opt.tsv
$ mysqlimport -u username -p -c owner,data --local test_opt_db test_opt.tsv

For more details on mysqlimport see the MySQL Manual

The primary queries you’ve been running against this table are single-row SELECTS, and  UPDATES using the PRIMARY KEY id.   Obviously these queries are quite fast.

mysql> explain select owner,data from test_opt where id=4393\G
*************************** 1. row ***************************
 id: 1
 select_type: SIMPLE
 table: test_opt
 type: const
 possible_keys: PRIMARY
 key: PRIMARY
 key_len: 4
 ref: const
 rows: 1
 Extra:

The explain for the query shows it’s using the PRIMARY KEY in the key column.  Since we’re using InnoDB the rows column is always an estimate, however in this case it knows we’re using the PRIMARY KEY and so there will be one row.

Even to look up all the rows for a given owner, we start to see a problem.  For example:

mysql> explain select id, data from test_opt where owner=10\G
*************************** 1. row ***************************
 id: 1
 select_type: SIMPLE
 table: test_opt
 type: ALL
 possible_keys: NULL
 key: NULL
 key_len: NULL
 ref: NULL
 rows: 10269
 Extra: Using where

You’ll see MySQL is now showing that possible_keys is NULL.  In the Extra field you’ll see Using where.  Since we’re not using any keys, this is a euphemism for a table scan (hence the 10300 rows estimate)!  If this table had millions of rows, it would scan all of them to show all the entries with the same owner.

In this case, the answer is easy–add an Index!

mysql> alter table test_opt add key owner (owner);
Query OK, 10000 rows affected (0.42 sec)
Records: 10000  Duplicates: 0  Warnings: 0

You’ll see the explain improve too:

mysql> explain select id, data from test_opt where owner=10\G
*************************** 1. row ***************************
 id: 1
 select_type: SIMPLE
 table: test_opt
 type: ref
 possible_keys: owner
 key: owner
 key_len: 4
 ref: const
 rows: 768
 Extra:

Problem solved!

In the real world, problems won’t always be this simple, but the key is to dig in and see what your options are.  Some challenges you might run into in the real world:

1. There’s already an index, but the queries are still slow
2. The table already has a bunch of indexes
3. The table is too large to add an index without extended downtime
4. The data isn’t clean enough for an index to be helpful (particularly when a UNIQUE index is appropriate)

All of these can be addressed and overcome with creativity and patience.

The keys of optimizing schema boil down to a few questions both when designing schema and evolving it:

1. What data do I need to store?
2. What questions/queries will be asked of the data — will it need to be joined, have ownership information, accessed for administrative purposes?
3. How will the table scale as data grows?

I’m not going to try and cover every trick for getting more out of your database here as that’s a whole topic in itself.  I will however link to several resources where you can read more about Schema Design, Query Optimization, and Performance Tuning of MySQL feel free to add your own resources in the comments:

1. Jay Pipes
2. MySQL Performance Blog

Making your site faster and more scalable by tuning queries and schema includes art and science so don’t forget to be persistent and use testing to verify whether your changes are providing the value you expected!

As we discussed in the last post, the whole idea of analyzing your database queries is to help speed up your application, reduce database load, and increase your scalability.

The easiest decision, is deciding which queries aren’t worth optimizing.

Try commenting the query out – In a dream world, database queries all take zero time, and cause zero load on the database.  Luckily, in the world of testing we can step into the looking glass!  Try commenting the query out, even put some static data in its place.  This helps you make sure you know how your application would behave if you didn’t do the query at all.

How often is the query run? – Is the query run often enough to cause enough load for others, or to have an impact on a sizable portion of your users?  If a query takes 10 minutes but is only run 4 times per day, then you’re really limiting your potential impact of optimization.  Instead of optimizing, can these queries be run in the background and save the results?  Even if you reduce these queries to 1 second each, the overall change in database load is not going to be significant.

The query is heavily indexed – Many of the queries in our application such as loading account data happen many, many times.  However, these queries are also heavily indexed and many of them are in tables that change relatively little.  These queries are very fast, return only a few rows, and run very quickly.  These queries are also typically not doing any sorting or grouping.  These queries are great candidates for caching, but they’re not likely to affect your database performance.

Experiment – You know your application best, so experiment a bit.  The most important lesson is to check every step along the way that what you thought would help performance is in fact improving.  Even with this warning, large scale applications are complicated and so is their performance.  Sometimes you’ll spend time optimizing things and find out they didn’t bear the improvements you wanted.  Make sure you learn from these and consider whether your changes are worth the complexity they’re introducing or whether you should just roll them back!

Now that we’ve discussed some quick ways to determine if the queries aren’t worth optimizing.  Stay tuned for Part 3.

iContact Labs recently decided to try and bring the toolbar into our home-grown framework and codebase so our developers could get similar feedback.  Luckily, we found out someone had done most of the work for us!  Some smart folks have extracted and older version of the Symfony toolbar and packaged it for inclusion in your project — called PHP Debug.

If you’re starting your own project, I’d recommend using Symfony or another framework that has a drop-in toolbar available.  There’s a toolbar plugin for Zend Framework called ZFDebug.

We integrated the toolbar into our existing code for profiling database connections and doing logging (using Log4PHP).

The information in the toolbar allows the developer to have data right at their fingertips for understanding both the baseline, and changed state for any page they’re working on or any new page they’re creating.  Having logging information at their fingertips also improves the quality of that information.

We’re starting to look at the data shown by the toolbar every time we commit new code.  For example:

1. How long does the page take to generate?
2. How much memory does it consume?
3. Are the log messages useful?
4. How does the page perform on different sized data sets?
5. How does the page load the second, third… time.  Does it use caching effectively?

Having this information right at our fingertips will be pivotal to continuing to improve the performance, quality, and scalability of everything we write.

We worked to change opens and clicks to process in the background using what’s called a Message Queue in between.  The side effect is that sometimes we’ll see a slight delay in processing opens and clicks.  We’ll be monitoring the latency of the opens clicks.

The week after we deployed this change, we started to see the benefit immediately.  The load for the databases decreased, and the opens and clicks buffered as designed.  During the first few days we saw as much as 30-45 minute delays in processing during some high traffic periods.  However, we saw no issues in the DB4 cluster which we’d been seeing consistent issues for weeks.  While these delays aren’t ideal, the fact that this cluster of opens and clicks didn’t bring down the database servers was a beautiful new trend.

Here you can see some of the buildup of opens and clicks during the first week, which created some processing delays for customers that weren’t within our tolerance, but we fixed those the following day by increasing the number of threads we had processing.  These days we see much lower queues, and only during very peak times do we see delays of > 1 minute.

Here’s the number of connections during the two week period before and after this was released.  You can see a decrease in the active connections and far less spikes in connections for the machine.  This means more responsiveness to customers and no running out of connections!

And a side benefit is that last week when we were experiencing database downtime, due to some corruption.  During this potential disaster, the silver lining was that we were able to build up a queue of 800,000 opens and clicks with delay being the only customer-impact.  In previous instances, we would have had to reap the Apache logs to process these opens and clicks.

1. Which queries are run the most often?
2. Which queries take the longest when run?
3. Which queries take the most database time overall?

There are many results you’d expect to see common among many web applications when looking at the list including:

• sessions – figuring out if the given user is logged in, and loading up their session
• logins – processing a login and assigning a user into a session
• permissions – should a user be allowed to access a given page
• billing checks – figuring out if a customer has paid

For each of the queries we get on the list, the key is coming up with a plan of attack and prioritized list of improvement areas.

There are several potential attack areas for a given query:
Do nothing, it doesn’t matter - The query is run often, but doesn’t really generate much load.  Queries that frequently run on tables that don’t change much can get cached by the MySQL query cache.  Queries with really low average execution times are often unlikely to yield high-impact optimizations or are already cached by MySQL.

The query itself – Can it be improved?  Can you reduce the number of joins, return less fields, do less sorting, avoid pagination, avoid generating temporary tables, make better use of existing indexes?  MySQL’s EXPLAIN is your friend to help you see what you should improve about your queries.

The application – How can the application work with the database to improve performance?  Can you eliminate the query entirely by not providing some data to the user?  Can you show different but easier to calculate data?  Can you reduce the frequency with which it’s accessed by moving more expensive data into drill-down pages rather than on dashboards?

Caching – This is actually related to your application, but deserves a category of its own.  How often does the data change? Does the user need the most up-to-date version?  Do you know when the data changes and you could invalidate the cache?

Writes – Can you batch them? Can you do them later?

In future posts, we’ll be delving into each of these areas to get more specific.