cookiestack

Testing Rails for the masses!

Tue, 04 May 2010 18:36:00 +0200

With recent posts about doing your share to help the Rails community, this post will be interesting for the folks that have yet to set up their Rails testing environment. Why is this a big deal you might ask? Well, if you run through the Rails guide for contributing you will notice that there are quite some steps involved to have a test environment set up. You need to set up postgres, mysql and in some cases even memcached. I’m not even talking about the cases where more exotic databases need to be tested. So this all makes it harder to become one of the selected few. Until now that is.

Not so long ago I came across the awesome Vagrant gem. This gem allows you to easily manage and share environments using Oracle’s VirtualBox. Vagrant is amazing, and the first use case I could think of was building a test box for Rails, which I did. You can find the code for the test box in this repository, the Readme contains all the instructions necessary to get your Rails Test environment up and running, making Rails contributions easy for all! If you want to make a short cut, here is a gist with instructions for the prepackaged version

If you have followed the instructions succesfully, you will be able to log into the Virtualbox and run the tests. You can then find an issue on https://rails.lighthouseapp.com and create a patch in the rails directory which is available in your IDE of choice (the directory is shared with the Virtualbox).

Currently, I have been able to get the suite running on master for 1.8.7, 1.9.1-p376 and Ree, thanks to RVM. I hope to get it working for other rubies as well. On the issue tracker I have listed all the issues that are still open for other Rubies. Most of the issues are “Nice to have” such as Ironruby and Maglev support, but it will take a while before they completely support Rails. JRuby on the other hand is probably already compliant and should be part of the testing environment.

Future work

Obviously Rails isn’t the only environment that could use an environment like this. So one of the TODO’s is to extract the Rails parts out of the recipes. Another important TODO is to make installation easier (maybe gemify this repo?). And last but not least some shortcut commands to create and apply patches in git.

Feedback

If you have ideas or suggestions for improvement. Please comment below, create an issue or go even further and do a patch.

Disclaimer

I will use this box to test my Rails patches, but I haven’t done much yet to be honest (So Ryan, please be kind to me ;-))

Jeroen

only index when you need to

Sun, 11 Apr 2010 13:09:00 +0200

a.k.a. tricks to be aware of when indexing a user model

black-cat cookie

Thinking Sphinx has been a god send for us, it brings the power of full text searching with such a beautiful dsl. Its powerful, configurable, flexible and every other kind of ful or ble you can think of.

I could go on for paragraphs on my love for Thinking Sphinx, and to some degree Sphinx, but lets get to the point, otherwise I will waffle on for ages.

When you save an indexed model, lets say a Location, two hook methods are called; a before save to toggle the delta and a after commit to run the delta index. As you can expect, if the delta isn’t toggled then the delta index isn’t run, but if you are saving a model wouldn’t you always want to toggle the delta and reindex it?

Thinking Sphinx is smart and likes to try to reduce the need to index by only toggling the delta if data in the indexed columns has changed. This works fine if you only index simple information (fields or relationships), but doesn’t work so well when you group two fields together in a custom SQL statement.

eg. Works fine:

define_index do
  indexes :name
  indexes :city
  indexes :description
end

Works fine:

define_index do
  indexes :name
  indexes [address_line_one,
           address_line_two,
           suburb, town_city], :as => :address
  indexes :type, :as => :location_type
end

Does not work fine:

define_index do
  indexes "LOWER(`locations`.`name`)", :as => :name, :sortable => true
  ...
end

The problem here is that TS can not check if something has changed when it has SQL in the definition, so you are always going to have the model indexed in the delta, even if nothing changed.

But is this really an issue? Does it really matter?

Over the last couple of months I saw a interesting problem occurring only on production, and I only got to see it because of the volume of requests. In an app I manage we have indexed our users for easy searching and filtering, but for some reason the user delta index was being reindexed On each request from a logged in user, even though the information indexed for a user wasn’t changing.

To understand the problem, cause and solution, we need to understand the infrastructure a little.

We use Authlogic for our authentication but the important part to note is that we also have user tracking turned on, which means information about last request date and time, as well as last login etc. is updated on each request.

When a user is updated Thinking Sphinx uses a before save hook to toggle the delta if any of the indexed information has changed. In our setup we don’t use the standard field definitions due to inconsistent sorting, so instead we use custom definitions which LOWER all the values, making sure sorting is standardized. But as explained above, this is where the problem lies.

We can correct this on a model to model basis by overriding the handy method indexed_model_changed? by using code similar to this:

def indexed_data_changed?
  indexed = [:first_name, :last_name, :email, :role]
  self.changed.any? { |col| indexed.include?(col.to_sym) }
end

Now when a user is updated upon each request the delta won’t be toggled and the index won’t be run on subsequent requests, reducing server load on delayed job and correcting a small bug which only showed its head when under the right conditions.

So what should you take away from this?

If you use SQL in your define_index blocks, make sure you override indexed_data_changed?

side note: Thinking Sphinx is the bees knees, read the source, understand the code, and spread the indexing love

some handy rack-rewrite rules

Mon, 05 Apr 2010 18:42:00 +0200

Every decision you make for a client should be based on sound reasoned advice. But what if you can’t give that advice because your facts are not accurate?

This was a recent problem we ran into when looking into some page stats on google analytics, we found that a page was appearing twice in the stats, one without a trailing slash and one with. I also saw in Google Webmaster Tools that sites were still linking to index.html pages (the old php site convention), which meant if people followed the links they would get a 404.

This might not sound like a big problem but it is, if your page view stats are off then your facts are off then your advice is off. Page views and navigation is a big part of advice to a client, if it be looking at current trends, or pages which can be improved, analytics should be as correct as possible.

That story aside, the problem is not hard to fix, in fact, you have three possible solutions to choose from.

Apache rewrite rules
Analytics filters
Rack middleware / before filter

Apache rewrite is certainly a good option, but if you are using a production infrastructure you do not have control over, like Heroku, this would not be an option. Also, it can be expensive to test, time wise, as each time you change the rule you have to restart, not a big problem but one to be aware of.

Using Analytics filters is also a very good option, but to be honest, I personally could not figure out how the filter system worked. Also, SEO wise, its better to redirect then to just fix the stats reporting, as you don’t want Google marking your pages down just because a trailing slash version exists.

That brings me to the third option, middleware and/or before filters.

If you are familiar with rails you would instantly think ‘just shove that in a before_filter in the application controller’, kind of how I did, but be warned, having a request go through the stack just to be redirected is slow. Also, if you are just creating more than a couple of checks you may find the separation of concerns gets a bit muddled.

Instead, I recommend rack-rewrite.

Rack-Rewrite is built to hook into the middleware config like so:

config.middleware.insert_before("Rack::Lock", "Rack::Rewrite") do
  r301 ....
end

Then you can setup your redirects:

r301 %r{^\/(.+)\/(\?.*)?$}, '/$1$2'

The redirect above finds everything before the trailing slash and puts it in $1, and all of the querystring and puts it in $2, then redirects to /$1$2.

We also added the following redirect:

r301 %r{^[\/]?(.*)\/index.html(\?.*)?$}, '/$1$2'

Very similar to the one above, this removed the /index.html and did a 301 (moved permanently).

The first redirect is useful for anyone, the second redirect is a good reminder for other sites that go through a rebuild or url structure change, Google might catch up over time, but old blog posts / reviews / referral links etc. won’t.

If you have any good rewrite rules you would like to share, please post them blog, or a link to a gist with your rewrite rules in them.

Happy easter,

Josh