Originally titled: Encoding Issues When Upgrading to Rails 3.2 - after doing some more detective work - the problems might actually be something else..
Before I start, I have to mention that 80% of what I discovered during this process comes from the patient mentoring of my colleague @chuckdbacon.
And so we attempted to upgrade our Rails 3.1.3 app to 3.2.2 few days ago. At first everything seems to be smooth sailing with tests passing and all that.
Unfortunatelly that’s not how the story ends, the day after deploying the upgrade, we got a flurry of exception notifications from the app.
LDAP dying
The issue comes from the our LDAP importing code.
Contact.import_from_ldap
Encoding::UndefinedConversionError: "\x82" from ASCII-8BIT to UTF-8
At first, I was blaming Rails upgrade for this - but the error is probably caused from the net-ldap gem that got upgraded when I did bundle update. So the proper “fix” is to locked the gem version on Gemfile.
Encoding issue that’s not really the issue
UPDATE: Ok, so everything from below of this paragraph is actually WRONG.
We suspected firstly that the Rails 3.2 upgrade introduced an encoding issue in our app. However we finally realized that the error that we’re experiencing (see below)
- happened after we rolled back the deployment (due to the first issue above).
So it’s the case of a bad deployment rollback rather than Rails issue.
Our theory so far is, the cache was corrupted somehow during the rollback. Restarting memcached fixed the issue.
Again, it’s unfortunate that I’ve spent hours working out “this encoding issue” as explained below. Feel free to see my detective work going after a false path below.
** WRONG START ** And then one of the controllers was returning this error:
A Encoding::CompatibilityError occurred in bos#index:
incompatible character encodings: UTF-8 and ASCII-8BIT
builder (3.0.0) lib/builder/xmlmarkup.rb:276:in `_text'
Obviously, the next step is to work out what has changed in Rails 3.2 with regards to encoding. Weird thing is, I can’t find anywhere in Rails release notes regarding encoding changes. And it doesn’t look like that this is a common issue that people have._
Other difficulty that I have is reproducing the problem locally for some reasons that’s hard to explain.
One of possible causes is the encoding of variables that set in the application.rb - this is quite interesting. Let the output explains itself:
[1] pry(main)> App::Application.config.pdfs_path.encoding
=> #<Encoding:US-ASCII>
[2] pry(main)> "just a string".encoding
=> #<Encoding:UTF-8>
[3] pry(main)>
As you can see above - the application config variable has US-ASCII encoding (which I don’t understand why) - while the normal string is UTF-8 encoded as expected as this is the default in Rails.
The closest thing that I can find is the blog post here: Incompatible character encoding I’ll try the suggested solution and will update this post with the outcome.
This is one of those problems when I just have absolutely no idea what’s going on. ** WRONG END **
So Rails is not at fault at all??
We just found out that the serialization works differently in Rails 3.2. And this has meant another deployment rollback for us. This is not the first time serialization issue bit us - we encountered similar issue when upgrading Rails 2.3 to Rails 3.0.
But that’s probably for another post.