Benjamin Curtis

Speculations on Web Development

Solr Recovery

| Comments

At Honeybadger this morning we had a failure of our SolrCloud cluster (of three servers). Each of the three servers has a replica of the eight shards of our notices collection. Theoretically this means that two of the three servers can go away and we can still serve search traffic. Sadly, reality doesn’t always match the theory.

What happened to us this morning is that some of the shards became leaderless when one of the servers ran out of disk space and started throwing errors. In other words, we kept seeing this error in the logs: No registered leader was found. As a result, the two remaining servers refused to update the index, which brought a halt to search-related operations. Since I’m relatively new to Solr, I had to bang my head against the wall for a bit before I stumbled upon the solution.

Simply bringing down one of the two remaining good servers didn’t solve the problem. The last remaining server refused to become the leader for four of the shards. To fix this, I had to unload each of the stubborn shards and load them again. This was accomplished easily enough via the admin UI, and, once completed, our search functionality was restored.

Once that was done I brought the other good server back up, and it quickly caught up the one server that was now the leader for all the shards. Easy-peasy. Sadly, bringing the last of the servers up — the culprit with the full disk — took a bit more work. Since its data directory was about twice the size of the directory on the other two servers, despite all three supposedly having the same documents, I decided to just blow away all the data on the third server and replace it with a copy of the snapshots from the leader. The process was basically this:

  1. Take a fresh snapshot of each shard from the leader
  2. Copy the eight snapshots from the leader to the damaged server
  3. Move the index data in place, renaming the shards
  4. Unload and load the cores on the damaged server

Here’s a ruby script for step 1 and a shell script for the other steps.

With that, the damaged server quickly got each of the shards back in sync (since I had just taken snapshots on the leader), and everything was back to normal.

Steppin’ Up

| Comments

Rob Walling wrote a great post yesterday about building up your bootstrapped business over time by taking on smaller projects before diving into big ones. His post reminded me of Amy Hoy’s Stacking the Bricks philosophy, and I think that taking the approach of learning to walk before learning to run makes sense. Rob’s post made me reflect on my experience building products that have gone from producing no income, to putting some change in my pocket, to providing a nice income for my family, and I thought it would be fun to share.

My day job has always been building web apps, so my first side projects were also web apps: first, a community site, and later, a SaaS app for managing test plan execution for software testers. Those were fun, but never amounted to much.

The first side project I did with the goal of making money was a self-published ebook about building e-commerce sites with Rails. This was in 2006, when Rails was young, and that $12 ebook sold pretty well.

In 2007 I started freelancing full time, and I decided that I needed a product with recurring revenue to help even out my cash flow, so I started on Catch the Best, a SaaS app that scratched my own itch. I launched that in October of 2007 (working on in it part-time while working on client projects), and it got some paying customers from day one. The revenue from that app has never been large on a MRR basis, but it has been consistent, so I’m pretty happy having that as a cash machine.

In 2008 I was building a SaaS billing system in Rails for the third time. The first time was for Catch the Best, and second two times were for clients that had engaged me to build SaaS apps for them.
It occurred to me that other developers might be interested in buying what I had built so that they could save themselves the time of building their own. So I cleaned up the code I had written and launched RailsKits to sell that billing code to other Rails developers. I priced it starting at $250, and it was a hit. It effectively replaced my freelance income for a while, and while it doesn’t make as much as it used to (since other options for implementing billing have become available), it still is a consisitent revenue stream for me.

After RailsKits, I knew I wanted to do another SaaS project, and in 2012 I found the right one: Honeybadger — an application health monitoring service for Ruby developers. It has been an incredibly fun project with awesome co-founders. Since its launch in the fall of 2012 it has grown consistently, allowing me to cut back and eventually eliminate my freelancing business.

I didn’t set out with a plan to start with an ebook, then move to a larger product, then move to a recurring revenue product, but after having considered Rob’s Stairstep Approach and having reflected on the past decade of my own experience, I can certainly recommend going that route. It’s not the only way to go, but it does give you a variety of opportunities to learn how to find customers and sell something to them, and it can be a whole lot of fun.

Inject Your App Data Into Help Scout

| Comments

At Honeybadger we use Help Scout to manage our customer support, and that has worked out well for us. One thing I’ve wanted for quite a while is more integration between Help Scout, our internal dashboard, and the Stripe dashboard. After taking a mini-vacation to attend MicroConf this week, I decided it was time to make my dreams come true. :)

Help Scout allows you to plug “apps” into their UI, and you can build your own apps to populate the sidebar when looking at a help ticket. All you have to do is provide a URL that Help Scout can hit which returns a blob of HTML to be rendered on the page. Your app receives a signed POST request where the payload is some information about the support ticket you are viewing, which includes the email address of the person who created the ticket. Here’s a Rails controller that receives the request, verifies the signature, and returns some HTML for the user found by email address:

require 'base64'
require 'hmac-sha1'

class HelpscoutController < ApplicationController
  skip_before_filter :verify_authenticity_token
  before_filter :verify_signature

  def user
    payload = JSON.parse(request.raw_post)
    if payload['customer'] && payload['customer']['email'] && @user = User.where(email: payload['customer']['email']).first
      render json: { html: render_to_string(action: :user, layout: false) }
      render json: { html: "User not found" }



    def verify_signature
      bail and return false unless (sig = request.headers['X-Helpscout-Signature']).present?

      (hmac ="secret-that-you-enter-in-helpscout's-ui")).update(request.raw_post)

      bail and return false unless sig.strip == Base64.encode64(hmac.digest).strip

    def bail
      render json: { html: "Bad signature" }, status: 403

After fetching the user record, it returns a blob of HTML via a HAML view:

  %li Created on #{l(@user.created_at.to_date, format: :long)}

Then you’re done! Now when you view a ticket in Help Scout you’ll see info from your database about that user in the sidebar.

Default a Postgres Column to the Current Date in a Rails Migration

| Comments

If you want to have a Postgres column (aside from created_at) that you want to be populated with the current date if no other date is specified, you may be tempted to create a migration like this:

add_column :invoices, :paid_on, :date, default: 'now()'

That will look like it works — you create a record, it gets populated with today’s date, and all is good. However, if you look at your schema, you will notice that new field has a default of today’s date instead of now(). Oops. :)

You might try to create the column with the recommendation from the Postgres documentation:

add_column :invoices, :paid_on, :date, default: 'CURRENT_DATE'

But that fails because Rails tries to quote that ‘CURRENT_DATE’ for you before it goes to Postgres, which blows up. Now what?

Here’s how to do what you want:

add_column :invoices, :paid_on, :date, default: { expr: "('now'::text)::date" }

This avoids the quoting problem (by using expr) and avoids the always-insert-migration-date’s-date problem (by using the default function of (‘now’::text)::date, which is effectively the same as CURRENT_DATE.

And now when you insert a record without specifying a value for that field, you get the date of the insertion, rather than the date of the field being created. :)

Searchlight and CanCan

| Comments

I’m currently working on a client project where site adminstrators use the same UI that site users do, so there are permissions checks in the views and controllers to ensure the current user has the right to do or see certain things. CanCan provides the access control, which takes care of most of the issues with a simple can? check or load_and_authorize_resource.

In one case I wanted to provide search on a list of items (the index action) to admins so they could search through all items in the database, but users should be able to only search on their own items. I’m using Searchlight (highly recommended) for search, which returns results as an ActiveRecord::Relation, so it’s easily chainable via CanCan, like so:

class InvoicesController < ApplicationController
  def index
    @search =[:search])
    @invoices = @search.results.accessible_by(current_ability, :index)

Searchlight is also smart enough to return all results if there no search params provided, so this also works as a typical index action that lists all items the user can see. If you’re curious about the @search instance variable, that is used in the search form in the index view.

So, if you need search with access control, use Searchlight and CanCan… they are a great combo!