Benjamin Curtis

Speculations on Web Development

2017 in Review

| Comments

I can’t recall having done a year-in-review type of blog post before, but when Patrick suggested it recently, it seemed like a great idea, so I thought I’d give it a shot.

In short, 2017 was a great year! :) I moved all of our servers from a colocation facility to AWS in January, which helped me sleep a lot better at night. Over the course of the year I continued to improve our infrastructure, and we now have a very reliable and self-healing system. Nearly everything we do (application, search, and database servers) is self-managed, so it’s been fun to level-up my distributed system skills. In December I put my AWS skills (literally) to the test by getting my first AWS certification: AWS Certified Developer – Associate exam.

I also spent some time working on developing my marketing skills by taking The Marketing Seminar by Seth Godin. The seminar was excellent, and I’m very glad I spent the time on it. Having spent a few years as a freelancer, I had to get good at one-to-one sales, but trying to market a SaaS business to a world of customers is a different kettle of fish, and I picked up lots of helpful info from Seth’s seminar. My favorite part of his philosophy is that if you belive you are bringing a great product or service to the world (and why would you be doing it if you didn’t believe that?), then you owe it to the world to be a champion for your product in a way that will attract people who will be best served by it. Of course I’m convinced that Honeybadger is a great addition to the world, so Seth’s approach resonates with me. I’m hoping that I’ll be able to apply those learnings and improve my marketing skills in 2018.

During the summer I had two small freelancing projects fall in my lap out of the blue. I hadn’t done any freelancing in several years, so it was fun to do some work on a side project and shift the mental gears a little bit from my day-to-day work at Honeybadger. One of the projects was TVTattle, which was a blast to work on. It’s a pretty simple content-management Rails app, but it was fun to play with caching plus a CDN to make it super fast.

So that’s my year — a lot of ops work with sprinklings of dev work along the way. I’m still learning all the time, and business is great, so what more could I want? :)

Writing Again

| Comments

A while back I changed the stack I used for publishing this blog with the hope that I would write more because it would be easier to publish. Looking back at how little I’ve written since I’ve made that change, I can see that that didn’t work out so well. :) I’m not going to change my stack again (just yet), but I am going to try and write a bit more. Hopefully it won’t be another year before my next post.

One cause of the lack of my writing is how busy I have been running my error tracking service. It has been a lot of fun, but it has also been a lot of work. The good news is that the business continues to grow, and we just passed the five-year mark. My co-founders and I are definitely living the bootstrapper’s dream, doing work that we love, and making our customers happy.

Work will fill the time allotted to it, though, so I’m going to try and carve out more time for writing. We’ll see how it goes. :)

Solr Recovery

| Comments

At Honeybadger this morning we had a failure of our SolrCloud cluster (of three servers). Each of the three servers has a replica of the eight shards of our notices collection. Theoretically this means that two of the three servers can go away and we can still serve search traffic. Sadly, reality doesn’t always match the theory.

What happened to us this morning is that some of the shards became leaderless when one of the servers ran out of disk space and started throwing errors. In other words, we kept seeing this error in the logs: No registered leader was found. As a result, the two remaining servers refused to update the index, which brought a halt to search-related operations. Since I’m relatively new to Solr, I had to bang my head against the wall for a bit before I stumbled upon the solution.

Simply bringing down one of the two remaining good servers didn’t solve the problem. The last remaining server refused to become the leader for four of the shards. To fix this, I had to unload each of the stubborn shards and load them again. This was accomplished easily enough via the admin UI, and, once completed, our search functionality was restored.

Once that was done I brought the other good server back up, and it quickly caught up the one server that was now the leader for all the shards. Easy-peasy. Sadly, bringing the last of the servers up — the culprit with the full disk — took a bit more work. Since its data directory was about twice the size of the directory on the other two servers, despite all three supposedly having the same documents, I decided to just blow away all the data on the third server and replace it with a copy of the snapshots from the leader. The process was basically this:

  1. Take a fresh snapshot of each shard from the leader
  2. Copy the eight snapshots from the leader to the damaged server
  3. Move the index data in place, renaming the shards
  4. Unload and load the cores on the damaged server

Here’s a ruby script for step 1 and a shell script for the other steps.

With that, the damaged server quickly got each of the shards back in sync (since I had just taken snapshots on the leader), and everything was back to normal.

Steppin’ Up

| Comments

Rob Walling wrote a great post yesterday about building up your bootstrapped business over time by taking on smaller projects before diving into big ones. His post reminded me of Amy Hoy’s Stacking the Bricks philosophy, and I think that taking the approach of learning to walk before learning to run makes sense. Rob’s post made me reflect on my experience building products that have gone from producing no income, to putting some change in my pocket, to providing a nice income for my family, and I thought it would be fun to share.

My day job has always been building web apps, so my first side projects were also web apps: first, a community site, and later, a SaaS app for managing test plan execution for software testers. Those were fun, but never amounted to much.

The first side project I did with the goal of making money was a self-published ebook about building e-commerce sites with Rails. This was in 2006, when Rails was young, and that $12 ebook sold pretty well.

In 2007 I started freelancing full time, and I decided that I needed a product with recurring revenue to help even out my cash flow, so I started on Catch the Best, a SaaS app that scratched my own itch. I launched that in October of 2007 (working on in it part-time while working on client projects), and it got some paying customers from day one. The revenue from that app has never been large on a MRR basis, but it has been consistent, so I’m pretty happy having that as a cash machine.

In 2008 I was building a SaaS billing system in Rails for the third time. The first time was for Catch the Best, and second two times were for clients that had engaged me to build SaaS apps for them.
It occurred to me that other developers might be interested in buying what I had built so that they could save themselves the time of building their own. So I cleaned up the code I had written and launched RailsKits to sell that billing code to other Rails developers. I priced it starting at $250, and it was a hit. It effectively replaced my freelance income for a while, and while it doesn’t make as much as it used to (since other options for implementing billing have become available), it still is a consisitent revenue stream for me.

After RailsKits, I knew I wanted to do another SaaS project, and in 2012 I found the right one: Honeybadger — an application health monitoring service for Ruby developers. It has been an incredibly fun project with awesome co-founders. Since its launch in the fall of 2012 it has grown consistently, allowing me to cut back and eventually eliminate my freelancing business.

I didn’t set out with a plan to start with an ebook, then move to a larger product, then move to a recurring revenue product, but after having considered Rob’s Stairstep Approach and having reflected on the past decade of my own experience, I can certainly recommend going that route. It’s not the only way to go, but it does give you a variety of opportunities to learn how to find customers and sell something to them, and it can be a whole lot of fun.

Inject Your App Data Into Help Scout

| Comments

At Honeybadger we use Help Scout to manage our customer support, and that has worked out well for us. One thing I’ve wanted for quite a while is more integration between Help Scout, our internal dashboard, and the Stripe dashboard. After taking a mini-vacation to attend MicroConf this week, I decided it was time to make my dreams come true. :)

Help Scout allows you to plug “apps” into their UI, and you can build your own apps to populate the sidebar when looking at a help ticket. All you have to do is provide a URL that Help Scout can hit which returns a blob of HTML to be rendered on the page. Your app receives a signed POST request where the payload is some information about the support ticket you are viewing, which includes the email address of the person who created the ticket. Here’s a Rails controller that receives the request, verifies the signature, and returns some HTML for the user found by email address:

require 'base64'
require 'hmac-sha1'

class HelpscoutController < ApplicationController
  skip_before_filter :verify_authenticity_token
  before_filter :verify_signature

  def user
    payload = JSON.parse(request.raw_post)
    if payload['customer'] && payload['customer']['email'] && @user = User.where(email: payload['customer']['email']).first
      render json: { html: render_to_string(action: :user, layout: false) }
      render json: { html: "User not found" }



    def verify_signature
      bail and return false unless (sig = request.headers['X-Helpscout-Signature']).present?

      (hmac ="secret-that-you-enter-in-helpscout's-ui")).update(request.raw_post)

      bail and return false unless sig.strip == Base64.encode64(hmac.digest).strip

    def bail
      render json: { html: "Bad signature" }, status: 403

After fetching the user record, it returns a blob of HTML via a HAML view:

  %li Created on #{l(@user.created_at.to_date, format: :long)}

Then you’re done! Now when you view a ticket in Help Scout you’ll see info from your database about that user in the sidebar.