Late last year, I started thinking about how we could better manage the distribution of Docker images from a private repository to Honeybadger customers who were interested in self-hosting. I looked at deploying our own instance of Harbor, an open source container registry with authentication, but then it started to feel like a capital-p project. I had plenty of other things to do, so I put it on the back burner. But as I thought more about it, I realized that while it felt like a less-than-exciting work project, it could be a fun side project if I turned it into a product. Because of course other people will have this problem and go look for a solution, right?

Breakwater, an app that helps software creators license their Docker images, is that product. You set up your products, associate them with your private repository, and create customer licenses for them. Customers can then use those licenses to pull your images the same way they would from Docker Hub or any other registry. If you want a way to sell your app as a self-hosted product, and you can package it as a Docker image, Breakwater is for you. In this post I'm going to walk you through how I built it with AI, and share some observations on how AI is changing the game for SaaS creators.

From idea to plan

Over the Christmas break I sat down with ChatGPT to hash out what the product would do and what features would be interesting. I find myself going to ChatGPT first when I have an idea to flesh out, or a question about how to approach a problem, etc. I think that's primarly because ChatGPT was the first AI chatbot I started using, but it helps that its icon lives in my menu bar. I'm sure Claude would be just fine for this, too.

But anyway, I planned to build a front-end for Harbor, and layer on concepts like Customers and Licenses — the things you need when you're distributing software commercially rather than hosting public images that are freely available to pull. Here's my initial prompt:

I'm interested in having a docker registry for customers where they use a web app to get credentials that are then used when pulling images. The goal is to sell licensed software that is delivered as docker containers, and once a customer purchases a license, they will have access to pull the containers. What technologies do you recommend I use for this?

I didn't say anything about Harbor, as I wanted to leave ChatGPT open to suggest anything. Naturally, Harbor got priority placement in its recommendations. Here's the initial response:

Short version: Use a standard OCI registry with token-based auth in front of it, and have your web app mint per-customer pull tokens/robot credentials based on license state. Don’t reinvent the registry itself; glue existing pieces together.

After presenting that and an overall architecture design, it presented three options:

1. Use Harbor as the registry and implement licensing in the web app, using the Harbor API to manage robot accounts and projects.
2. Use a standard OCI registry like distribution and implement token-based authentication using a custom web app.
3. Use a managed registry service like Amazon ECR, Google Artifact Registry, or Azure Container Registry.

I decided to go with option 1, since it seemed like the fastest path to a working solution. I also liked the idea of being able to use its features for pulling images from upstream sources (my future customers' own registries) and enforcing quotas. When I selected option 1, ChatGPT then generated a high-level implementation plan which seemed reasonable.

Then, knowing that I love to dive head-first into building stuff without doing much research, I asked about what solutions already existed for this problem:

Before we get too far into this, is there a SaaS that already offers this kind of functionality?

It gave me info about Quay.io and JFrog Artifactory as well as providing suggestions for making GHCR or AWS ECR do what I wanted. None of those solutions were exactly what I was looking for, so I decided to plow ahead:

I think I want to build my own thing rather than relying on a SaaS. Can you help me come up with a project plan? I'm thinking I'd build a Rails app for this.

It gave me a reasonable-looking plan, so I decided to go with it, and I asked it to give me the plan as a markdown download so I could save it to my new Rails project and use it with Claude Code.

Brainstorming the brand

Once I had settled on what I wanted to build, I decided to start a new chat session to brainstorm a name. It gave me names emphasizing licensing / entitlement, names themed around harbors / registries / containers, names focused on keys / access / authentication, names centered on delivery of software artifacts, etc., ending with a top-10 list of recommended names. After a bit of back-and-forth, and my suggestion to focus on "terms related to defending a harbor", Breakwater was the top choice. As we wrapped up the name selection process, ChatGPT suggested creating a brand board, so I had it do that.

It provided three options, and each of them had a vibe (Strong, crisp, premium, enterprise SaaS. Feels like Cloudflare, HashiCorp, or Stripe.), color palette, typography, logo/mark ideas, iconography, and a UI direction. After I picked the option "Modern Industrial Security", it provided a detailed brand board with all the elements mentioned above, and then suggested building a landing page, logo concepts (in SVG format), etc., and I asked it create the landing page using Tailwind. The current home page is still largely what ChatGPT created, though I made some changes while workshopping with Claude as the project progressed.

I wrapped up that chat by asking for a final deliverable before getting to work:

Can you give me the brand board, visual design instructions, etc., in a markdown file I can download to be used to guide an AI while building HTML and CSS for my Rails app?

Claude gets to work

Once I had the brand and vision in place, I switched to Claude for the actual development work. Claude quickly built me a Rails 8 app using Tailwind, following the color scheme in the brand board doc that I downloaded from the conversation with ChatGPT. It took just a few hours to get the models in place, get Harbor running, get the Rails app working with the Harbor API, and have authenticated pushes and pulls working. All along the way, I had Claude updating the project plan document to keep track of progress and to provide context for new work sessions. Every phase (Harbor infrastructure, Rails app skeleton, Harbor integration, etc.) was a new Claude session, and at the end of each session I would have Claude update the project plan document and generate a new document summarizing the decisions made and the work done in that phase.

As I played with the app I realized that using Harbor was adding friction — my models weren't lining up well with Harbor's structure — so I revisited my first chat with ChatGPT, picking up where we left off with this:

I've been going down this path with Harbor, and I'm not sure it's working for me... I'm feeling a bit of friction between my app and Harbor. I'm not sure I'm ready to abandon that approach yet, but I would like to explore option B: providing an auth token layer for plain registry. Can you dive into that option a bit more?

And of course ChatGPT was happy to help. After a bit of back and forth and clarifying questions on both sides, I was ready to point Claude in the new direction, and I asked for an updated plan from ChatGPT:

I'd be interested in seeing a detailed implementation plan that includes the info you mentioned, with the plan written as downloadable markdown so I can use it to drive an AI in my project to build the task list.

With the new plan in hand, and with an hour or two of work time per day, Claude and I made good progress. Claude was much faster than I would have been at building the auth token support in my Rails app that the registry needed for handling Docker authentication. After about ten hours of work, I had a working Rails app that was deployed and ready for alpha testing.

When plans change

Eventually I realized that my registry + authentication setup wouldn't support a key feature I wanted: being able to grant or deny access to images based on tag versions. The registry didn't pass that information along to the Rails app when it received a pull request, so I couldn't implement version-based licensing. I was pretty bummed by this, and, packing it up for the night, I explained my disappointment to my wife. I knew what needed to be done — build an authenticating proxy in front of the registry — but it felt like a lot of work, and I wasn't sure it would be worth it. I considered dropping the feature, but then my wife said:

Why don't you just ask Claude to help you build it?

Well, duh.

So the next morning I had Claude help me build an authenticating proxy in Go that would sit in front of the registry and pass the information to the Rails app that I needed. Again, it went much, much faster with Claude than if I had tried to do it on my own. The initial code session for that took less than an hour, and within three hours or so, over the course of a few days, it was done.

It was around this time that I decided to start having Codex review Claude's work, and that has been awesome. I'll ask Codex to do a review of the current changes, then copy paste Codex's findings into my Claude Code session, and have it implement the fixes. Then when I push a PR to GitHub, Copilot will review the changes and provide feedback, which I'll then have Claude address. My solo project has turned into a team project, and it's been a blast.

Why this works so well

Claude is amazing when it's helping you with work that you already know how to do. Being an expert web developer and operator (19 years of running SaaS apps and counting!) and using AI to build a new web app is crazy fast for a few reasons:

1. Claude has been trained on a lot of web app code. Granted, not all of that code is awesome, so Claude doesn't always generate the exact code I would write, but it's usually pretty good. Having Claude generate code much faster than I could write it, even if it's not perfect, is a win.

2. I know what features I need. "Claude, build vendor onboarding. Claude, build an audit log." I can describe the features at a high level because I know the domain well. And when I don't know exactly what I want, Claude can often suggest relevant features or improvements. I can literally type, "what's this app missing" and Claude will offer suggestions that are usually very good. Or I can type, "I'm thinking of building X, please help me create a plan to build it and ask me questions to flesh out the details", which is surprisingly helpful in finding blind spots and ironing out implementation details.

3. I know good output when I see it. I can validate that Claude is not just producing something, but is producing the right thing. When Claude's implementation doesn't quite fit my needs, a simple prompt gets it back on track. AI in the hands of a novice can be dangerous... if I asked ChatGPT a question about a medical condition or a legal issue, I'd have no idea whether it was accurate or not. But when I ask Claude to build a feature, I know what it should do and how it should work. I know how to evaluate the generated code for potential security and performance issues.

4. I don't have to focus 100% on the work. Getting in the zone while writing code is awesome. I love experiencing flow while working on a project. But having work get done while I'm doing something else is also awesome. I can spin up a task with Claude, then switch to doing something else, and come back to find a working feature. I can prompt Claude Code Web to do something from my phone when I'm out and about, and then when get back to my computer, I can review and ship what it built for me.

5. It's easier to get started on a task. Activation energy is a thing. Starting with blank slate, whether it be prose or code, can feel daunting, but starting with a prompt can be low-effort. For example, I'm usually not excited to work on frontend code. Typing something like "I want the product page to be updated when a repository is linked to it" can get me 80% or 90% of the way there, making it feel less daunting to get started.

I'm sure there are more benefits I could list, but you get the point. I'm so much faster at delivering features with AI than I am without it. I've lost track of how many times I've sat down to build out the next phase of the plan and it's done within ten minutes. Build an admin UI? Done. Add a multi-step vendor onboarding workflow? Done. Create a documentation portal? API? Self-hosted analytics? Done, done, and done.

I'm not spending time in the editor, typing code. Claude is typing the code. I'm directing the work, thinking about the bigger picture, and shipping faster than ever before. It's amazing.

Looking ahead

B2B SaaS in 2026 is going to be wild thanks to Claude and other AI coding assistants. It's scary, exciting, and mind-blowing all at the same time. The barrier to building new products has dropped dramatically. Solo founders and small teams can move at a pace that wasn't possible before, and be more ambitious in what they can build. The bottleneck is no longer "can I build this?" Instead, it's "should I build this?" and "do people want this?" and "how will I get this in front of people?"

I'm excited to see where Breakwater goes, and I'm even more excited about what this new era of AI-assisted development means for indie hackers and bootstrappers everywhere.

In that time I picked up motorcycle riding as a hobby, I started a podcast with my business partners, and I've continued to plug away at Honeybadger. Honeybadger has been my day job for more than a decade, and it's still fun to show up every day and solve problems. I continue to be focused on building the product and scaling it — but now with the help of three more developers. It's so nice to be able to take a little more time off than I could in years past, and I'm not on-call 24/7 these days. 😅

In short, life is good!

Keeping track of the S3 objects we create

As our processing pipeline receives and processes error notifications, we store the payload from each notification in an S3 object. We also create objects in a separate S3 bucket that contain a batched list of the resulting S3 keys and the expiration time for each of those keys. These objects are just JSON arrays that look like this:

[
  {
    "key": "pu3We2Ie/ea40b606-1b48-40cb-942f-a046755c7a0f",
    "expire_at": "2017-03-03T23:59:58Z"
  },
  {
    "key": "Ieb0ieVu/fe3c6b8a-76d7-48d8-ab71-ea8a4f3bce08",
    "expire_at": "2017-07-31T23:59:58Z"
  }
]

The S3 bucket that is used to store these lists of keys sends a message to an SNS topic for each PUT request. We have a Lambda function that is subscribed to this topic to process each object as it is created, configured in our serverless.yml as the send-to-dynamo function:

functions:
  send-to-dynamo:
    handler: bin/send-to-dynamo
    events:
      - sns: "arn:aws:sns:us-east-1:*:expirations-${opt:stage, self:provider.stage}"

Tip: If you start your Serverless project with serverless create --template aws-go, you will get a project layout that puts the code for each function in a main.go file with its own subdirectory and a Makefile to help with compiling the code before deploying. Handy!

Ok, back to our configuration... Since this function needs to read the contents of the S3 objects referenced by the SNS notification, we have these permissions defined in serverless.yml:

provider:
  name: aws
  runtime: go1.x
  environment:
    STAGE: "${opt:stage, self:provider.stage}"
  iamRoleStatements:
   - Effect: "Allow"
     Action:
       - "s3:GetObject"
     Resource: "arn:aws:s3:::expirations-bucket-${opt:stage, self:provider.stage}/*"

That STAGE environment variable is used in the Go code to know which DynamoDB table to use, since we create a table per serverless environment that we deploy. Here's how we define that table and give the Lambda function permission to write to it in our config:

provider:
  # ...
  iamRoleStatements:
    # ...
    - Effect: "Allow"
      Action:
        - "dynamodb:PutItem"
      Resource:
        Fn::GetAtt:
          - expirationsTable
          - Arn

resources:
  Resources:
    expirationsTable:
      Type: AWS::DynamoDB::Table
      Properties:
        TableName: "expirations-${opt:stage, self:provider.stage}"
        AttributeDefinitions:
          - AttributeName: ID
            AttributeType: S
          - AttributeName: ExpireAt
            AttributeType: N
        KeySchema:
          - AttributeName: ID
            KeyType: HASH
          - AttributeName: ExpireAt
            KeyType: RANGE
        StreamSpecification:
          StreamViewType: OLD_IMAGE
        TimeToLiveSpecification:
          AttributeName: ExpireAt
          Enabled: true
        # ...

You'll notice that the DynamoDB table has been configured with a TTL field called ExpireAt, and that it will emit a stream of events. Since we only care about the TTL events, we use OLD_IMAGE as the StreamViewType, since that will give us the contents of the fields in DynamoDB row when it is deleted.

Since we have a lot of keys arriving with the same expiration time, the Lambda function groups the list of keys by expiration time and creates one record per group in DynamoDB to reduce the write throughput. This results in one DyanmoDB record per expiration time (down to the second) which contains a list of S3 keys that are to be deleted at the expiration time.

{% gist 7cb3d82cdc8bde17299c2e77b7301d83 send-to-dynamo-main.go %}

The handler function receives the SNS event, looks for S3 records in the event data, then calls the getItems function to load each of those S3 objects and return lists of S3 keys to be deleted, grouped by expiration time. Each of those groups gets inserted as one DynamoDB record.

Using DynamoDB streams to know when to delete

Now that we have the triggers and code in place to store the keys of the objects that are to be deleted, we need to finish the job by watching for TTL events in the DynamoDB stream and delete the S3 objects. The purge-from-s3 function does this. Here's the configuration from serverless.yml:

provider:
  # ...
  iamRoleStatements:
   - Effect: "Allow"
     Action:
       - "s3:DeleteObject"
     Resource: "arn:aws:s3:::data-bucket-${opt:stage, self:provider.stage}/*"

functions:
  # ...
  purge-from-s3:
    handler: bin/purge-from-s3
    events:
      - stream:
          type: dynamodb
          arn:
            Fn::GetAtt:
              - expirationsTable
              - StreamArn

We have granted the Lambda function the permission to delete objects from the bucket that stores the error payloads, and we have configured that function to be triggered by the events in the stream from our DynamoDB table.

This function is a simpler than the first one. When it receives a REMOVE event from DynamoDB, signifying that a record has been deleted due to the TTL, it iterates over the list of keys stored in the record, deleting each one from the bucket.

{% gist 7cb3d82cdc8bde17299c2e77b7301d83 purge-from-s3-main.go %}

Unfortunately, this Lambda function will get invoked for every event in the DynamoDB stream, which includes all the inserts that we did in the send-to-dynamo function. We don't care about those records, so we just ignore them in this function. I have filed a feature request with Amazon to add the ability to filter the types of stream events that trigger a function, and you're more than welcome to do the same. :)

I hope you have found this post to be a useful example of how to work with S3 and DynamoDB streams using the Serverless Framework, Lambda, and Go!

In short, 2017 was a great year! :) I moved all of our servers from a colocation facility to AWS in January, which helped me sleep a lot better at night. Over the course of the year I continued to improve our infrastructure, and we now have a very reliable and self-healing system. Nearly everything we do (application, search, and database servers) is self-managed, so it's been fun to level-up my distributed system skills. In December I put my AWS skills (literally) to the test by getting my first AWS certification: AWS Certified Developer - Associate exam.

I also spent some time working on developing my marketing skills by taking The Marketing Seminar by Seth Godin. The seminar was excellent, and I'm very glad I spent the time on it. Having spent a few years as a freelancer, I had to get good at one-to-one sales, but trying to market a SaaS business to a world of customers is a different kettle of fish, and I picked up lots of helpful info from Seth's seminar. My favorite part of his philosophy is that if you belive you are bringing a great product or service to the world (and why would you be doing it if you didn't believe that?), then you owe it to the world to be a champion for your product in a way that will attract people who will be best served by it. Of course I'm convinced that Honeybadger is a great addition to the world, so Seth's approach resonates with me. I'm hoping that I'll be able to apply those learnings and improve my marketing skills in 2018.

During the summer I had two small freelancing projects fall in my lap out of the blue. I hadn't done any freelancing in several years, so it was fun to do some work on a side project and shift the mental gears a little bit from my day-to-day work at Honeybadger. One of the projects was TVTattle, which was a blast to work on. It's a pretty simple content-management Rails app, but it was fun to play with caching plus a CDN to make it super fast.

So that's my year -- a lot of ops work with sprinklings of dev work along the way. I'm still learning all the time, and business is great, so what more could I want? :)

One cause of the lack of my writing is how busy I have been running my error tracking service. It has been a lot of fun, but it has also been a lot of work. The good news is that the business continues to grow, and we just passed the five-year mark. My co-founders and I are definitely living the bootstrapper's dream, doing work that we love, and making our customers happy.

Work will fill the time allotted to it, though, so I'm going to try and carve out more time for writing. We'll see how it goes. :)

What happened to us this morning is that some of the shards became leaderless when one of the servers ran out of disk space and started throwing errors. In other words, we kept seeing this error in the logs: No registered leader was found. As a result, the two remaining servers refused to update the index, which brought a halt to search-related operations. Since I'm relatively new to Solr, I had to bang my head against the wall for a bit before I stumbled upon the solution.

Simply bringing down one of the two remaining good servers didn't solve the problem. The last remaining server refused to become the leader for four of the shards. To fix this, I had to unload each of the stubborn shards and load them again. This was accomplished easily enough via the admin UI, and, once completed, our search functionality was restored.

Once that was done I brought the other good server back up, and it quickly caught up the one server that was now the leader for all the shards. Easy-peasy. Sadly, bringing the last of the servers up -- the culprit with the full disk -- took a bit more work. Since its data directory was about twice the size of the directory on the other two servers, despite all three supposedly having the same documents, I decided to just blow away all the data on the third server and replace it with a copy of the snapshots from the leader. The process was basically this:

1. Take a fresh snapshot of each shard from the leader
2. Copy the eight snapshots from the leader to the damaged server
3. Move the index data in place, renaming the shards
4. Unload and load the cores on the damaged server

Here's a ruby script for step 1 and a shell script for the other steps.

With that, the damaged server quickly got each of the shards back in sync (since I had just taken snapshots on the leader), and everything was back to normal.

My day job has always been building web apps, so my first side projects were also web apps: first, a community site, and later, a SaaS app for managing test plan execution for software testers. Those were fun, but never amounted to much.

The first side project I did with the goal of making money was a self-published ebook about building e-commerce sites with Rails. This was in 2006, when Rails was young, and that $12 ebook sold pretty well.

In 2007 I started freelancing full time, and I decided that I needed a product with recurring revenue to help even out my cash flow, so I started on Catch the Best, a SaaS app that scratched my own itch. I launched that in October of 2007 (working on in it part-time while working on client projects), and it got some paying customers from day one. The revenue from that app has never been large on a MRR basis, but it has been consistent, so I'm pretty happy having that as a cash machine.

In 2008 I was building a SaaS billing system in Rails for the third time. The first time was for Catch the Best, and second two times were for clients that had engaged me to build SaaS apps for them.
It occurred to me that other developers might be interested in buying what I had built so that they could save themselves the time of building their own. So I cleaned up the code I had written and launched RailsKits to sell that billing code to other Rails developers. I priced it starting at $250, and it was a hit. It effectively replaced my freelance income for a while, and while it doesn't make as much as it used to (since other options for implementing billing have become available), it still is a consisitent revenue stream for me.

After RailsKits, I knew I wanted to do another SaaS project, and in 2012 I found the right one: Honeybadger -- an application health monitoring service for Ruby developers. It has been an incredibly fun project with awesome co-founders. Since its launch in the fall of 2012 it has grown consistently, allowing me to cut back and eventually eliminate my freelancing business.

I didn't set out with a plan to start with an ebook, then move to a larger product, then move to a recurring revenue product, but after having considered Rob's Stairstep Approach and having reflected on the past decade of my own experience, I can certainly recommend going that route. It's not the only way to go, but it does give you a variety of opportunities to learn how to find customers and sell something to them, and it can be a whole lot of fun.

Help Scout allows you to plug "apps" into their UI, and you can build your own apps to populate the sidebar when looking at a help ticket. All you have to do is provide a URL that Help Scout can hit which returns a blob of HTML to be rendered on the page. Your app receives a signed POST request where the payload is some information about the support ticket you are viewing, which includes the email address of the person who created the ticket. Here's a Rails controller that receives the request, verifies the signature, and returns some HTML for the user found by email address:

require 'base64'
require 'hmac-sha1'

class HelpscoutController < ApplicationController
  skip_before_filter :verify_authenticity_token
  before_filter :verify_signature

  def user
    payload = JSON.parse(request.raw_post)
    if payload['customer'] && payload['customer']['email'] && @user = User.where(email: payload['customer']['email']).first
      render json: { html: render_to_string(action: :user, layout: false) }
    else
      render json: { html: "User not found" }
    end

  end

  protected

    def verify_signature
      bail and return false unless (sig = request.headers['X-Helpscout-Signature']).present?

      (hmac = HMAC::SHA1.new("secret-that-you-enter-in-helpscout's-ui")).update(request.raw_post)

      bail and return false unless sig.strip == Base64.encode64(hmac.digest).strip
    end

    def bail
      render json: { html: "Bad signature" }, status: 403
    end
end

After fetching the user record, it returns a blob of HTML via a HAML view:

%ul
  %li Created on #{l(@user.created_at.to_date, format: :long)}

Then you're done! Now when you view a ticket in Help Scout you'll see info from your database about that user in the sidebar.

add_column :invoices, :paid_on, :date, default: 'now()'

That will look like it works -- you create a record, it gets populated with today's date, and all is good. However, if you look at your schema, you will notice that new field has a default of today's date instead of now(). Oops. :)

You might try to create the column with the recommendation from the Postgres documentation:

add_column :invoices, :paid_on, :date, default: 'CURRENT_DATE'

But that fails because Rails tries to quote that 'CURRENT_DATE' for you before it goes to Postgres, which blows up. Now what?

Here's how to do what you want:

add_column :invoices, :paid_on, :date, default: { expr: "('now'::text)::date" }

This avoids the quoting problem (by using expr) and avoids the always-insert-migration-date's-date problem (by using the default function of ('now'::text)::date, which is effectively the same as CURRENT_DATE.

And now when you insert a record without specifying a value for that field, you get the date of the insertion, rather than the date of the field being created. :)

In one case I wanted to provide search on a list of items (the index action) to admins so they could search through all items in the database, but users should be able to only search on their own items. I'm using Searchlight (highly recommended) for search, which returns results as an ActiveRecord::Relation, so it's easily chainable via CanCan, like so:

class InvoicesController < ApplicationController
  def index
    @search = InvoiceSearch.new(params[:search])
    @invoices = @search.results.accessible_by(current_ability, :index)
  end
end

Searchlight is also smart enough to return all results if there no search params provided, so this also works as a typical index action that lists all items the user can see. If you're curious about the @search instance variable, that is used in the search form in the index view.

So, if you need search with access control, use Searchlight and CanCan... they are a great combo!