← Blog
From 4-Hour Deploy Hell to 20-Minute Victory Laps

From 4-Hour Deploy Hell to 20-Minute Victory Laps

devopsdeploymentliquibasedockerprocess-improvement

The deployment was supposed to start at 6 PM. By 10 PM, we were still manually running SQL scripts, rolling back half of them, and praying the application would boot. Twenty developers’ worth of changes, stitched together by hand, deployed through a process that felt more like defusing a bomb than shipping software.

That was the reality I walked into. This is the story of how I turned it into something almost boring. and why boring deployments are the best kind.

The Horror Show

Here’s what “deployment day” looked like before I got involved:

flowchart TD
    A[Developers email code changes] --> B[Lead manually merges files]
    B --> C[Guess at DB migration order]
    C --> D[Run SQL scripts by hand on production]
    D --> E{Did it break?}
    E -->|Yes| F[Frantically debug & roll back]
    F --> D
    E -->|No| G[Deploy application manually]
    G --> H{Does it start?}
    H -->|No| I[More frantic debugging]
    I --> G
    H -->|Yes| J[Cross fingers & go home at 11 PM]

Every deployment was a 4+ hour event. Not an exaggeration. that was the good case. Here’s what made it so painful:

No real version control. Developers would work on their own copies and send files around. The lead developer’s job was to manually piece together everyone’s changes. If two people touched the same file? Good luck.

Database migrations were guesswork. SQL scripts lived in shared folders, sometimes with names like migration_v2_final_FINAL.sql. Nobody knew what order to run them in. Nobody knew which ones had already been applied. Every deployment was a unique snowflake of database chaos.

No testing before production. The first time those database changes ran against a full dataset was in production. On deploy night. While everyone watched.

Manual everything. Every step was a human clicking buttons, copying files, running scripts. One wrong step and the whole thing could cascade.

The team had just accepted this as normal. “Deployments are hard” was the shared belief. And honestly, with a 20-person dev team all pushing changes into a low-code platform with a complex database backend, it was hard. but it didn’t have to be this hard.

The Temptation to Nuke It All

My first instinct was to build the dream CI/CD pipeline. Automated testing, blue-green deployments, infrastructure as code. the works. Ship the whole thing in one glorious rewrite of the deployment process.

I’ve seen that movie before. It ends with an over-engineered system nobody understands, half-finished because the scope exploded, and the team quietly going back to their old ways.

Instead, I started with the most boring thing possible.

Step 1: Teach People Git

Seriously. That was step one.

The team had access to a git repository, but most developers treated it like a file dump. No branching strategy, no meaningful commit messages, no pull requests. Some people weren’t using it at all.

So I ran workshops. Not “Advanced Git Internals”. just the basics:

  • How to create a feature branch
  • How to write a commit message that means something
  • How to open a pull request and review someone else’s code
  • How to resolve a merge conflict without panicking

This sounds trivial. It wasn’t. For a team that had been emailing zip files around, this was a fundamental shift in how they worked together. And it immediately solved the “manually stitching 20 developers’ code together” problem.

Within a few weeks, the lead developer’s merge nightmare was gone. Code flowed through PRs, got reviewed, and landed in a single source of truth. No more guessing whose changes were included.

Lesson: Don’t skip the fundamentals because they seem too basic. If the foundation is broken, nothing you build on top will hold.

Step 2: Tame the Database with Liquibase

With code under control, the database was the next bottleneck. Those hand-crafted SQL scripts with no execution order? That had to go.

I introduced Liquibase. a database migration tool that tracks what’s been applied, enforces ordering, and makes migrations repeatable. Each developer would write their schema changes as versioned changesets. Liquibase would handle the rest.

The key benefits:

  • Ordering is automatic. No more guessing which script runs first.
  • State tracking. Liquibase knows what’s been applied and what hasn’t. Run it twice, nothing breaks.
  • Rollback support. Each changeset can define how to undo itself.
  • Code review for database changes. Changesets go through the same PR process as application code.

This alone cut deployment time dramatically. Instead of a developer manually running scripts and checking results, it was one command: liquibase update. Done.

But I wasn’t satisfied with just automating the execution. The real problem was that we didn’t know if migrations would work until they hit production.

Step 3: Containerize the Migrations

This was the key insight that changed everything. If we could run Liquibase against a production-like database before deployment, we’d catch breaking changes early. not at 9 PM on deploy night.

I built a Docker container that:

  1. Spun up a fresh database instance from a recent production snapshot
  2. Ran all pending Liquibase migrations against it
  3. Ran validation queries to verify data integrity
  4. Reported pass/fail before anyone touched production
# docker-compose.test-migrations.yml
services:
  db:
    image: postgres:15
    volumes:
      - ./snapshots/latest.sql:/docker-entrypoint-initdb.d/init.sql
    healthcheck:
      test: ["CMD-SHELL", "pg_isready"]
      interval: 5s
      timeout: 5s
      retries: 5

  migration-test:
    build: ./migrations
    depends_on:
      db:
        condition: service_healthy
    command: >
      liquibase
        --url=jdbc:postgresql://db:5432/app
        --changelog-file=changelog.xml
        update

Now every PR that included database changes got tested automatically. The container would spin up, apply the migrations, and tell you if anything exploded. all in about two minutes.

The number of deployment-day database surprises dropped to nearly zero.

Step 4: Containerize the Platform

I also containerized the low-code platform itself. but that’s a story for another post. The short version: wrapping the entire application in Docker gave us reproducible builds, consistent environments, and the ability to test the full stack before deployment.

The After

Here’s what deployment looks like now:

flowchart TD
    A[Developer opens PR] --> B[Code review + approval]
    B --> C[Automated migration test in container]
    C --> D{Migrations pass?}
    D -->|No| E[Fix and re-submit PR]
    D -->|Yes| F[Merge to main]
    F --> G[Run Liquibase update on production]
    G --> H[Deploy containerized application]
    H --> I[Verify health checks]
    I --> J[Done in 20 minutes. Go home.]

Twenty minutes. Not four hours. Not six. Twenty.

And the best part? Nobody’s stressed about it anymore. Deployments used to be this dreaded event that the whole team braced for. Now they’re a non-event. Someone kicks it off, it runs, it’s done. Sometimes I’ll deploy and then grab a coffee while it finishes. That’s the victory lap. it’s so routine that it’s almost fun.

What I Learned

Start with the humans, not the tools. The fanciest CI/CD pipeline in the world won’t help if developers are emailing zip files. Fix the workflow first.

Incremental beats big-bang. Each step I described took a few weeks to implement and immediately improved things. If I’d tried to do everything at once, I’d probably still be working on it.

Make the scary thing boring. The goal of good DevOps isn’t to make deployments exciting. it’s to make them so reliable that nobody even thinks about them. “Boring” is the highest compliment a deployment process can receive.

Test before production, always. The containerized migration testing was the single biggest quality-of-life improvement. Knowing that your database changes work before you touch production changes everything about how confident you feel on deploy day.

Respect the existing team. I could have walked in and said “everything you’re doing is wrong.” Instead, I met people where they were and built up from there. The team adopted each change because they could see the immediate benefit, not because someone told them to.

The Scorecard

  • 4+ hour deployments → 20-minute deployments
  • Manual code merging → Git-based PRs with code review
  • SQL scripts in shared folders → Versioned Liquibase changesets
  • First migration test = production → Containerized pre-deploy testing
  • Whole team dreads deploy day → Deploy is a non-event
  • Frequent rollbacks and hotfixes → Rare issues, easy rollback

If your deployments still feel like defusing a bomb, start small. Teach your team git. Automate one thing. Test one migration in a container. You don’t need to fix everything at once. you just need to start making it a little less painful, one step at a time.

The victory laps will come.