Implementing Flyway with Docker

You can implement Flyway in a Docker-based application by adding it as a service in the docker-compose.yml and configuring the correct environment variables.

Sample Set-up

Here is an example docker-compose.yml file based on a simple back-end application that has an API layer and a Postgres database layer.

version: '3'
services:
   api:
      container_name: my-api
      env_file:
         - .env
      build: .
      ports:
         - 5050:${SERVER_PORT}/tcp
      links:
         - postgres
   postgres:
      container_name: my-db
      environment:
         - POSTGRES_USER=${POSTGRES_USER}
         - POSTGRES_PASSWORD=${POSTGRES_PASSWORD
         - POSTGRES_DB=${POSTGRES_DB}
      image: library/postgres:latest
      ports:
         - ${POSTGRES_PORT}:${POSTGRES_PORT}/tcp
   flyway:
      container_name: my-flyway
      environment:
         - FLYWAY_USER=${POSTGRES_USER}
         - FLYWAY_PASSWORD=${POSTGRES_PASSWORD}
         - FLYWAY_URL=jdbc:postgresql://postgres:${POSTGRES_PORT}/${POSTGRES_DB}
         - FLYWAY_SCHEMAS=flyway,${POSTGRES_SCHEMA}
         - FLYWAY_GROUP=true
         - FLYWAY_LICENSE_KEY=${FLYWAY_LICENSE_KEY}
         - FLYWAY_EDITION=${FLYWAY_EDITION}
     image: flyway/flyway:latest
     command: -locations=filesystem:/flyway/sql -connectRetries=60 migrate
      volumes:
         - $PWD/sql_versions:/flyway/sql
      depends_on:
          - postgres
      image: flyway/flyway:latest
      command: -locations=filesystem:/flyway/sql -connectRetries=60 migrate
      volumes:
         - $PWD/sql_versions:/flyway/sql
      depends_on:
          - postgres

There are three services listed here: the API (api), the database (postgres), and Flyway (flyway). It also refers to an .env file that contains the following keys concerning the database:

POSTGRES_USER=aUser
POSTGRES_PASSWORD=youllNeverGuess
POSTGRES_DB=my_database
POSTGRES_SCHEMA=my_schema
POSTGRES_PORT=5432

Environment Variable Configuration

You can be more or less exhaustive with your environment variables — the important thing is that Flyway needs to know the following information under its environment header, and, since you need the same information for your database container, DRY!

FLYWAY_USER: the user that Flyway logs in as
FLYWAY_PASSWORD: the password for the above user
FLYWAY_URL: this is specifically for the jdbc connector, so plug your values in: jdbc:postgresql://postgres:${POSTGRES_PORT}/${POSTGRES_DB}
FLYWAY_SCHEMAS: a comma-separated, case-sensitive list of schemas.

• Omitting this defaults the schema to “public”, and Flyway will create the “public” schema if it does not already exist, and store its metadata table “flyway_schema_history” there as well. Otherwise, the first schema listed here becomes Flyway’s default schema (and home to its metadata table).
• I chose to make a new schema for Flyway called “flyway” because of Postgres’s default wide open permissions on the “public” schema.
• This is an important variable because I found that Flyway’s “clean” command, which wipes your database, ignores any schemas (and their children) which are created in migration scripts. In order for “clean” to truly, well, clean, I needed to give Flyway explicit access to the schemas I wanted it to be able to clean.

FLYWAY_GROUP: boolean, default: false. If true, Flyway executes all migrations in a single transaction. As an example, say we have two new migrations, V3, which is valid, and V4, which contains a syntax error. Flyway will run the migrations in order of version number, so first V3 and then V4. If FLYWAY_GROUP is false, V3 will be applied to the database and then V4 will fail, but any changes made by V3 remain intact. If FLYWAY_GROUP is true, then when V4 errors, the changes made by V3 are rolled back as well, leaving the database as it was before the attempted migration. The same thing happens with Undo scripts.

For Pro and Enterprise edition users:

FLYWAY_LICENSE_KEY/FLYWAY_EDITION: Flyway threw an error saying I required an upgrade unless I had both of these environment variables set. FLYWAY_LICENSE_KEY is exactly what it sounds like. FLYWAY_EDITION can accept “community”, “pro”, or “enterprise”, and defaults to “community” (so if you are using the free version, there’s no need to set either of these).

- FLYWAY_EDITION=${FLYWAY_EDITION}
- FLYWAY_LICENSE_KEY=${FLYWAY_LICENSE_KEY}

Here is Docker’s documentation on other environment variables you can configure.

docker-compose.yml Configuration

container_name and image are pretty standard Docker stuff, but here are a few Flyway-specific details:

volumes: Here I am setting up a location for storing my migration files. $PWD/sql_versions is the directory in my repo where I am storing the version-numbered migration files (more on that below). I am mapping that to the directory /flyway/sql in the Flyway container using the : operator.

command: a few flags go here.

-locations=filesystem:/flyway/sql

• It’s /flyway/sql again because this corresponds to the volume I just set up! This is a parameter of the Flyway migrate command, telling it that the location(s) of the files it is to run. filesystem refers to the filesystem on the Docker container.
• This can be a list of locations, comma-separated and case-sensitive. Even if you store your files into multiple locations, Flyway still executes them in version order. Therefore, you cannot have version overlap between them.

-connectRetries=60: Automatic retries every second for x seconds! Sometimes your database container takes a few seconds to spin up — this tells Flyway to keep looking for it.

migrate: Flyway documentation. Run all the SQL scripts in the assigned locations, in order of version.

depends_on: Docker-compose uses depends_on flags to determine the order of container creation. This ensures that the database is spun up before Flyway is. I ran into some timing issues omitting this.

Building the Containers

At this point, you should be able to simply run docker-compose up along with any desired options (my current project uses the following options: docker-compose up -d — -build (that’s supposed to be two hyphens before build, but Medium’s autoformatting is foiling me)). Please note that Flyway does not persist for longer than it takes to run the migrations (at least not in any way I’ve figured out yet!). The Flyway container will be created, run the migrations, and close itself (if you are fast enough with docker container ls, you might see it before it exits).

This means that, in the future, anytime you add new migrations, you must run docker-compose up flyway to spin up the Flyway container again to apply them. Once again, the Flyway container will exit once it finishes.

A real-life example: for my current project, my SQL scripts are currently contained in the same repo as my API, so the deployment strategy is to spin up the Flyway container to check for any migrations anytime there is a change to the API repo — basically docker-compose up api && docker-compose up flyway in order to apply any changes to either the API or the database. As I learn more about these interactions, this strategy may be updated later, at which point so will this documentation.

Flyway Commands Besides migrate

• clean: Flyway documentation. Drops all schemas and children objects per your configuration in FLYWAY_SCHEMAS. This is best reserved for local dev environments. As the Flyway documentation aptly puts it:

running clean can be quite a career limiting move

HANDLE WITH CARE!!

NOTE: An environment variable exists for FLYWAY_CLEAN_DISABLED that we may want to implement in Production deployments.

• info: Flyway documentation. Prints status info about your migrations, including current version. Similar to running SELECT * FROM flyway.flyway_schema_history; in your database.
• validate: Flyway documentation. Checks your current database structure against your SQL scripts. For instance, if I ran Flyway’s migrate command against the following SQL script:

CREATE TABLE some_schema.some_table (
    id SERIAL PRIMARY KEY,
    name TEXT NOT NULL
);

And then changed the SQL script without running another migrate:

CREATE TABLE some_schema.some_table (
    id SERIAL PRIMARY KEY,
    name TEXT NOT NULL,
    date_created timestamp
);

And ran a validate check, I’d get an error message that looks like this:

my-flyway | Flyway Community Edition 6.5.5 by Redgate
my-flyway | Database: jdbc:postgresql://postgres:5432/my_project (PostgreSQL 12.3)
my-flyway | ERROR: Validate failed:
my-flyway | Migration checksum mismatch for migration version 1
my-flyway | -> Applied to database : 989045914
my-flyway | -> Resolved locally : -1977374147
my-flyway |
my-flyway exited with code 1

“This is very useful to detect accidental changes that may prevent you from reliably recreating the schema.”

Please note that running migrate will also perform a validate, and return the same sort of message.

• undo: Rolls back the database to the previous version. If you want to roll back multiple versions, you must call undo multiple times. When you undo your migrations to before your very first version, your schema version will be “<< Empty Schema >>” and any further undos are ignored.

Migration Files

See Flyway documentation here. The “__” in each filename is two underscores, although this is not necessary unless you are also adding a description. In a nutshell:

versioned migrations: filenames begin with “Vx” where x is the version number — for example, “V1__Init_table.sql” or “V2”. THESE CANNOT OVERLAP — having a “V1__Init_table.sql” and a “V1__do_something_else.sql” will tank the migration.

repeatable migrations: filenames begin with “R__”. Run after every set of versioned migrations.

• Repeatable migrations must be idempotent, or able to applied multiple times without failing or messing up your expected set-up — examples of this is the use of CREATE OR REPLACE clauses, or truncating a table first if you are adding seed data.

undo migrations: filenames begin with “Ux” where x is the version number — for example, “U1__Init_table.sql” or “U2”. THESE CANNOT OVERLAP, and you must have a corresponding undo migration for every version migration if you want to use the undo migrations.

Other Resources

• Flyway’s documentation homepage
• GitHub repo that was the basis of a lot of this research