↓Skip to main content

Testing in Go: Golden Files

·14 mins
Table of Contents

Hardcoding the expected values in an assertion is a straightforward approach in testing. Most of the time, we know the expected output of the unit under test, so simply adding the raw value to the assertion works well.

Things can get tricky when we are testing a unit whose output is cumbersome to hardcode. The straightforward remedy is to extract this cumbersome value to a file that we can then read and compare the output of the unit under test to the output of the file.

In Go, we call such files golden files. Golden files contain the expected output of a test. When tests run, they read the contents of the golden file and compare it to the output of the unit under test.

As always, let’s make our lives easier and discuss golden files using an example.

Book reports #

Imagine you own a small library or a book store. Your inventory software exposes an API where you can get the list of books in said inventory. You want a program that will take all that data and format it in a Markdown table. Maybe you want to print this table or send it to your accountant. The possibilities are endless!

Let’s see a simulation of such a program:

// report/report.go

package report

import (
	"bytes"
	"log"
	"text/template"

	"github.com/fteem/go-playground/golden-files/books"
)

const (
	header string = `
| Title         | Author        | Publisher |  Pages  |  ISBN  |  Price  |
| ------------- | ------------- | --------- | ------- | ------ | ------- |
`
	rowTemplate string = "|  {{ .Title }}  |  {{ .Author }}  |  {{ .Publisher }}  |  {{ .Pages }}  |  {{ .ISBN }}  |  {{ .Price }}  |\n"
)

func Generate(books []books.Book) string {
	buf := bytes.NewBufferString(header)

	t := template.Must(template.New("table").Parse(rowTemplate))

	for _, book := range books {
		err := t.Execute(buf, book)
		if err != nil {
			log.Println("Error executing template:", err)
		}
	}

	return buf.String()
}

In the report package, we have a single function Generate which generates the report Markdown table. It gets all the books from the books.Books package, iterates over them and uses the rowTemplate to generate each of the rows of the table. In the end, it returns the whole Markdown table as in a single string.

Let’s take a quick peek at the books package:

// books/books.go

package books

type Book struct {
	ISBN      string
	Title     string
	Author    string
	Pages     int
	Publisher string
	Price     int
}

var Books []Book = []Book{
	Book{
		ISBN:      "978-1591847786",
		Title:     "Hooked",
		Author:    "Nir Eyal",
		Pages:     256,
		Publisher: "Portfolio",
		Price:     19,
	},
	Book{
		ISBN:      "978-1434442017",
		Title:     "The Great Gatsby",
		Author:    "F. Scott Fitzgerald",
		Pages:     140,
		Publisher: "Wildside Press",
		Price:     12,
	},
	Book{
		ISBN:      "978-1784756260",
		Title:     "Then She Was Gone: A Novel",
		Author:    "Lisa Jewell",
		Pages:     448,
		Publisher: "Arrow",
		Price:     29,
	},
	Book{
		ISBN:      "978-1094400648",
		Title:     "Think Like a Billionaire",
		Author:    "James Altucher",
		Pages:     852,
		Publisher: "Scribd, Inc.",
		Price:     9,
	},
}

The books package contains just the list of the books in our fake inventory.

If we ran the code we would get the following output:

$ go run main.go

| Title         | Author        | Publisher |  Pages  |  ISBN  |  Price  |
| ------------- | ------------- | --------- | ------- | ------ | ------- |
|  Hooked  |  Nir Eyal  |  Portfolio  |  256  |  978-1591847786  |  19  |
|  The Great Gatsby  |  F. Scott Fitzgerald  |  Wildside Press  |  140  |  978-1434442017  |  12  |
|  Then She Was Gone: A Novel  |  Lisa Jewell  |  Arrow  |  448  |  978-1784756260  |  29  |
|  Think Like a Billionaire  |  James Altucher  |  Scribd, Inc.  |  852  |  978-1094400648  |  9  |

Given that this blog is generated using Markdown, here’s the table rendered:

Title Author Publisher Pages ISBN Price
Hooked Nir Eyal Portfolio 256 978-1591847786 19
The Great Gatsby F. Scott Fitzgerald Wildside Press 140 978-1434442017 12
Then She Was Gone: A Novel Lisa Jewell Arrow 448 978-1784756260 29
Think Like a Billionaire James Altucher Scribd, Inc. 852 978-1094400648 9

Not the prettiest table in the world, still a good overview of the inventory. Now that we have the report in place let’s see how we can test it.

Testing our reports #

Testing the Generate function is straightforward, using the table-driven approach:

package report

import (
	"strings"
	"testing"

	"github.com/fteem/go-playground/golden-files/books"
)

func TestGenerate(t *testing.T) {
	testcases := []struct {
		name  string
		books []books.Book
		want  string
	}{
		{
			name: "WithInventory",
			books: []books.Book{
				books.Book{
					Title:  "The Da Vinci Code",
					Author: "Dan Brown",
					Pages:  592,
					ISBN:   "978-0552161275",
					Price:  7,
				},
				books.Book{
					Title:  "American on Purpose",
					Author: "Craig Ferguson",
					Pages:  288,
					ISBN:   "978-0061959158",
					Price:  19,
				},
			},
			want: `
| Title         | Author        | Pages  | ISBN  | Price  |
| ------------- | ------------- | ------ | ----- | ------ |
| The Da Vinci Code | Dan Brown | 592 | 978-0552161275 | 7 |
| American on Purpose | Craig Ferguson | 288 | 978-0061959158 | 19 |
`,
		},
		{
			name:  "EmptyInventory",
			books: []books.Book{},
			want: `
| Title         | Author        | Pages  | ISBN  | Price  |
| ------------- | ------------- | ------ | ----- | ------ |
`,
		},
	}

	for _, testcase := range testcases {
		got := Generate(testcase.books)
		if got != testcase.want {
			t.Errorf("Want:\n%s\nGot:%s", testcase.want, got)
		}
	}
}

The formatting of the expected output is rather hard to scan. That’s because our test cases test the actual output of the table, which is a string with a particular format.

Even worse, in cases when these inconvenient strings are hardcoded, a failing test can be hard to debug if there’s a mismatch in leading or trailing whitespace. Consider this case, for example:

$ go test -v
=== RUN   TestGenerate
--- FAIL: TestGenerate (0.00s)
    report_test.go:54: Want:

        | Title         | Author        | Pages  | ISBN  | Price  |
        | ------------- | ------------- | ------ | ----- | ------ |
        | The Da Vinci Code | Dan Brown | 592 | 978-0552161275 | 7 |
        | American on Purpose | Craig Ferguson | 288 | 978-0061959158 | 19 |

        Got:
        | Title         | Author        | Pages  | ISBN  | Price  |
        | ------------- | ------------- | ------ | ----- | ------ |
        | The Da Vinci Code | Dan Brown | 592 | 978-0552161275 | 7 |
        | American on Purpose | Craig Ferguson | 288 | 978-0061959158 | 19 |
FAIL
exit status 1
FAIL	github.com/fteem/go-playground/golden-files/report-verbose	0.005s

While the wanted and the gotten values look identical, the test is failing because one of the values has trailing whitespace. To avoid hardcoding such outputs in our tests, we resort to using golden files.

Converting to golden files #

To use the golden files, we need to extract the tables to a separate file in the testdata directory. If the testdata directory is foreign to you: it is a directory that the go build tool will ignore when building the binaries of your programs. You can read more about it in my post on fixtures in Go.

Using this approach, we want to move the two table outputs to two different files:

  1. testdata/empty_inventory.golden, and
  2. testdata/with_inventory.golden

Both of them will have the corresponding output of the test cases:

$ cat report/testdata/empty_inventory.golden
| Title         | Author        | Publisher |  Pages  |  ISBN  |  Price  |
| ------------- | ------------- | --------- | ------- | ------ | ------- |

$ cat report/testdata/with_inventory.golden
| Title         | Author        | Publisher |  Pages  |  ISBN  |  Price  |
| ------------- | ------------- | --------- | ------- | ------ | ------- |
| The Da Vinci Code | Dan Brown | Corgi | 592 | 978-0552161275 | 7 |
| American on Purpose | Craig Ferguson | Harper Collins | 288 | 978-0061959158 | 19 |

Now, let’s change up our test cases to use the golden files to compare the outputs:

package report

import (
	"io/ioutil"
	"testing"

	"github.com/fteem/go-playground/golden-files/books"
)

func TestGenerate(t *testing.T) {
	testcases := []struct {
		name   string
		books  []books.Book
		golden string
	}{
		{
			name: "WithInventory",
			books: []books.Book{
				books.Book{
					Title:     "The Da Vinci Code",
					Author:    "Dan Brown",
					Publisher: "Corgi",
					Pages:     592,
					ISBN:      "978-0552161275",
					Price:     7,
				},
				books.Book{
					Title:     "American on Purpose",
					Author:    "Craig Ferguson",
					Publisher: "Harper Collins",
					Pages:     288,
					ISBN:      "978-0061959158",
					Price:     19,
				},
			},
			golden: "with_inventory",
		},
		{
			name:   "EmptyInventory",
			books:  []books.Book{},
			golden: "empty_inventory",
		},
	}

	for _, testcase := range testcases {
		got := Generate(testcase.books)
		content, err := ioutil.ReadFile("testdata/" + testcase.golden + ".golden")
		if err != nil {
			t.Fatalf("Error loading golden file: %s", err)
		}
		want := string(content)

		if got != want {
			t.Errorf("Want:\n%s\nGot:\n%s", want, got)
		}
	}
}

The notable changes are on the highlighted lines above:

  • we add a golden attribute to each test case, representing the name of its golden file
  • for each of the test cases, we open the respective golden file and read all of its contents
  • we use the contents to compare the expected (want) and the actual (got) values

This approach removes hardcoded strings from our tests, and it comes with two important features:

First, golden files are coupled to the test cases (a.k.a. the inputs). That means that if the inputs change, the output will change, so you will have to update the contents of the golden file. Also, if you add another test case, you will probably have to create a golden file for it. If you forget to create it, your tests will fail - which is a good thing!

Second, golden files contain the expected outcome of a test case. So, if the implementation of the code under test changes, your test will fail - which is also a good thing! But that also means our golden files will be out of date, and we have to update them.

Keeping our golden files up to date is a caveat that we always have to keep in mind. As our applications evolve and their functionalities change, the golden files will have to follow suit. If we’re working on a project with many golden files, keeping them up to date can be a frustrating exercise. Therefore it’s often a good idea to automate this, so it becomes an effortless task.

Here’s a straightforward way to automate the golden file updating.

Keeping our golden files up to date #

The whole idea here is, after any change of the implementation or the test inputs, to update the golden files saving ourselves time from copy-pasting outputs to files.

The usual approach in the wild is to provide a command-line flag, usually -update, which you can use with the go test tool:

// report/report_test.go
var (
	update = flag.Bool("update", false, "update the golden files of this test")
)

func TestMain(m *testing.M) {
	flag.Parse()
	os.Exit(m.Run())
}

The update flag will be a bool that we will use in our function that will read and write to the golden files.

Before Go v1.13, the test’s init function invoked flag.Parse(). Since Go v1.13, the TestMain function should invoke the flag.Parse() function. This change of behavior is due to changes introduced in Go 1.13, where the testing package internally parses the flags before it runs the tests. Consequently, if we would skip the TestMain function, go test would not recognize the -update flag.

(You can read more about the change in this Github issue.)

If you are unsure about the meaning of the TestMain function, you can read more about it in my article on the topic here.

Let’s see the rest of the test file and the notable changes:

func TestGenerate(t *testing.T) {
	testcases := []struct {
		name   string
		books  []books.Book
		golden string
	}{
		{
			name: "WithInventory",
			books: []books.Book{
				books.Book{
					Title:     "The Da Vinci Code",
					Author:    "Dan Brown",
					Publisher: "Corgi",
					Pages:     592,
					ISBN:      "978-0552161275",
					Price:     7,
				},
				books.Book{
					Title:     "American on Purpose",
					Author:    "Craig Ferguson",
					Publisher: "Harper Collins",
					Pages:     288,
					ISBN:      "978-0061959158",
					Price:     19,
				},
			},
			golden: "with_inventory",
		},
		{
			name:   "EmptyInventory",
			books:  []books.Book{},
			golden: "empty_inventory",
		},
	}

	for _, testcase := range testcases {
		got := Generate(testcase.books)
		want := goldenValue(t, testcase.golden, got, *update)

		if got != want {
			t.Errorf("Want:\n%s\nGot:\n%s", want, got)
		}
	}
}

func goldenValue(t *testing.T, goldenFile string, actual string, update bool) string {
	t.Helper()
	goldenPath := "testdata/" + goldenFile + ".golden"

	f, err := os.OpenFile(goldenPath, os.O_RDWR, 0644)
	defer f.Close()

	if update {
		_, err := f.WriteString(actual)
		if err != nil {
			t.Fatalf("Error writing to file %s: %s", goldenPath, err)
		}

		return actual
	}

	content, err := ioutil.ReadAll(f)
	if err != nil {
		t.Fatalf("Error opening file %s: %s", goldenPath, err)
	}
	return string(content)
}

The significant changes are in the highlighted lines above.

The goldenValue function takes the goldenFile name as an argument, with the actual value of the test returned by the function under test and with the update bool passed from the flag.

Then it takes three steps:

  1. It opens the golden file in an R/W mode.
  2. If the update flag is true, it will update the golden file with the contents saved in the actual argument and return it.
  3. If the update flag is not set, it will continue to read the contents of the golden file and return them as a string.

If any of the reading or writing steps fail, it will crash the test and stop further test execution.

Let’s test this in action. First, the usual, flag-less run of go test:

$ go test -v ./report/...
=== RUN   TestGenerate
--- PASS: TestGenerate (0.00s)
PASS
ok  	github.com/fteem/go-playground/golden-files/report0.005s >}}

Now, let’s change our implementation. We will remove the publisher column from the report that will make our tests fail because the golden file expects the Publisher column to be present:

$ go test -v ./report/...
=== RUN   TestGenerate
--- FAIL: TestGenerate (0.00s)
    report_test.go:61: Want:

        | Title         | Author        | Publisher |  Pages  |  ISBN  |  Price  |
        | ------------- | ------------- | --------- | ------- | ------ | ------- |
        | The Da Vinci Code | Dan Brown | Corgi | 592 | 978-0552161275 | 7 |
        | American on Purpose | Craig Ferguson | Harper Collins | 288 | 978-0061959158 | 19 |

        Got:

        | Title         | Author        |  Pages  |  ISBN  |  Price  |
        | ------------- | ------------- | ------- | ------ | ------- |
        | The Da Vinci Code | Dan Brown | 592 | 978-0552161275 | 7 |
        | American on Purpose | Craig Ferguson | 288 | 978-0061959158 | 19 |
    report_test.go:61: Want:

        | Title         | Author        | Publisher |  Pages  |  ISBN  |  Price  |
        | ------------- | ------------- | --------- | ------- | ------ | ------- |

        Got:

        | Title         | Author        |  Pages  |  ISBN  |  Price  |
        | ------------- | ------------- | ------- | ------ | ------- |
FAIL
FAIL	github.com/fteem/go-playground/golden-files/report	0.006s
FAIL

The difference in the two failing tests is noticeable – we’re missing the column. Since we know that this is the new behavior of the report going forward, we only want to update the golden files. Let’s add the -update flag:

$ go test -v ./report/... -update
=== RUN   TestGenerate
--- PASS: TestGenerate (0.00s)
PASS
ok  	github.com/fteem/go-playground/golden-files/report	0.006s

Nothing special, only passing tests? Let’s inspect the contents of the golden files:

$ cat report/testdata/with_inventory.golden report/testdata/empty_inventory.golden

report/testdata/with_inventory.golden
| Title         | Author        |  Pages  |  ISBN  |  Price  |
| ------------- | ------------- | ------- | ------ | ------- |
| The Da Vinci Code | Dan Brown | 592 | 978-0552161275 | 7 |
| American on Purpose | Craig Ferguson | 288 | 978-0061959158 | 19 |

report/testdata/empty_inventory.golden

| Title         | Author        |  Pages  |  ISBN  |  Price  |
| ------------- | ------------- | ------- | ------ | ------- |

Voilà! The using the -update flag, we removed the “Publisher” from our golden files, and the tests pass.

Few words of caution #

While the golden files are a simple technique that improves our tests' legibility, they come with a few caveats.

It’s important to remember that you have to version these files. They are part of the project, and if you do not check them in your version control, your test suite will fail the moment it’s run in continuous integration.

Also, like fixtures, you should keep them in your testdata directory. The go build tool ignores any files in a testdata directory, and it excludes these files when building the binary. While we are comparing golden files and fixtures, there is another similarity between the two - the test files are not self-sufficient. This means that you will have to know the contents of another file to understand what is the expected outcome of your tests.

Furthermore, parsing the golden files can be funny (or frustrating), depending on their content. Be careful when you’re storing content like the one I showed in the example above. If the golden file has lots of whitespaces, new lines or indentation, parsing, and comparing it to other strings can be annoying to debug.

Lastly, the -update flag has an important caveat: it must be available in all packages of your project to function. If you would like to be able to run go test ./... -update to update all golden files in your project, all packages in the project must know of -update.

Even if there’s one package that is not aware of -update, running the tests will error out:

❯ go test ./... -v -update
?   	github.com/fteem/go-playground/golden-files	[no test files]
flag provided but not defined: -update

Such a behavior is because my Go module has more than one package in it, while only one of them (report) knows of -update’s existence.

In general, the benefits you get with golden files are great, but I suggest treading carefully, given the caveats mentioned above.

↑