An Introduction to fakemake

Andreas Dominik Cullmann

2020-01-10, 14:19:59

Why Mock the Unix Make Utility?

There are many build systems, and even more uses for build systems (see Powers et al. 2002, sec. 11.10 and 11.11).

I have been using the unix make utility when developing R packages since 2012. But sometimes I get caught on a machine where make is not available and where I am not entitled to install it1.

This is why I wrote fakemake: to build an R package conditionally on the modification times of (file) dependencies without having to rely on external software. If you have any proper build system at hand: stick to it, do not use fakemake.

withr and knitr

Throughout this vignette I use Rs temporary directory, often by usingwithr::with_dir(tempdir(), …)`. Because this is a vignette and the codes are examples. In real life, we would skip the temporary directory stuff.

This vignette is built using knitr, which itself uses sink(). As sink() is central to fakemake for redirecting output to files in the make chain, I have to disable some of knitrs output here and there. Dont worry, its just because *knitr* and *fakemake* both want to usesink()` exclusively and it only affects vignettes built with knitr.

Makelists

A makelist is fakemakes representation of a Makefile. Its just a list of lists. Look at the minimal makelist provided by fakemake:

str(fakemake::provide_make_list("minimal", clean_sink = TRUE))
## List of 4
##  $ :List of 3
##   ..$ target       : chr "all.Rout"
##   ..$ prerequisites: chr [1:2] "a1.Rout" "a2.Rout"
##   ..$ code         : chr "print(\"all\")"
##  $ :List of 2
##   ..$ target: chr "a2.Rout"
##   ..$ code  : chr "print(\"a2\")"
##  $ :List of 3
##   ..$ target       : chr "a1.Rout"
##   ..$ prerequisites: chr "b1.Rout"
##   ..$ code         : chr "print(\"a1\")"
##  $ :List of 2
##   ..$ target: chr "b1.Rout"
##   ..$ code  : chr "print(\"b1\")"

Each sublist represents a Makefiles target rule and has several items: at least a *target* and either *code* or *prerequisites*, possibly both. This makelist would still be a Makefiles valid representation if target rule #3 with target “a1.Rout” had no (or an empty) code item.`

Other possible target rule entries are:

A Minimal Example

Suppose we would have a minimal makelist:

ml <- fakemake::provide_make_list("minimal", clean_sink = TRUE)

We can visualize the makelist (giving the root is optional, in this case it just makes a neater plot):

fakemake::visualize(ml, root = "all.Rout")

Building and Rebuilding

Now build the “all.Rout” target:

withr::with_dir(tempdir(), print(fakemake::make("all.Rout", ml)))
## [1] "b1.Rout"  "a1.Rout"  "a2.Rout"  "all.Rout"

We can see the files created:

show_file_mtime <- function(files = list.files(tempdir(), full.names = TRUE,
                                               pattern = "^.*\\.Rout")) {
    return(file.info(files)["mtime"])
}
show_file_mtime()
##                                        mtime
## /tmp/RtmpLYHpPG/a1.Rout  2020-04-12 11:12:16
## /tmp/RtmpLYHpPG/a2.Rout  2020-04-12 11:12:16
## /tmp/RtmpLYHpPG/all.Rout 2020-04-12 11:12:16
## /tmp/RtmpLYHpPG/b1.Rout  2020-04-12 11:12:16

If we wait for a second and rerun the build process, we get:

# ensure the modification time would change if the files were recreated
Sys.sleep(1)
withr::with_dir(tempdir(), print(fakemake::make("all.Rout", ml)))
## NULL
show_file_mtime()
##                                        mtime
## /tmp/RtmpLYHpPG/a1.Rout  2020-04-12 11:12:16
## /tmp/RtmpLYHpPG/a2.Rout  2020-04-12 11:12:16
## /tmp/RtmpLYHpPG/all.Rout 2020-04-12 11:12:16
## /tmp/RtmpLYHpPG/b1.Rout  2020-04-12 11:12:16

Nothing changed. Good. Now, we change one file down the build chain:

fakemake::touch(file.path(tempdir(), "b1.Rout"))
withr::with_dir(tempdir(), print(fakemake::make("all.Rout", ml)))
## [1] "a1.Rout"  "all.Rout"
show_file_mtime()
##                                        mtime
## /tmp/RtmpLYHpPG/a1.Rout  2020-04-12 11:12:18
## /tmp/RtmpLYHpPG/a2.Rout  2020-04-12 11:12:16
## /tmp/RtmpLYHpPG/all.Rout 2020-04-12 11:12:18
## /tmp/RtmpLYHpPG/b1.Rout  2020-04-12 11:12:18

Since a1.Rout depends on b1.Rout and all.Rout depends on a1.Rout, these targets get rebuilt while a2.Rout stays untouched.

Had we touched a1.Rout, b1.Rout would not have been rebuilt:

fakemake::touch(file.path(tempdir(), "a1.Rout"))
withr::with_dir(tempdir(), print(fakemake::make("all.Rout", ml)))
## [1] "all.Rout"
show_file_mtime()
##                                        mtime
## /tmp/RtmpLYHpPG/a1.Rout  2020-04-12 11:12:19
## /tmp/RtmpLYHpPG/a2.Rout  2020-04-12 11:12:16
## /tmp/RtmpLYHpPG/all.Rout 2020-04-12 11:12:19
## /tmp/RtmpLYHpPG/b1.Rout  2020-04-12 11:12:18

Forcing the Build

If you set the force option, you can force the target and all its prerequisites down the build chain to be built:

withr::with_dir(tempdir(), print(fakemake::make("all.Rout", ml, force = TRUE)))
## [1] "b1.Rout"  "a1.Rout"  "a2.Rout"  "all.Rout"

If you want to force the target itself, but not all its prerequisites, set recursive = FALSE:

withr::with_dir(tempdir(), print(fakemake::make("all.Rout", ml, force = TRUE,
                                                recursive = FALSE)))
## [1] "all.Rout"

Faking the Build

If you dont actually want to run the recipes but would rather like to know what would happen if you ran the build chain (this mocks GNU makes -n option), you can set dry_run = TRUE:

file.remove(dir(tempdir(), pattern = ".*\\.Rout", full.names = TRUE))
## [1] TRUE TRUE TRUE TRUE
withr::with_dir(tempdir(), print(fakemake::make("all.Rout", ml, dry_run = TRUE)))
## [1] "b1.Rout"  "a1.Rout"  "a2.Rout"  "all.Rout"

Note that no files have been created:

dir(tempdir(), pattern = ".*\\.Rout")
## character(0)

So we recreate them now:

withr::with_dir(tempdir(), print(fakemake::make("all.Rout", ml)))
## [1] "b1.Rout"  "a1.Rout"  "a2.Rout"  "all.Rout"
dir(tempdir(), pattern = ".*\\.Rout")
## [1] "a1.Rout"  "a2.Rout"  "all.Rout" "b1.Rout"

Using Aliases

If you find a target rule`s target too hard to type, you can use an alias:

i <- which(sapply(ml, "[[", "target") == "all.Rout")
ml[[i]]["alias"] <- "all"
withr::with_dir(tempdir(), print(fakemake::make("all", ml, force = TRUE)))
## [1] "b1.Rout"  "a1.Rout"  "a2.Rout"  "all.Rout"

This is pointless here, but targets might be files down a directory tree like log/roxygen2.Rout when building R packages: you might want to alias that target to roxygen

Diverting Output / Programmatically Creating a Target Rule`s Target

Target rule b1 dumps its output to b1.Rout:

cat(readLines(file.path(tempdir(), "b1.Rout")), sep = "\n")
## [1] "b1"

Suppose it would programmatically create the target:

i <- which(sapply(ml, "[[", "target") == "b1.Rout")
ml[[i]]["code"]  <- paste(ml[[i]]["code"],
                      "cat('hello, world\n', file = \"b1.Rout\")",
                      "print(\"foobar\")",
                      sep = ";")
withr::with_dir(tempdir(), print(fakemake::make("b1.Rout", ml, force = TRUE)))
## [1] "b1.Rout"
cat(readLines(file.path(tempdir(), "b1.Rout")), sep = "\n")
## hello, wo[1] "foobar"

You end up with a broken target file, so you need to add a sink:

ml[[i]]["sink"] <- "b1.txt"
withr::with_dir(tempdir(), print(fakemake::make("b1.Rout", ml, force = TRUE)))
## [1] "b1.Rout"

Now you get what you wanted:

cat(readLines(file.path(tempdir(), "b1.Rout")), sep = "\n")
## hello, world
cat(readLines(file.path(tempdir(), "b1.txt")), sep = "\n")
## [1] "b1"
## [1] "foobar"

We need sinks when the targets code creates the target, for example when it builds a packages tarball: we would want to get the output of building the tarball to be written to a file the path of which we specify via the target`s sink.

No Code Targets

Rule a1 has code

i <- which(sapply(ml, "[[", "target") == "a1.Rout")
ml[[i]]["code"]
## $code
## [1] "print(\"a1\")"

that prints “a1” into “a1.Rout”:

cat(readLines(file.path(tempdir(), "a1.Rout")), sep = "\n")
## [1] "a1"

If we remove that code and its output file and rerun

ml[[i]]["code"]  <- NULL
withr::with_dir(tempdir(), print(fakemake::make("a1.Rout", ml, force = TRUE)))
## [1] "b1.Rout" "a1.Rout"

the file is still created (note that target rule b1 down the make chain is run since we did not set recursive = FALSE ) but empty:

file.size(file.path(tempdir(), "a1.Rout"))
## [1] 0

Phony Targets

As you have seen, you can temporarily force a build. You may set a target to be .PHONY which forces it (but not its prerequisites) to be built:

ml[[i]][".PHONY"]  <- TRUE
withr::with_dir(tempdir(), print(fakemake::make("a1.Rout", ml)))
## [1] "a1.Rout"

References

Powers, Shelley, Jerry Peek, Tim O’Reilly, and Mike Loudikes. 2002. Unix Power Tools. O’Reilly & Associates.

  1. This is a nice example of what restrictive software policies are good for: you end up with a buggy imitation like fakemake instead of the well established original. You should not regulate software installations for programmers, unless you take away their interpreters/compilers.↩