= The Mill Evaluation Model
:page-aliases: The_Mill_Evaluation_Model.adoc, depth/execution-model.adoc



This page does a deep dive on how Mill evaluates your build tasks, so you can better understand
what Mill is doing behind the scenes when building your project.

== Example Project

For the purposes of this article, we will be using the following example build
as the basis for discussion:

[source,scala]
----
// build.mill
package build
import mill.*, javalib.*

object foo extends JavaModule {}

object bar extends JavaModule {
  def moduleDeps = Seq(foo)

  /** Total number of lines in module source files */
  def lineCount = Task {
    allSourceFiles().map(f => os.read.lines(f.path).size).sum
  }

  /** Generate resources using lineCount of sources */
  override def resources = Task {
    os.write(Task.dest / "line-count.txt", "" + lineCount())
    Seq(PathRef(Task.dest))
  }
}
----

This is a simple two-module build with two ``JavaModule``s, one that depends on the other.
There is a custom task `bar.lineCount` implemented that replaces the default `resources/`
folder with a generated resource file for use at runtime, as a simple example of a
xref:javalib/intro.adoc#_custom_build_logic[Custom Build Logic].

This expects the source layout:

[source]
----
foo/
    src/
        *.java files
    package.mill (optional)
bar/
    src/
        *.java files
    package.mill (optional)
build.mill
----

You can operate on this build via commands such as

[source,console]
----
> ./mill bar.compile

> ./mill foo.run

> ./mill _.assembly # evaluates both foo.compile and bar.compile
----


For the purposes of this article, we will consider what happens when you run
`./mill _.assembly` on the above example codebase.

== Primary Phases

=== Compilation

Initial `.mill` build files:

[source,txt]
----
bar/
    package.mill #optional
foo/
    package.mill #optional
build.mill
----


This stage involves compiling your `build.mill` and any
xref:large/multi-file-builds.adoc[subfolder package.mill files] into JVM classfiles.
Mill build files are xref:depth/why-scala.adoc[written in Scala], so this is done
using the normal Mill Scala compilation toolchain (`mill.scalalib.ScalaModule`), with
some minor pre-processing to turn `.mill` files into valid `.scala` files.

Compilation of your build is _global_ but _incremental_: running any `./mill` command
requires that you compile all `build.mill` and `package.mill` files in your entire
project, which can take some time the first time you run a `./mill` command in a project.
However, once that is done, updates to any `.mill` file are re-compiled incrementally,
such that updates can happen relatively quickly even in large projects.

After compilation, the `.mill` files are converted into JVM classfiles as shown below:

[source,txt]
----
bar/
    package.class
foo/
    package.class
build.class
----


These classfiles are dynamically loaded into the Mill process and instantiated into
a concrete Mill `RootModule` object, which is then used in the subsequent tasks below:

=== Resolution

Resolution converts the Mill xref:cli/query-syntax.adoc[task selector] ``_.assembly`` given from the command line
into a list of xref:fundamentals/tasks.adoc[Task] objects. This explores the `build` and `package`
files generated in the <<Compilation>> step above, instantiates the xref:fundamentals/modules.adoc[Modules]
and xref:fundamentals/tasks.adoc[Tasks] as necessary, and returns a list of the final tasks that
were selected by selector:

[graphviz]
....
digraph G {
  node [shape=box width=0 height=0 style=filled fillcolor=white]
  bgcolor=transparent
  newrank=true;
  build -> foo -> "foo.assembly"
  build -> bar -> "bar.assembly"
}
....

Mill starts from the root module `build` instantiated after <<Compilation>>, and uses
Java reflection to walk the tree of modules and tasks to find the tasks that match
your given selector.

Task and module resolution is _lazy_, so only modules that are required by the given
selector `_.assembly` are instantiated. This can help keep task resolution fast even
when working within a large codebase by avoiding instantiation of modules that are
unrelated to the selector you are running.


=== Planning

Planning is the step of turning the tasks selected during <<Resolution>> into a full
build graph that includes all transitive upstream dependencies. This is done by
traversing the graph of task dependencies, and generates a (simplified) task graph
as shown below:

[graphviz]
....
digraph G {
  rankdir=LR
  node [shape=box width=0 height=0 style=filled fillcolor=white]
  bgcolor=transparent
  newrank=true;
  subgraph cluster_0 {
    style=dashed
    node [shape=box width=0 height=0 style=filled fillcolor=white]
    label = "foo";

    "foo.sources" -> "foo.compile" -> "foo.classPath" -> "foo.assembly"
    "foo.resources" -> "foo.assembly"
    "foo.classPath"
  }
  subgraph cluster_1 {
    style=dashed
    node [shape=box width=0 height=0 style=filled fillcolor=white]
    label = "bar";


    "bar.sources" -> "bar.compile" -> "bar.classPath" -> "bar.assembly"

    "bar.sources" -> "bar.lineCount" -> "bar.resources" -> "bar.assembly"
  }
  "foo.classPath" -> "bar.compile" [constraint=false]
  "foo.classPath" -> "bar.classPath"
}
....

In this graph, we can see that even though <<Resolution>> only selected `foo.assembly`
and `bar.assembly`, their upstream task graph requires tasks such as `foo.compile`,
`bar.compile`, as well as our custom task `bar.lineCount` and our override of `bar.resources`.


You can use xref:cli/builtin-commands.adoc#_plan[./mill plan] to see what tasks would be
planned out for a given selector, or run xref:cli/builtin-commands.adoc#_visualizeplan[./mill visualizePlan]
or look at xref:fundamentals/out-dir.adoc#_mill_dependency_tree_json[mill-dependency-tree.json] after
evaluation to see how how these upstream tasks are depended upon by the tasks you selected.

=== Execution

The last phase is execution. Execution depends not only on the tasks you selected at the
command line, and those discovered during <<Resolution>>, but also what input files changed
on disk. Tasks that were not affected by input
changes may have their value loaded from cache (if already evaluated earlier) or skipped entirely
(e.g. due to xref:large/selective-execution.adoc[Selective Execution]).

For example, a change to `foo/src/*.java` would affect the `foo.sources` task, which
would invalidate and cause evaluation of the tasks highlighted in red below:

[graphviz]
....
digraph G {
  rankdir=LR
  node [shape=box width=0 height=0 style=filled fillcolor=white]
  bgcolor=transparent
  newrank=true;
  subgraph cluster_0 {
    style=dashed
    node [shape=box width=0 height=0 style=filled fillcolor=white]
    label = "foo";

    "foo.sources" -> "foo.compile" -> "foo.classPath" -> "foo.assembly"  [color=red, penwidth=2]
    "foo.resources" -> "foo.assembly"
    "foo.classPath"
    "foo.sources" [color=red, penwidth=2]

    "foo.assembly" [color=red, penwidth=2]
    "foo.compile" [color=red, penwidth=2]
    "foo.classPath" [color=red, penwidth=2]
  }
  subgraph cluster_1 {
    style=dashed
    node [shape=box width=0 height=0 style=filled fillcolor=white]
    label = "bar";


    "bar.sources" -> "bar.compile" ->  "bar.classPath"
    "bar.classPath" -> "bar.assembly"  [color=red, penwidth=2]

    "bar.classPath" [color=red, penwidth=2]
    "bar.assembly" [color=red, penwidth=2]
    "bar.sources" -> "bar.lineCount" -> "bar.resources" -> "bar.assembly"
  }
  "foo.classPath" -> "bar.compile" [constraint=false]
  "foo.classPath" -> "bar.classPath"  [color=red, penwidth=2]
}
....

On the other hand a change to `bar/src/*.java` would affect the `bar.sources` task, which
would invalidate and cause evaluation of the tasks highlighted in red below:

[graphviz]
....
digraph G {
  rankdir=LR
  node [shape=box width=0 height=0 style=filled fillcolor=white]
  bgcolor=transparent
  newrank=true;
  subgraph cluster_0 {
    style=dashed
    node [shape=box width=0 height=0 style=filled fillcolor=white]
    label = "foo";

    "foo.sources" -> "foo.compile" -> "foo.classPath" -> "foo.assembly"
    "foo.resources" -> "foo.assembly"
    "foo.classPath"
  }
  subgraph cluster_1 {
    style=dashed
    node [shape=box width=0 height=0 style=filled fillcolor=white]
    label = "bar";

    "bar.sources" -> "bar.compile" -> "bar.classPath" -> "bar.assembly" [color=red, penwidth=2]

    "bar.sources" [color=red, penwidth=2]
    "bar.lineCount" [color=red, penwidth=2]
    "bar.resources" [color=red, penwidth=2]
    "bar.assembly" [color=red, penwidth=2]
    "bar.compile" [color=red, penwidth=2]
    "bar.classPath" [color=red, penwidth=2]
    "bar.sources" -> "bar.lineCount" -> "bar.resources" -> "bar.assembly" [color=red, penwidth=2]
  }
  "foo.classPath" -> "bar.compile" [constraint=false]
  "foo.classPath" -> "bar.classPath"
}
....

In the example changing `bar/src/*.java`, Mill may also take the opportunity to parallelize
things:

- `bar.compile` and `bar.classPath` can run on a separate thread from `bar.lineCount` and `bar.resources`

- `bar.assembly` must wait for both `bar.classPath` and `bar.resources` to complete before proceeding.

This parallelization is automatically done by Mill, and requires no effort from the user to enable.
The exact parallelism may depend on the number of CPU cores available and exactly when each task
starts and how long it takes to run, but Mill will generally parallelize things where possible
to minimize the time taken to execute your tasks.

Some other things to note:

- Tasks have their metadata cached to xref:fundamentals/out-dir.adoc#_task_json[<task>.json] files
  in the xref:fundamentals/out-dir.adoc[out/ folder], with any files created by the task are cached
  in xref:fundamentals/out-dir.adoc#_task_dest[<task>.dest/] folders. These file paths are all
  automatically assigned by Mill.

- Mill treats builtin tasks (e.g. `compile`) and user-defined (e.g. `lineCount`) exactly the same:
  both get automatically cached and parallelized where possible.
  This happens without the task author needing to do anything to enable caching or parallelization

- Mill evaluation does not care about the _module_ structure of `foo` and `bar`. Mill modules are
  simply a way to define and re-use parts of the task graph, but it is the task graph that matters
  during evaluation

== Bootstrapping

One part of the Mill evaluation model that is skimmed over above is what happens before
*Compilation*: how does Mill actually get everything necessary to compile your `build.mill`
and `package.mill` files? This is called bootstrapping, and proceeds roughly in the following phases:

1. Mill's xref:cli/installation-ide.adoc#_bootstrap_scripts[bootstrap script] first checks
   if the right version of Mill is already present, and if not it downloads the launcher executable
   (either xref:cli/installation-ide.adoc#_mill_native_and_jvm_executables[a JVM assembly jar or
   Graal native image]) to `~/.cache/mill/download`

2. The Mill launcher resolves and downloads dependencies for the Mill daemon using Coursier, and
   spawns the daemon sub-process if necessary.

3. The Mill daemon instantiates an in-memory `MillBuildRootModule.BootstrapModule`,
   which is a hard-coded `build.mill` used for bootstrapping Mill

4. If there is a xref:extending/meta-build.adoc[meta-build] present `mill-build/build.mill`, Mill processes that
   first and uses the `MillBuildRootModule` returned for the next steps.
   Otherwise it uses the `MillBuildRootModule.BootstrapModule` directly

5. Mill evaluates the `MillBuildRootModule` to parse the `build.mill`, generate
   a list of `mvnDeps` as well as appropriately wrapped Scala code that we can
   compile, and compiles it to classfiles (<<Compilation>> above)

For most users, you do not need to care about the details of the Mill bootstrapping
process, except to know that you only need a JVM installed to begin with and
Mill will download everything necessary from the standard Maven Central package repository
starting from just the bootstrap script (available as `./mill` for Linux/Mac and `./mill.bat`
for Windows). The documentation for xref:extending/meta-build.adoc[The Mill Meta Build]
goes into more detail of how you can configure and make use of it.

== Consequences of the Mill Execution Model

This four-phase evaluation model has consequences for how you structure your
build. For example:

1. You can have arbitrary code outside of ``Task``s that helps
   set up your task graph and module hierarchy, e.g. computing what keys exist
   in a `Cross` module, or specifying your `def moduleDeps`. This code runs
   during <<Resolution>>

2. You can have arbitrary code inside of ``Task``s, to perform your build
   actions. This code runs during <<Execution>>

3. *But* your code inside of ``Task``s cannot influence the shape of the task
   graph or module hierarchy, as all <<Resolution>> logic happens first
   *before* any <<Execution>> of the ``Task``s bodies.

This should not be a problem for most builds, but it is something to be aware
of. In general, we have found that having "two places" to put code - outside of
``Task``s to run during <<Planning>> or inside of ``Task``s to run during
<<Execution>> - is generally enough flexibility for most use cases. You
can generally just write "direct style" business logic you need - in the example
above counting the lints in `allSourceFiles` - and Mill handles all the caching,
invalidation, and parallelism for you without any additional work.

The hard boundary between these two phases is what lets users easily query
and visualize their module hierarchy and task graph without running them: using
xref:cli/builtin-commands.adoc#_inspect[inspect], xref:cli/builtin-commands.adoc#_plan[plan],
xref:cli/builtin-commands.adoc#_visualize[visualize], etc.. This helps keep your
Mill build discoverable even as the `build.mill` codebase grows.
