= Caching in Mill

Mill has a multi-layered approach to caching: every step in the
xref:depth/evaluation-model.adoc[Mill Evaluation Model] is cached if possible,
re-using prior results rather than computing them from scratch. This helps ensure
the overall workflow remains fast even for large projects.

## Caching Per Phase

This section will discuss the caching that Mill performs in each phase of
xref:depth/evaluation-model.adoc[Mill's Evaluation Model]:

### Compilation

* If there were changes in `build.mill` or `package.mill` files, compilation is done
  incrementally using the Scala incremental compiler https://github.com/sbt/zinc[Zinc].
  Typically, this limits compilation to the `.mill` files you changed, though the
  exact amount of caching and reuse that occurs may vary depending on the nature
  of the code change.

* If no `.mill` files were changed, this phase is skipped entirely.

### Resolution

* In the common case where the `build.mill` and `package.mill` files were not
  changed - and re-compilation of the `.mill` files did not occur - any `Module`
  objects instantiated during previous resolutions are kept around and re-used.

* If re-compilation of `.mill` files did occur due to a code change, then
  all `Module` objects are discarded along with their enclosing classloader,
  and re-instantiated using the latest code

### Planning

* Planning is relatively quick most of the time, and is not currently cached.

### Execution

* Mill ``Task``s are evaluated in dependency order

* Default cached xref:fundamentals/tasks.adoc#_tasks[Task]s only re-evaluate if their
  input ``Task``s have their value change.

* xref:fundamentals/tasks.adoc#_persistent_tasks[Persistent Tasks]s preserve the `Task.dest`
  folder on disk between runs, allowing for finer-grained caching than Mill's default task-by-task
  caching and invalidation. For example, this is used to support
  xref:#_fine_grained_incremental_compilation[fine-grained incremental compilation] in Java, Scala,
  and Kotlin

* xref:fundamentals/tasks.adoc#_workers[Worker]s are kept in-memory between runs where possible, and only
  invalidated if their input ``Task``s change as well.

* ``Task``s in general are invalidated if the code they depend on changes,
  at a method-level granularity via callgraph reachability analysis. See
  https://github.com/com-lihaoyi/mill/pull/2417[#2417] for more details

### Bootstrapping

* Mill's bootstrapping process essentially involves running the four phases above, but
  for the meta-build rather than the primary build. All the caching techniques described
  above apply the same for Mill's bootstrap builds as they do for the primary build.

## Debugging Mill Caching Issues

To dig into how Mill's execution caching, you can look at the following files:

* xref:fundamentals/out-dir.adoc#_mill_profile_json[mill-profile.json]: this file
is generated during every Mill evaluation, and contains a `"cached": boolean` flag
for each task indicating whether or not that task was cached.

* xref:fundamentals/out-dir.adoc#_mill_invalidation_tree_json[mill-invalidation-tree.json]:
this file groups together the un-cached files in a tree structure according to their
task dependencies. It is useful to find the "root" uncached tasks, which are the cause
of downstream tasks having their caches invalidated. For tasks invalidated due to
__code changes in the `build.mill` or `package.mill` files__, the tree also shows
the method-level code changes with `def` and `call` nodes indicating how code changes
propagate through the build definition to reach the invalidated tasks.

## Consequences of Caching in Mill

This approach to caching does assume a certain programming style inside your
Mill build:

- Mill may-or-may-not instantiate the modules in your `build.mill` the first time
  you run something (due to laziness)

- Mill may-or-may-not *re*-instantiate the modules in your `build.mill` in subsequent runs
  (due to caching)

- Mill may-or-may-not re-execute any particular task depending on caching,
  but your code needs to work either way.

- Execution of any task may-or-may-not happen in parallel with other unrelated
  tasks, and may happen in arbitrary order

Your build code needs to work regardless of which order they are executed in.
However, for code written in a typical Scala style (which tends to avoid side effects),
and limits filesystem operations to the `Task.dest` folder, this is not a problem at all.

One thing to note is for code that runs during *Resolution*: any reading of
external mutable state needs to be wrapped in an `mill.api.BuildCtx.watchValue{...}`
wrapper. This ensures that Mill knows where these external reads are, so that
it can check if their value changed and if so re-instantiate `RootModule` with
the new value.

## Fine-Grained Incremental Compilation

Incremental compilation for the various JVM languages is implemented via:

- https://github.com/sbt/zinc[Zinc] for incrementally compiling Java and Scala
- https://kotlinlang.org/docs/build-tools-api.html#enable-incremental-compilation-with-in-process-strategy[Kotlin's Build Tool API]
  for incrementally compiling Kotlin

These are used within xref:fundamentals/tasks.adoc#_persistent_tasks[persistent tasks], to allow
the incremental compilers to use the previous runs output cache and analysis files for each
subsequent run.

Like any other persistent task, incremental compilation relies on the correctness of their
implementation to ensure that their final output converges to the desired state regardless
of the sequence of inputs and changes that happened before. In practice, there are occasionally
bugs in the incremental compilers that may cause the caches to get into a bad state. In such cases,
running `./mill clean foo` on the module in question will wipe out these caches and let the
compilation start afresh.