How Nix provides GHC with packages

Recently I am making myself familiar with Nix. I personally think that the actual Nix manual is a little bit overwhelming to start with. Instead I recommend the Nix pills as an entry point to the Nix world.

My current interest circles around how to use Nix for Haskell development. Nix’s manual packages devotes an entire section to the Haskell Infrastructure in Nix. In this blog post I want to investigate how Nix actually builds Haskell targets, i.e. what drops out of all the *.nix files.

Nixpkgs and Haskell

All the Haskell related stuff of nixpkgs can be found under <nixpkgs>/pkgs/development/haskell-modules, where <nixpkgs> is the path to your local nixpkgs directory. You can find the path via nix repl:

$ nix repl
nix-repl> "${<nixpkgs>}"
"/nix/store/91lz3yamb6hr31hpjdkpbxjv806s651b-nixpkgs"

There’s actually quite a bit of stuff in there:

$ ls /nix/store/91lz3yamb6hr31hpjdkpbxjv806s651b-nixpkgs/pkgs/development/haskell-modules
configuration-common.nix     configuration-hackage2nix.yaml  hackage-packages.nix     non-hackage-packages.nix
configuration-ghc-8.2.x.nix  configuration-nix.nix           hoogle-local-wrapper.sh  patches
configuration-ghc-8.4.x.nix  configuration-tensorflow.nix    hoogle.nix               stack-hook.sh
configuration-ghc-8.6.x.nix  default.nix                     initial-packages.nix     with-packages-wrapper.nix
configuration-ghc-head.nix   generic-builder.nix             lib.nix
configuration-ghcjs.nix      generic-stack-builder.nix       make-package-set.nix

Reading into all these files is quite a mouthful! But the nice thing is we don’t need to do that! The whole purpose of a Nix expression is to eventually produce a derivation that will be built. As an example the following line will drop us into a new shell environment with a ghc executable and the lens and mtl libraries installed:

$ nix-shell --pure -p "haskellPackages.ghcWithPackages (pkgs: with pkgs; [lens mtl])"
[nix-shell]$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 8.6.3
[nix-shell]$ ghc-pkg list
/nix/store/llplrrsi5dqbvsl9j6kfhadfvf94yg4v-ghc-8.6.3-with-packages/lib/ghc-8.6.3/package.conf.d
    ...
    kan-extensions-5.2
    lens-4.17
    libiserv-8.6.3
    mtl-2.2.2
    parallel-3.2.2.0
    ...

We can use nix repl to tell us where the derivation can be found:

$ nix repl
Welcome to Nix version 2.2.1. Type :? for help.

nix-repl> d = haskellPackages.ghcWithPackages (pkgs: with pkgs; [lens mtl])
nix-repl> d
«derivation /nix/store/gnry81f0fx07jn8yqwsa53hhwz9m0y8h-ghc-8.6.3-with-packages.drv»

Now we know where to find the derivation and we can print it to a file.

$ nix show-derivation /nix/store/gnry81f0fx07jn8yqwsa53hhwz9m0y8h-ghc-8.6.3-with-packages.drv | python -m json.tool > ghc-8.6.3-with-packages.drv

Note: I used the Python JSON tools as described in a previous post.

This is the content of the derivation file:

{
    "/nix/store/gnry81f0fx07jn8yqwsa53hhwz9m0y8h-ghc-8.6.3-with-packages.drv": {
        "args": [
            "-e",
            "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
        ],
        "builder": "/nix/store/53wi068kjrqfr2j0hzcxhbw2xaa990jr-bash-4.4-p23/bin/bash",
        "env": {
            "allowSubstitutes": "",
            "buildCommand": "mkdir -p $out\nfor i in $paths; do\n  /nix/store/n9x3d54nkv819ifbxla3aqj5bxzrvrkz-lndir-1.0.3/bin/lndir -silent $i $out\ndone\n. /nix/store/l1d26c47xx8z46cgp9spy2x868bkl0r9-hook/nix-support/setup-hook\n\n# wrap compiler executables with correct env variables\n\nfor prg in ghc ghci ghc-8.6.3 ghci-8.6.3; do\n  if [[ -x \"/nix/store/7874h075nf8yikvr47642xqrwqwyv99s-ghc-8.6.3/bin/$prg\" ]]; then\n    rm -f $out/bin/$prg\n    makeWrapper /nix/store/7874h075nf8yikvr47642xqrwqwyv99s-ghc-8.6.3/bin/$prg $out/bin/$prg                           \\\n      --add-flags '\"-B$NIX_GHC_LIBDIR\"'                   \\\n      --set \"NIX_GHC\"        \"$out/bin/ghc\"     \\\n      --set \"NIX_GHCPKG\"     \"$out/bin/ghc-pkg\" \\\n      --set \"NIX_GHC_DOCDIR\" \"$out/share/doc/ghc/html\"                  \\\n      --set \"NIX_GHC_LIBDIR\" \"$out/lib/ghc-8.6.3\"                  \\\n       \\\n      \n  fi\ndone\n\nfor prg in runghc runhaskell; do\n  if [[ -x \"/nix/store/7874h075nf8yikvr47642xqrwqwyv99s-ghc-8.6.3/bin/$prg\" ]]; then\n    rm -f $out/bin/$prg\n    makeWrapper /nix/store/7874h075nf8yikvr47642xqrwqwyv99s-ghc-8.6.3/bin/$prg $out/bin/$prg                           \\\n      --add-flags \"-f $out/bin/ghc\"                           \\\n      --set \"NIX_GHC\"        \"$out/bin/ghc\"     \\\n      --set \"NIX_GHCPKG\"     \"$out/bin/ghc-pkg\" \\\n      --set \"NIX_GHC_DOCDIR\" \"$out/share/doc/ghc/html\"                  \\\n      --set \"NIX_GHC_LIBDIR\" \"$out/lib/ghc-8.6.3\"\n  fi\ndone\n\nfor prg in ghc-pkg ghc-pkg-8.6.3; do\n  if [[ -x \"/nix/store/7874h075nf8yikvr47642xqrwqwyv99s-ghc-8.6.3/bin/$prg\" ]]; then\n    rm -f $out/bin/$prg\n    makeWrapper /nix/store/7874h075nf8yikvr47642xqrwqwyv99s-ghc-8.6.3/bin/$prg $out/bin/$prg --add-flags \"--global-package-db=$out/lib/ghc-8.6.3/package.conf.d\"\n  fi\ndone\n\n# haddock was referring to the base ghc, https://github.com/NixOS/nixpkgs/issues/36976\nif [[ -x \"/nix/store/7874h075nf8yikvr47642xqrwqwyv99s-ghc-8.6.3/bin/haddock\" ]]; then\n  rm -f $out/bin/haddock\n  makeWrapper /nix/store/7874h075nf8yikvr47642xqrwqwyv99s-ghc-8.6.3/bin/haddock $out/bin/haddock    \\\n    --add-flags '\"-B$NIX_GHC_LIBDIR\"'  \\\n    --set \"NIX_GHC_LIBDIR\" \"$out/lib/ghc-8.6.3\"\nfi\n\n$out/bin/ghc-pkg recache\n\n$out/bin/ghc-pkg check\n\n",
            "buildInputs": "",
            "builder": "/nix/store/53wi068kjrqfr2j0hzcxhbw2xaa990jr-bash-4.4-p23/bin/bash",
            "configureFlags": "",
            "depsBuildBuild": "",
            "depsBuildBuildPropagated": "",
            "depsBuildTarget": "",
            "depsBuildTargetPropagated": "",
            "depsHostHost": "",
            "depsHostHostPropagated": "",
            "depsTargetTarget": "",
            "depsTargetTargetPropagated": "",
            "doCheck": "",
            "doInstallCheck": "",
            "name": "ghc-8.6.3-with-packages",
            "nativeBuildInputs": "",
            "out": "/nix/store/llplrrsi5dqbvsl9j6kfhadfvf94yg4v-ghc-8.6.3-with-packages",
            "outputs": "out",
            "passAsFile": "buildCommand",
            "paths": "/nix/store/qx8c8021dwyrlvzns78xxlf9mg48pj8p-primitive-0.6.4.0 /nix/store/51c1jlizakl193m0k0mx7yxxkgb2kvx5-vector-0.12.0.2 /nix/store/ks3yg9gs1p09jk171l9qlcvmb8kbh9fz-reflection-2.1.4 /nix/store/gv9vshcsnjsickqp7zpk4q6bhsp78hha-parallel-3.2.2.0 /nix/store/mn4ryfpyvsh9mjydy9a2yq3783rrrlnd-invariant-0.5.1 /nix/store/s9hnj889y3bhg5h9x472g259b6agi856-void-0.7.2 /nix/store/lx861rrh3xrdj8hhvga2m0jqn3949bjl-adjunctions-4.4 /nix/store/15g3rh81f3jaw24i7l9fly520l32j7cf-kan-extensions-5.2 /nix/store/499cfyhsssrjxrf400j5p8nm6fa0g0g8-transformers-base-0.4.5.2 /nix/store/8v4wzwqav56x2psq6n575yb6jjlg3n46-unordered-containers-0.2.9.0 /nix/store/fk8wbkfj2mkfgp613h7r586p3q3fqi5f-hashable-1.2.7.0 /nix/store/wbhbvs9gqx7mds1ssn2j36pbrr6jyvld-semigroupoids-5.3.2 /nix/store/yqx3rh1fl362p364xmqfqzc3ycqi72wx-profunctors-5.3 /nix/store/dvjkbcvv8g3cdigb12gpyqaf6fp1b0bg-free-5.1 /nix/store/n4a04c8qwqnjf8bgbwb6bh97ridwiwrs-exceptions-0.10.0 /nix/store/600xynkv215dw7biwxywz5cqy6zir081-call-stack-0.1.0 /nix/store/i5maa8zqi8j9qr9gg9g9c3cwwpmzyrfr-th-abstraction-0.2.10.0 /nix/store/ydm0mzgjfaqbgx2ghx107d1xawpy25lv-transformers-compat-0.6.2 /nix/store/qx31v7ccah2a6zvkcnmnk2162mhsd2px-semigroups-0.18.5 /nix/store/sphwxjz334v98m91fva59rnpbykws59w-tagged-0.8.6 /nix/store/sfjxk3hxc9c6yyjzpam28xhzjkmaayj6-distributive-0.6 /nix/store/ghh99q758idi878cfnp9wwp6cn5l33d7-StateVar-1.1.1.1 /nix/store/flyj4sy2c689np1r9r4d0fc4kaz3h9bm-contravariant-1.5 /nix/store/mnkijbnig4yaz7jwnsf0qmw59pjhcsxw-comonad-5.0.4 /nix/store/b7cg4xzcl3wz6j7a9wxpq1rajr7k6537-bifunctors-5.5.3 /nix/store/c9jzyrl1dcrk7hs24z5246jbvlsfggax-base-orphans-0.8 /nix/store/zxk67vza4pa0drq79yv900nvq6nsjhhs-lens-4.17 /nix/store/7874h075nf8yikvr47642xqrwqwyv99s-ghc-8.6.3",
            "preferLocalBuild": "1",
            "propagatedBuildInputs": "",
            "propagatedNativeBuildInputs": "",
            "stdenv": "/nix/store/3lg181dr5pydwn8mqz9xm4w2jc8liraq-stdenv-linux",
            "strictDeps": "",
            "system": "x86_64-linux"
        },
        "inputDrvs": {
            "/nix/store/12gyh3c7d0ddgiv3c0xwsy755z12ihlf-hook.drv": [
                "out"
            ],
            "/nix/store/381zfimsd8hhzgshwhafl37izi0h1iqf-void-0.7.2.drv": [
                "out"
            ],
            "/nix/store/3w9h06fgk6pw88i47gp1qilgvlczyx1h-primitive-0.6.4.0.drv": [
                "out"
            ],
            "/nix/store/5g5vwr3fvhvxbifsfwxlkmxzk5y8zbic-bifunctors-5.5.3.drv": [
                "out"
            ],
            "/nix/store/5h786j7n208szqa48y165i4nzm35h08z-comonad-5.0.4.drv": [
                "out"
            ],
            "/nix/store/729sc8mbqcxxvhg5yy76dhx3yil21bkc-base-orphans-0.8.drv": [
                "out"
            ],
            "/nix/store/84by0c5a0d00w3frd8wvq2zfs8dgl6l7-profunctors-5.3.drv": [
                "out"
            ],
            "/nix/store/9mmf07zi8vx6a06jzmdkbvxmwwjk0y4g-unordered-containers-0.2.9.0.drv": [
                "out"
            ],
            "/nix/store/bjzqjxzha16pjlb9kkns9dv77w649bsv-free-5.1.drv": [
                "out"
            ],
            "/nix/store/c57fqmdvh04748dp4hi4mzvr9mn49vkb-th-abstraction-0.2.10.0.drv": [
                "out"
            ],
            "/nix/store/ca55ghbsagacvn3irdsa8v9ry5hcdli0-stdenv-linux.drv": [
                "out"
            ],
            "/nix/store/gh6hkhyl7rnkpnilwcwjwiq93yk8s8jc-lndir-1.0.3.drv": [
                "out"
            ],
            "/nix/store/giy8bq0a06qw9yz4ax3s83630vww0bps-kan-extensions-5.2.drv": [
                "out"
            ],
            "/nix/store/hvy5f310d4jrpm16hdrx4dzn23wlyxdf-adjunctions-4.4.drv": [
                "out"
            ],
            "/nix/store/hyzca3hf9g6vzy7x7amd44iyq4jfkykn-lens-4.17.drv": [
                "out"
            ],
            "/nix/store/i93xj3xpg7j89dp679537sw6aw9i87hw-semigroupoids-5.3.2.drv": [
                "out"
            ],
            "/nix/store/irafg40g0nn60k6wj1135wlfr5c49pc3-ghc-8.6.3.drv": [
                "out"
            ],
            "/nix/store/j2ga18ls4xyb78mbi5whzj5vli23vxw6-parallel-3.2.2.0.drv": [
                "out"
            ],
            "/nix/store/jhah0y9qrg0kl0h14da137a451wz8ssm-transformers-base-0.4.5.2.drv": [
                "out"
            ],
            "/nix/store/ji9izf2qxwgwzrv016gqv5d70bhq0gjp-reflection-2.1.4.drv": [
                "out"
            ],
            "/nix/store/jjbkc7drlkvim3vyrdxpa79qnl0pbsw5-exceptions-0.10.0.drv": [
                "out"
            ],
            "/nix/store/m1a1gqar6jmha09bp611fcp4ypn59x87-StateVar-1.1.1.1.drv": [
                "out"
            ],
            "/nix/store/mldrp2p7cw2nfk6im7blhhdnyj0pvjli-call-stack-0.1.0.drv": [
                "out"
            ],
            "/nix/store/p5r6akkq1qyhbfp9m47557hm9dl00gk9-tagged-0.8.6.drv": [
                "out"
            ],
            "/nix/store/pww1j6bpnblfcs7h8nkdgnfs5npirq6m-contravariant-1.5.drv": [
                "out"
            ],
            "/nix/store/rc2c7xlak579xl8564rq396qjdxrs1cw-transformers-compat-0.6.2.drv": [
                "out"
            ],
            "/nix/store/rdhj9rf6pvgv7rvqlqhj4q78gdck5pbl-invariant-0.5.1.drv": [
                "out"
            ],
            "/nix/store/xjjmsdz36qg9psawr72a5i2nghrr2a3w-semigroups-0.18.5.drv": [
                "out"
            ],
            "/nix/store/y5raa3zj6b2xh6zjv07ajfk7g3rhn8ik-distributive-0.6.drv": [
                "out"
            ],
            "/nix/store/yq18hqx69mgi6pff2k5127c1jvwl5xaf-hashable-1.2.7.0.drv": [
                "out"
            ],
            "/nix/store/zaxp1jn5z4mb75r06h0yi25av9rhksds-vector-0.12.0.2.drv": [
                "out"
            ],
            "/nix/store/zvy7mbpxqlplqpflqn5xk9szx25s4mhg-bash-4.4-p23.drv": [
                "out"
            ]
        },
        "inputSrcs": [
            "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
        ],
        "outputs": {
            "out": {
                "path": "/nix/store/llplrrsi5dqbvsl9j6kfhadfvf94yg4v-ghc-8.6.3-with-packages"
            }
        },
        "platform": "x86_64-linux"
    }
}

This is actually pretty straight forward to read. We can ignore the “inputDrvs” attribute, as it is mostly relevant for Nix, not to understand what’s going on. We want to look at “builder”, “args” and “env”. Nix will call the builder with the given arguments from “args” and inject the variables from “env” into the environment of the call to builder. Ignoring most of the environment variables and the exact paths this will more or less result in the following call:

$ buildCommand='mkdir -p...' paths='...' bash -e default-builder.sh

The script default_builder.sh is just a wrapper around the generic build of Nix’s stdenv generic build. As the name suggests it is a very generic build script capable to build almost everything. It provides a lot of functionality, e.g. running different installation phases, and is controlled via hooks that must be defined in the environment. Here’s what’s in default_builder.sh:

source $stdenv/setup
genericBuild

The setup script defines the genericBuild function. In our case it will simply execute the script defined by the buildCommand attribute:

...
genericBuild() {
    if [ -f "${buildCommandPath:-}" ]; then
        local oldOpts="$(shopt -po nounset)"
        set +u
        source "$buildCommandPath"
        eval "$oldOpts"
        return
    fi
    if [ -n "${buildCommand:-}" ]; then
        local oldOpts="$(shopt -po nounset)"
        set +u
        eval "$buildCommand"
        eval "$oldOpts"
        return
    fi
...

The buildCommand attribute is written in one line and thus hard to read. I translated the literal newlines into actual newlines by copying the line in a Vim buffer and then issuing the command :s/\\n/\r/g. Here is the result:

mkdir -p $out
for i in $paths; do
  /nix/store/n9x3d54nkv819ifbxla3aqj5bxzrvrkz-lndir-1.0.3/bin/lndir -silent $i $out
done
. /nix/store/l1d26c47xx8z46cgp9spy2x868bkl0r9-hook/nix-support/setup-hook

# wrap compiler executables with correct env variables

for prg in ghc ghci ghc-8.6.3 ghci-8.6.3; do
  if [[ -x \"/nix/store/7874h075nf8yikvr47642xqrwqwyv99s-ghc-8.6.3/bin/$prg\" ]]; then
    rm -f $out/bin/$prg
    makeWrapper /nix/store/7874h075nf8yikvr47642xqrwqwyv99s-ghc-8.6.3/bin/$prg $out/bin/$prg                           \\
      --add-flags '\"-B$NIX_GHC_LIBDIR\"'                   \\
      --set \"NIX_GHC\"        \"$out/bin/ghc\"     \\
      --set \"NIX_GHCPKG\"     \"$out/bin/ghc-pkg\" \\
      --set \"NIX_GHC_DOCDIR\" \"$out/share/doc/ghc/html\"                  \\
      --set \"NIX_GHC_LIBDIR\" \"$out/lib/ghc-8.6.3\"                  \\
       \\
      
  fi
done

for prg in runghc runhaskell; do
  if [[ -x \"/nix/store/7874h075nf8yikvr47642xqrwqwyv99s-ghc-8.6.3/bin/$prg\" ]]; then
    rm -f $out/bin/$prg
    makeWrapper /nix/store/7874h075nf8yikvr47642xqrwqwyv99s-ghc-8.6.3/bin/$prg $out/bin/$prg                           \\
      --add-flags \"-f $out/bin/ghc\"                           \\
      --set \"NIX_GHC\"        \"$out/bin/ghc\"     \\
      --set \"NIX_GHCPKG\"     \"$out/bin/ghc-pkg\" \\
      --set \"NIX_GHC_DOCDIR\" \"$out/share/doc/ghc/html\"                  \\
      --set \"NIX_GHC_LIBDIR\" \"$out/lib/ghc-8.6.3\"
  fi
done

for prg in ghc-pkg ghc-pkg-8.6.3; do
  if [[ -x \"/nix/store/7874h075nf8yikvr47642xqrwqwyv99s-ghc-8.6.3/bin/$prg\" ]]; then
    rm -f $out/bin/$prg
    makeWrapper /nix/store/7874h075nf8yikvr47642xqrwqwyv99s-ghc-8.6.3/bin/$prg $out/bin/$prg --add-flags \"--global-package-db=$out/lib/ghc-8.6.3/package.conf.d\"
  fi
done

# haddock was referring to the base ghc, https://github.com/NixOS/nixpkgs/issues/36976
if [[ -x \"/nix/store/7874h075nf8yikvr47642xqrwqwyv99s-ghc-8.6.3/bin/haddock\" ]]; then
  rm -f $out/bin/haddock
  makeWrapper /nix/store/7874h075nf8yikvr47642xqrwqwyv99s-ghc-8.6.3/bin/haddock $out/bin/haddock    \\
    --add-flags '\"-B$NIX_GHC_LIBDIR\"'  \\
    --set \"NIX_GHC_LIBDIR\" \"$out/lib/ghc-8.6.3\"
fi

$out/bin/ghc-pkg recache

$out/bin/ghc-pkg check

All this script does is sym-linking every library in the output directory of the derivation in the Nix store and then create small wrapper scripts for all the necessary executables (ghc, runghc, etc.). Finally the packages are recached into the respective installation of GHC.

Instead of looking at the makeWrapper function it is easier to just have a look at the result. Here is the wrapper script for ghc:

$ cat /nix/store/llplrrsi5dqbvsl9j6kfhadfvf94yg4v-ghc-8.6.3-with-packages/bin/ghc
#! /nix/store/53wi068kjrqfr2j0hzcxhbw2xaa990jr-bash-4.4-p23/bin/bash -e
export NIX_GHC='/nix/store/llplrrsi5dqbvsl9j6kfhadfvf94yg4v-ghc-8.6.3-with-packages/bin/ghc'
export NIX_GHCPKG='/nix/store/llplrrsi5dqbvsl9j6kfhadfvf94yg4v-ghc-8.6.3-with-packages/bin/ghc-pkg'
export NIX_GHC_DOCDIR='/nix/store/llplrrsi5dqbvsl9j6kfhadfvf94yg4v-ghc-8.6.3-with-packages/share/doc/ghc/html'
export NIX_GHC_LIBDIR='/nix/store/llplrrsi5dqbvsl9j6kfhadfvf94yg4v-ghc-8.6.3-with-packages/lib/ghc-8.6.3'
exec "/nix/store/7874h075nf8yikvr47642xqrwqwyv99s-ghc-8.6.3/bin/ghc"  "-B$NIX_GHC_LIBDIR" "${extraFlagsArray[@]}" "$@"

So we call the specified version of ghc, ghc-8.6.3 in this case, and tell it where to find its libraries. As it is typical for Nix the ghc executable is located in its respective location in the store and the environment is set up so that everything works together.