Impression of using NixOS for 2.5 months

For a Berlin expat, between the last Christmas and the new year is always the best time for trying out a small project that I otherwise never make time to do it. This time I have built an acrylic lamp and installed NixOS on my laptop (Lenovo E495 - AMD Ryzen 7). Since then, I have been using it and the experience was quite satisfactory. I would like to share here what I have tried and what impression I have gotten as a 2.5-month-old baby user.

For reference, my NixOS configurations are here.

Shortly about my General Experiences of Different Operating Systems:

I started using Unix-like variants in 2004 - the first one I tried was FreeBSD and Red Hat. Since 2006, I am using Linux-based systems only. I have used U/Ku/Xu-buntu (~ 2011) and switched to Mint (~2014), to Gentoo (~2017), and to Debian (~2020). At work, I have been using mostly Ubuntu, mainly due to the easier integration of the GPU acceleration with CUDA.

Pros

1. Stress-free Installation

I somehow expected that the installation of NixOS would be as complicated as Gentoo or Arch but it was as easy and fast as Debian. I wanted to install it with a LUKS setup for the disk and specifying spaces for /tmp with TMPFS and /var. This is very simple to do in Debian stable but not in the current Ubuntu stable. It worked out all smooth and quick and I realized that I don’t need to reserve a few days for installing NixOS. If you are interested, you can find my bash script for partitioning with LUKS here.

In general I felt that to install Gentoo I need to understand and configure every single detail. I can use most Debian-based distros out of the box and be agnostic about a lot of detailed setups. But it gets often too complicated if I want to customize the configurations here and there. With NixOS I was able to decide easily where on this spectrum I want to be throughout different stages of the installation.

2. Up-to-date Rolling Updates and Package Build and Managements directly from a Github Repository

• Installing graphic cards, touchpads, sound devices and any other hardware-related firmwares in NixOS was much easier for me than in Ubuntu or Debian. Especially for recent AMD-based laptops, it was quite a struggle with the current Debian stable, because its kernel only supports quite old firmwares.

With the Nix language, I could also build a package from the sources with overriding specific arguments and this package can be still managed by the Nix package manager. It happened often to me that if I wanted to build a package with a different option than what’s supported from the package manager of some distro, then I had to clone the repository and build from the sources. Then it can be quite a problem to keep them up to date and I often ended up using a deprecated version for long without realizing. This problem is more or less resolved with the Nix package manager. It is also possible to update a package directly from a Github repository with an usual update command by specifying it in the Nix configuration.

• Installing the AMD GPU support rocm in Debian and Ubuntu was really a struggle – I tried to build it from the source but then it didn’t work for me either. But with NixOS it was a one-liner and worked without efforts.

• The examples of other applications that I benefitted by using NixOS were conky and MuseScore. Both did some major upgrades and most of the major changes were not available in the package manager of some other stable distros. Also I was happy that the versions of Haskell ghc compilers are much more up to date in the Nix package manager.

3. Centralized and Modularized Management of All rc/dotfiles and User-Specific Package Management Using Home Manager

Officially Home Manager is still under development but I like it quite a lot. I have been using a shaky self-crafted rc/dotfile manager for a few years by creating a centralized folder of those configuration files and a bash script for symbolic linking and updating them automatically. It got complicated because I was using by then 1) zsh with powerline9k, 2) i3 with polybar and rofi, 3) neovim, 4) conky and 5) some customized zsh aliases for Python developments and SSH connections. It took quite some time to translate my rc/dotfiles to be compatible this the NixOS Home Manager. Fortunately I feel that they are now better structured and not based on deprecated options or dependencies. For example, powerline9k is also deprecated now.

4. Development Environment With nix-shell

I haven’t tried this for Python mainly because I don’t want to force other people to use Nix. All of the Python projects that I am now working on are collaborative projects. I mainly used nix-shell for Haskell. I like that I should specify the dependent Nix packages for Haskell packages in the development environment. This is helpful to test the build environment and not to keep all the packages for a specific build to the whole user environment. Otherwise one can also use nix-env to install them in the user environment.

5. Community and Documentation

The community is very active and most of the documentation is quite friendly and nice. Still I have the impression that many features and developments are evolving faster than the documentation. This is probably normal and I believe that it will settle in the near future.

6. Last but not Least: Package Dependency and Rollback

The package dependency tree is implemented by building derivations first by creating *.drv files and the outputs, stored in /nix/store/ and by generating symlinks in /nix/var/nix/gcroots/ to the ones in /nix/store/. Probably this symlink-based derivations for the package manager is the most distinctive feature of the Nix package manager.

I was able to observe interesting side effects thereof: for example a package A and a package B both have the dependency to a package C but they have different version requirements and it ends up that I have two different versions of C’ and C’’ in the same system.

I observed that this system made indeed the desktop manager and the default running component of the OS much more stable; less danger of crashing the whole system due to dependency conflicts. This was the main reason that I gave up on using Gentoo at some point.

After quasi-“compiling” the configuration with the Nix language, it creates a new version, while still keeping the old version for the rollback. These old versions will be removed only if I explicitly run a garbage collection. The Home Manager takes a similar approach.

Cons

1. Compatibility

It seems to me that in general this “Nix world” is not really compatible to the world outside.

• The package maintainers have to use their own Nix package manager to create a hash to update the package. I found this somewhat unfriendly for people outside of this religion. I am currently using some less popular geometry software and I saw the maintainer explicitly mentioned that it is not supported in the Nix package manager. I am considering to upload it by myself soon because it seems to be not so complicated.

• All my user configuration files were not directly compatible with Home Manager and it took quite a while to convert them.

• As mentioned before, I cannot use nix-shell when I collaborate in a programming project with other people.

• Currently the support of *.Appimage, *.deb and *.rpm for NixOS is somewhat subtle. I was able to run *.Appimage for the note program boostnote that I am using. I was not able to run any *.deb or *.rpm files. Although they are intended only for the debian-based and RedHat-based package managers respectively, I was able to run them in Gentoo.

2. Relatively Smaller User Group compared to Debian or Ubuntu and Shorter History

The user group is relatively small (bigger than Gentoo or void Linux by now) and most of users are quite advanced so it is hard to find an answer for somewhat trivial questions.

• I had troubles configuring triple monitors with i3 in NixOS. I assume that this is probably due to an AMD-related graphic driver issue but I couldn’t find any useful information. Probably there are not many people who are using NixOS with i3 with a recent AMD laptop.

• While installing some package, I had some specific error to this package. I reported this issue quite some time ago. I haven’t gotten any replies yet.

I imagine that I could have found solutions to these problems already if the Nix community were a bigger.

In any case, I have compiled this post with nix-shell on NixOS. I’m curios how it will develop further with time. Any feedbacks on my NixOS configuration would also be appreciated.

Dynamic Sidebar or Header Activation based on the Current Page in Hakyll

Situation

Hakyll is a haskell-based static site generator that is used to generate my blog. For the migration I ported Lanyon theme designed for Jekyll to lanyon-hakyll and here I describe one of the problems that I encountered.

The problem is an extension of what is described in this blog: Hakyll, where am I?

The page that you are navigating will be linked to one in your sidebar or your header depending on your layout unless it is one of posts. It is possible to statically link each of such pages manually. But if this list of pages is dynamically generated by loading all pages in some folder (e.g. in the pages folder in lanyon-hakyll), it gets somewhat more complicated in Hakyll for the following reason.

An example based on liquid syntax in ruby for such activation will look like (the excerpt from Lanyon):

{% for node in pages_list %}
    {% if node.title != null %}
          <a class="sidebar-nav-item{% if page.url == node.url %} active{% endif %}" href="{{ node.url | absolute_url }}">{{ node.title }}</a>
    {% endif %}
{% endfor %}

A direct translation of {% if page.url == node.url %} in Hakyll is not possible, because the control flow of Hakyll $if(variable)$ does not evaluate the boolean value of variable but merely checks whether the key variable exists in the current context or not. Check out this tutorial for understanding the control flow of Hakyll templates.

Approaching the Solution

We will dynamically generate this constField having the page title as a key in the context of listField with a key pages_list. First we add a snapshot to avoid a dependency cycle in compiling the pages folder:

 match "pages/*" $ do
    ...
    pandocCompiler
    ...
    >>= saveSnapshot "page-content"

Define the context containing such listField:

sidebarCtx :: Context String -> Context String
sidebarCtx nodeCtx =
    listField "pages_list" nodeCtx (loadAllSnapshots "pages/*" "page-content") `mappend`
    defaultContext

baseNodeCtx :: Context String
baseNodeCtx =
    urlField "node-url" `mappend`
    titleField "title" `mappend`
    baseCtx

baseSidebarCtx = sidebarCtx baseNodeCtx

Add dynamically generated constField with the current page title.

--- This is not enough.
import           System.FilePath               (takeBaseName)

match "pages/*" $ do
    route $ setExtension "html"
    compile $ do
        pageName <- takeBaseName . toFilePath <$> getUnderlying
        let pageCtx = constField pageName "" `mappend`
                      baseNodeCtx
        let activeSidebarCtx = sidebarCtx pageCtx

        pandocCompiler
            >>= saveSnapshot "page-content"
            ...
            >>= loadAndApplyTemplate "templates/default.html" (activeSidebarCtx <> siteCtx)
            >>= relativizeUrls

The translation of the above html layout will be similar to:

<!-- THIS DOES NOT WORK -->
$for(pages_list)$
    $if(title)$
          <a class="sidebar-nav-item$if($title$)$ active$endif$" href="$baseurl$$node-url$">$title$</a>
    $endif$
$endfor$

As you see in the comment, this is not enough because inside of $if(...)$ syntax, you cannot evaluate the key by surrounding them with $.

Solution

We can add functionField for evaluating a key for a given context. The functionField needs a function with a type [String] -> Item String -> Compiler String.

We define the following evalCtxKey function for this purpose:

evalCtxKey :: Context String -> [String] -> Item String -> Compiler String
evalCtxKey context [key] item = (unContext context key [] item) >>= \cf ->
        case cf of
            StringField s -> return s
            _             -> error $ "Internal error: StringField expected"

Just if you need, you can also access the meta data as follows:

getMetadataKey :: [String] -> Item String -> Compiler String
getMetadataKey [key] item = getMetadataField' (itemIdentifier item) key

The functions unContext, getMetadataField', and data itemIdentifier are already defined in Hakyll.

The following is the working version of compiling pages/*:

match "pages/*" $ do
    route $ setExtension "html"
    compile $ do
        pageName <- takeBaseName . toFilePath <$> getUnderlying
        let pageCtx = constField pageName "" `mappend`
                      baseNodeCtx
        let evalCtx = functionField "eval" (evalCtxKey pageCtx)
        let activeSidebarCtx = sidebarCtx (evalCtx <> pageCtx)

        pandocCompiler
            >>= saveSnapshot "page-content"
            ...
            >>= loadAndApplyTemplate "templates/default.html" (activeSidebarCtx <> siteCtx)
            >>= relativizeUrls

and the sidebar layout:

$for(pages_list)$
    $if(title)$
          <a class="sidebar-nav-item$if(eval(title))$ active$endif$" href="$baseurl$$node-url$">$title$</a>
    $endif$
$endfor$

If you want to look at an example code, please check out the codes of lanyon-hakyll.

Berlin Machine Learning Seminar - Examples of Reinforcement Learning Applications in the Financial Market

I have given this talk on 22nd October 2020 at 19:00 to Berlin Machine Learning Seminar with the following abstract.

Abstract:

Reinforcement Learning has been broadly employed in financial markets for the last few years by benefiting from its nature of combining the behavior optimization (in this case buy and sell) and the market prediction. We will first discuss the hierarchical reinforcement learning scheme deployed by JPMorgan (arXiv, NIPS Workshop 2018) and then take a look at other two examples of DRL applied in trading. The first one (arXiv, KDD 2019) is implementing an interpretable network that works similar to a traditional trading strategy (Buying-Winners-and-Selling-Losers). The second one (arXiv, ICML 2019) is extending a traditional mathematical model (the Almgren and Chriss model) to a multi-agent setting in order to optimize a liquidation strategy.

The slides: here

Please contact me if you have any followup questions.

You can find my other technical talks in the github repo hahey/Talks.