The OCaml Planet

Caml Weekly News

2025-01-07T12:00:00-00:00

Dmitrii Kovanikov

2025-01-06T00:00:00-00:00

CCL: Categorical Configuration Language

Thomas Letan’s Blog

2025-01-01T00:00:00-00:00

    <h1>What Happened in 2024?</h1><div><span class="icon"><svg><use href="/~lthms/img/icons.svg#tag"></use></svg></span>&nbsp;<a href="https://soap.coffee/~lthms/tags/meta.html" marked="" class="tag">meta</a> <span class="icon"><svg><use href="/~lthms/img/icons.svg#tag"></use></svg></span>&nbsp;<a href="https://soap.coffee/~lthms/tags/opinions.html" marked="" class="tag">opinions</a> </div>

We are done with 2024, and now is a good time to reflect on what has happened over the past 12 months. I was not planning to, but my feed convinced me to give it a try . Plus, it is a good opportunity to revive my “Retrospective” series.

Free and Open Source Software

I’ve been a “prolific contributor” at $WORK, but less so with my personal projects.

Spatial Shell remains my most “popular” projectWe should reach 100 stars on GitHub in 2025 😅. , but except for a very minor 7th release in January, I have not touched it. It’s basically a done project, and I very much enjoy using it on a daily basis. My main regret is that, contrary to what is stated in its README, Spatial Shell does not work at all with i3.
ezjsonm-encoding was initially written for Spatial Shell, but I turned it into its own OCaml package in 2024. It is a JSON-only encoding library heavily borrowing on Data-encoding API , but with a more flexible default behavior for object parsing. I enjoyed writing the documentation, inspired by a tweet from Dmitrii Kovanikov Yes, cool kids moved to Bluesky in 2024, and Dmitrii is definitely a cool kid. . That being said, I have never bothered to benchmark this package properly, so if performances are important, it may not be a good fit for you.
jsonrpc2 is, as of January 1st, 2025, an experiment in providing a general-purpose framework for servers and clients communicating with the JSON RPC 2.0 protocol. I quite like the API I’m proposing there, and maybe I’ll try to polish and publish it in 2025.
bepo-tsrn.nvim is another thing I have done for myself, but published as if it was a public good. Now, instead of having to copy/paste the same Neovim configuration file on every computer I use, I can just type yay -S neovim-bepo-tsrn-git and be done with it.
celtchar has seen its first commits since 2021, which is not nothing. As a reminder, celtchar is a little tool I have written to generate ebooks and static websites for the stories I write; and as I was doing the 2024 edition of NaNoWriMo , I found myself in need to add a missing feature (supporting books split in parts, not only chapters).

Overall, I’ve been defaulting to OCaml for the past two years or so, and I am starting to think it is time to widen my perspective again. I will probably start with relearning GoI don’t know why, but I have been mildly obsessed with this language lately. .

Blog posts

2024 was not a very productive year when it comes to this website. I have published 5 articles, which is half the number of publications of 2023. As a logical consequence, not a lot of folks have visited my website this year. Funnily enough, the most read article (by far) in 2024 was published in 2023To be fair, it was published on December 30, 2023. .

Yes, 2024 was a quiet year for this website

That being said, I am quite happy with the content published in 2024.

Using git maintenance with Encrypted SSH Keys is the first article I published in 2024. It is a direct consequence of my trip to Brussels in February to attend to FOSDEM . If you haven’t already, you should watch Scott Chacon’s talk about Git less known commands; it is the only reason why I learned about git maintenance.
Installing a LUKS-Encrypted Arch Linux on a Vultr VPS is mostly a gift I have made to Future Me. It is a very specific how-to that I can use to quickly set up a new server with disk encryption. Funny story, I was planning to publish a follow-up about how I use systemd-nspawn to run my web services in containers , but no matter how many times I tried, I’ve never quite found a good way to tell this story. As I plan to educate myself on Kubernetes in 2025, it is not clear I will ever publish it now.
Introducing bepo-tsrn.nvim is probably the less useful article I have published in 2024, considering I expect the userbase to bepo-tsrn.nvim to stick to 1 until the very endBut who knows? Maybe one of you will prove me wrong! .
On Vestigial Structures hardly qualifies as a blog post, and is mostly a joke. It is also the only content on my website that was mostly generated by ChatGPT, and it is flagged as such. I don’t like using AI to write, but I do appreciate having a reviewer always at hand.
Serving This Article from RAM for Fun and No Real Benefit was very fun to write. This little experiment was stuck in my head for basically two years, and it turned out basically exactly as I had pictured it. That being said, I want to learn about CDNs now.

Overall, I still enjoy having my own little corner of the Internet, but if there is one thing I’d like to improve in 2025, it is its reach. I’d like you folks to run into my website, instead of having to promote it every time I write something. 2025, the year of SEO?

`$WORK`

2024 started with my decision to go back to a Software Engineering position, after giving an honest try at being an Engineering Manager in late 2023.

I want to remember 2024 for two things.

This year, more than ever, I have tried to appreciate my work beyond my individual contributions. I am confident in my programming skills (although I have so much to learn), but being an accomplished engineer is much more than contributing code. Making sure every engineer in the team can work to the best of their current ability, fostering a work environment favoring growth and initiative, estimating as precisely as possible the amount of time needed to deliver the next important thing, collaborating efficiently with non-technical teams... I am becoming increasingly interested in these areas.

Besides, this year was all about delivering and deploying in production. It’s been a wild ride , and I loved it even if it was very demanding. After having mostly contributed to R&D projects, the focus on UX, backward compatibility, etc. was very refreshing. I learned so much through the year, and had many opportunities to make significant impacts.

In 2025, I want to keep learning about software engineering, and maybe start sharing my thoughts on the subject on my website.

Talks

I gave only one talk in 2024, at a conference called EthCC . It was actually a follow-up to the talk I gave the year before. You can watch me deliver the talk here , but if you are more into written content, I have also published a transcript on this very website. I actually loved writing it down, and plan to systematically publish similar content for every recorded talk I will give in the future.

I also had the opportunity to participate in a Twitter Space .

This year, my public speaking opportunities were all $WORK-related. I would like to change this in the future, because there are enough events out there for me to start speaking about something other than workIt’s too late to apply to FOSDEM, but maybe I can find something to say to an event later in the year! I think. .

Sport

This year was a bit of a disappointment, sport-wise. I tried several times to get back to running regularly, and failed miserably. I went to Lyon for a 10km run without proper training, and skipped the half-marathon I signed up for. I should update my Running Log nonetheless. I started swimming regularly during the Summer, only to pierce my earlobes in September 😅.

Let’s hope I do better in 2025! I am planning to register for the Triathlon de Deauville with my sister. That promises to be fun! And I want to commit to the 20km de Paris as well.

On the bright side, I have started to use my bike again. I love riding around Paris, especially at night.

Final Notes

Overall, I’ve devoted a large part of my time to $WORK in 2024. Hopefully, I will find a better balance over the course of 2025, which should give me more time to explore and experiment more things.

Anyway, happy new year everyone! And happy Dry January!

Caml Weekly News

2024-12-31T12:00:00-00:00

Debajyati's Blog

2024-12-27T15:31:54-00:00

In this tutorial we are building a simple CLI named textsearch which is able to search a provided piece of string (probably a word) in a given text file.

Prior knowledge of components of a dune project is recommended but not required. I will guide you setting up the dune project.

Entire source code of this demo project is available at - Release v0.2.1 Debajyati/textsearch

This tutorial assumes you have dune preinstalled. If not, then install it via opam -

opam install dune

Summary of the Content Prior to Reading the Article

In this tutorial, you'll learn to build a simple command-line interface (CLI) tool called "textsearch" using OCaml and the Cmdliner library. The tool searches for a specified term in a text file with options for case sensitivity and character order. You'll be guided through setting up a Dune project, configuring the necessary files, writing the search logic, and defining command-line arguments using Cmdliner. The tutorial covers the project structure, dependencies, and detailed explanations of the code components to help you understand and implement the CLI tool effectively.

What is Cmdliner?

Cmdliner is a library that helps you build command line interface programs in OCaml.

It provides a simple and compositional mechanism to create commands and arguments in a declarative syntax.

So, lets get started!

Create a Dune Project

Lets start writing code. But before that we need to create a dune project.

Initialize the dune project by -

dune init proj textsearch

We will need the Cmdliner library installed in our system to build and run our project. So, install it via opam.

opam install cmdliner

Great! Now before you write the actual code, you need to setup the dune project by writing the configuration files properly.

Setting Up the Dune Project

At the end of this tutorial the file tree of our project will be like -

. bin    dune    main.ml lib    dune    search.ml test    dune    test_textsearch.ml dune-project textsearch.opam

Here the textsearch.opam file is not your concern, dont touch it. This file is generated by dune, edit dune-project instead.
The dune-project file defines project-wide settings.

For now, you dont need to edit most of the parts of the file dune-project.
Currently the file must look like -

(lang dune 3.17)(name textsearch)(generate_opam_files true)(source (github username/reponame))(authors "Author Name <author@example.com>")(maintainers "Maintainer Name <maintainer@example.com>")(license LICENSE)(documentation https://url/to/documentation)(package (name textsearch) (synopsis "A short synopsis") (description "A longer description") (depends ocaml) (tags  ("add topics" "to describe" your project))); See the complete stanza docs at https://dune.readthedocs.io/en/stable/reference/dune-project/index.html

Now change the package stanza to this -

(package (name textsearch) (synopsis "A simple text search CLI tool") (description "CLI tool to search terms in files") (depends  (ocaml (>= 5.2))  (cmdliner (>= 1.3.0))  (dune (>= 3.17))) (tags  ("cmdline" "text" "search")))

Nice!
We have two important directories here -

lib/ - Holds the source code for libraries within your project.
bin/ - Contains the source code for executables.

Forget the test folder for now as we are not going to write test cases for our project (At least not in this tutorial).

We also have a dune file in each of the folders. These files define how to build the specific executable within that directory.
Specifically, they are for -

Specifying dependencies (both internal and external libraries).
Specifying rules for building the target.

Paste the following code in bin/dune file -

(executable (public_name textsearch) (name main) (libraries search cmdliner))

Paste the code below in the lib/dune file -

(library  (name search) (public_name textsearch))

All good! Finally, the configuration work is DONE! HUH!

We can finally write the actual code.

Writing The Actual Code

Our Internal library

We will need an internal library called search.ml as we have specified in the dune files. Create that file in the lib/ directory.

touch lib/search.ml

Now, paste this code in the file -

let search_in_file ~case_sensitive ~consider_order term file =  let normalize s =    if case_sensitive then s else String.lowercase_ascii s  in  let term = normalize term in  let contains_term line =    let line = normalize line in    if consider_order then      try        let line_len = String.length line in        let term_len = String.length term in        let rec check_substring i =          if i + term_len > line_len then false          else if String.sub line i term_len = term then true          else check_substring (i + 1)        in        check_substring 0      with _ -> false    else      let term_chars = String.to_seq term |> List.of_seq in      let line_chars = String.to_seq line |> List.of_seq in      List.for_all (fun c -> List.mem c line_chars) term_chars  in  let ic = open_in file in  let rec process_lines line_num acc =    try      let line = input_line ic in      let new_acc = if contains_term line then (line_num, line) :: acc else acc in      process_lines (line_num + 1) new_acc    with End_of_file ->      close_in ic;      List.rev acc  in  process_lines 1 [];;let print_matches matches =  List.iter    (fun (line_num, line) -> Printf.printf "%d: %s\n" line_num line)    matches;;

Weve two functions in this file.

search_in_file - This function takes four arguments:
- case_sensitive: A boolean value indicating whether the search should be case sensitive or not.
- consider_order: A boolean value indicating whether the order of characters in the term matters during the search.
- term: The string to search for in the file.
- file: The path to the file to search in.
  The function opens a file for reading. Then it recursively iterates over the lines of the file. On each iteration, the function reads a line, checks if the line contains the term. If the line contains the term, it adds the line number and the line itself as a tuple to an accumulator list. It recursively calls itself on the next line number and the updated accumulator list. When it reaches the end of the file (End_of_file exception), it closes the file and reverses the accumulated list (to display lines in their original order).
print_matches - This function takes a list of matches as input where each match is a tuple containing the line number and the line itself. It iterates over the list of matches and prints each match in the format "line_num: line".

I hope this explanation is clear and good to go ahead.

Our Executable Program

This is our bin/main.ml file. Here we will import both of our internal library Search and external library Cmdliner, and then define the command and arguments.

open Cmdlineropen Search(* Define arguments )let search_term =  let doc = "The term to search for" in  Arg.(required & pos 0 (some string) None & info [] ~docv:"TERM" ~doc)let file_name =  let doc = "The file to search in" in  Arg.(required & pos 1 (some string) None & info [] ~docv:"FILE" ~doc)let case_sensitive =  let doc = "Perform a case-sensitive search" in  Arg.(value & flag & info ["c"; "case-sensitive"] ~doc)let consider_order =  let doc = "Consider the order of characters in the search term" in  Arg.(value & flag & info ["o"; "consider-order"] ~doc)( Core command logic )let run case_sensitive consider_order term file =  let matches = search_in_file ~case_sensitive ~consider_order term file in  print_matches matches;  if List.length matches = 0 then    (Printf.eprintf "No matches found.\n"; exit 1)  else    exit 0( Create the command )let cmd =  let info =    Cmd.info "textsearch"      ~version:"1.0.0"      ~doc:"Search for a term in a file with options for case sensitivity and character order"  in  Cmd.v info    Term.(const run $ case_sensitive $ consider_order $ search_term $ file_name)( Main entry point *)let () = Stdlib.exit (Cmd.eval cmd)

Lets break down the bin/main.ml file to understand its components and the utilities of the Cmdline library used here.

After imports, we first define all the arguments as variables. The arguments for our program are defined using Cmdliner.Arg module.
In this program we defined 4 arguments.

search_term (positional argument): - The term (word) to search for in the file.
file_name (positional argument): - The file in which the term is to be searched.
consider_order (flag): - Ensures that the order of characters in the search term is considered.
case_sensitive (flag): - Enables case-sensitive search.

Explanation of the Cmdliner library utilities used here: -

Arg: This module provides functions for defining command-line arguments, including their types, descriptions, and help messages.

Arg.required: This function defines a required argument (makes the argument mandatory).
Arg.pos: This function specifies the position of an argument in the command-line syntax. So, for example, pos 0 will make the argument the first positional argument.
Arg.info: This function provides a docstring and other metadata for an argument.
Arg.value: This function defines the value of an argument. The documentation says, - (value a) is a term that evaluates to a's value.
Arg.flag: This function defines a flag argument that may appear at most once on the command line. The argument holds true if the flag is present on the command line and false otherwise.
Arg.some: This function wraps the argument value in an Option type.
Arg.(&): This is an infix operator which performs a right associative composition operation, which simply means Arg.(&) enables composition of the combinators of a command-line argument by connecting individual components (arguments type, default value, parsing logic and docs) in a declarative way.

After imports we defined the core command logic. The whole command logic is encapsulated in the run function.

The run function calls the search_in_file function from the Search library. Prints the matching lines using print_matches function. Exits with code 1 if no matches are found, else exit code 0.

Next, we define the command in the cmd variable.

And finally, there is the main entry point of our program where it starts its execution from. That is -

let () = Stdlib.exit (Cmd.eval cmd)

Here, the cmd command gets evaluated and then exits the program with the return code from run.

Explanation of all the Cmdliner library utilities used here: -

Cmd: This module is the core of Cmdliner. It provides functions to define the commands, specify their options and arguments, and handle argument parsing, grouping and usage messages.

Cmd.info: This function creates a command description, including its name, version, and helptexts (a docstring explaining its purpose).
Cmd.v: This function creates a fully functional command combining the provided info and terms.
Cmd.eval: This function evaluates the given command, parses arguments, and executes the corresponding logic.

Term: This module deals with creating and manipulating command-line terms (arguments and subcommands).

Term.const: This function creates a term that evaluates to a constant value. In the given code, it's used to create terms for the run function and its arguments.
Term.($) - An infix operator that combines terms to pass arguments to a function (eventually the command to run from terminal).

So, you finally understood the Cmdline library.

Now its time to build and run our project to see if this works or not.

Building & Running the Project

Without waiting, let's build and run it!

Building the Project

Open your terminal and navigate to the root directory of your project (where dune-project is located).
Run the following command to build the program:
```
  dune build
```

This command will use the dune files to compile the source code in lib/search.ml and bin/main.ml and create an executable named textsearch in the _build/install/default/bin/ directory.

Now, you can run the executable by this command -

dune exec -- textsearch -c -o <term> "path/to/file/to/search/in"

Here, <term> is the term you want to search in the file.

Replace "path/to/file/to/search/in" with the actual path of the file you want to search the term in.

-c is the optional flag for case-sensitivity,

-o is the optional flag for respecting order of characters.

For example, I use vim and have the vim config file .vimrc located in the $HOME directory of my system.

I want to search my name in the file if its there. I would run -

dune exec -- textsearch -c -o Debajyati ~/.vimrc

And the program would print the matching lines in the console.

Tadaaa! And it works pretty well and smooth! Neat!

Congratulations! 🎉 You now have a functional CLI tool built with OCaml and Cmdliner!

Ending

I would really appreciate your feedback on this tutorial. How would you rate it? How can I improve it?

If you found this project or tutorial helpful, please consider starring the repository below. It will encourage me to create more content like this.

https://github.com/Debajyati/textsearch

If you found this POST helpful, if this blog added some value to your time and energy, please show some love by giving the article some likes and share it with your developer friends.

Feel free to connect with me at - Twitter, LinkedIn or GitHub :)

Happy Coding 🧑🏽💻👩🏽💻! Have a nice day ahead! 🚀

]]>

Thomas Letan’s Blog

2024-12-25T00:00:00-00:00

    <h1>Serving This Article from RAM with Dream for Fun and No Real Benefit</h1><div><span class="icon"><svg><use href="/~lthms/img/icons.svg#tag"></use></svg></span>&nbsp;<a href="https://soap.coffee/~lthms/tags/ocaml.html" marked="" class="tag">ocaml</a> <span class="icon"><svg><use href="/~lthms/img/icons.svg#tag"></use></svg></span>&nbsp;<a href="https://soap.coffee/~lthms/tags/meta.html" marked="" class="tag">meta</a> </div>

In 2022, Xe Iaso published a transcript of their talk on how their website was working at the time . In a nutshell, their approach consisted of a server preprocessing the website from its source at startup, then serving its contents from memory. If you have not already, I can only encourage you to read the article or watch the talk, as the story they tell is very interesting. For me personally, it sparked a question: what if, instead of preprocessing the website at startup, one decided to embed the already preprocessed website within the program of the HTTP server tasked to serve it?

Fast-forward today, and this question has finally been answered. The webpage you are currently reading has been served to you by an ad hoc HTTP server built with Dream , whose binary is the only file I need to push to my server to deploy the latest version of my website. I have actually deployed it, and it’s been serving the contents of this website for more than a week now.

What did I learn from this fun, little experiment? Basically, that this approach changes nothing, as far as Lighthouse and my monitoring is concerned. I couldn’t find any meaningful differences between a static website served by Nginx, a piece of software with thousands and thousands of engineering work behind it, and my little toy web server pieced together in an hour or so. Still. It was fun, so why not write about it?

This article is a kind of experience report. I’ll dive into what I have done to turn my website into a single, static binary. Not only does it mean writing some OCaml, which is always fun, but it also requires understanding a little some key HTTP headers, as well as using Docker to build easily deployable binaries. All in all, I hope it will be an interesting read for the curious minds.

Embedding My Website in a Binary

Not much had changed much since I stopped using cleopatra to generate this website, and the article I published in 2023 still stands. In a nutshell, I work in the site/ directory, and soupault generates my website in the out/~lthms directory, thanks to a collection of built-in and ad hoc pluginsFor instance, Markdown footnotes are turned into side notes with a soupault plugin. . To deploy the website, I was relying on rsync to sync the contents of the out/~lthms directory with the directory statically served by a Nginx instance on my personal server.

The first step of my little toy project is to actually embed the output of soupault into an OCaml program.

That’s where ocaml-crunch comes in handy. It is a little program published by Tarides , whose only job is to generate an OCaml module from a file system directory. It is straightforward to use it from Dune.

; file: out/dune
(rule
 (target website_content.ml)
 (deps (source_tree ~lthms))
 (action
  (run ocaml-crunch -m plain -o %{target} -s ~lthms)))

This snippet generates the website_content.ml module, which we can then expose through a library with the library stanza.

; file: out/dune
(library
 (name website_content))

And we are basically done. Excluding an Internal module, the signature of Website_content is pretty straightforward.

val file_list : string list
val read : string -> string option
val hash : string -> string option
val size : string -> int option

Serving the content with Dream

Dream is a cool project, and provides a straightforward API that we can leverage to turn our list of in-memory files into an HTTP server.

Naive Approach

Our goal now is to create a Dream.handler for each item in file_list. Done naively (as was my first attempt), it gives you something of the form:

let make_handler ~content path =
  Dream.get path (fun req ->
    Lwt.return (Dream.response content)))

Which we can use to build the main route we will then pass to Dream.router.

let website_route =
  Dream.scope "~lthms" []
  @@ List.map
       (fun path ->
         let content = Option.get (Website_content.read path) in
         make_handler ~content path)
       Website_content.file_list

With this approach, we build our handlers once, and then the lookup is done by Dream’s router. It could be an interesting experiment to see if doing the lookup ourselves is more performant (since Dream’s router is very generic, while in our case we don’t really need to parse anything). I remember Xe routing is basically going through a linked list, which seems strange at first, but works very well in practice because they have ordered said list with the most recent articles up front, and everybody comes to their website to read the latest article anyway.

It does not take an extensive QA process to figure out that this approach is far from being enough. To name a few things:

My website assumes http://path/index.html can be accessed with http://path/ or http://path. Our little snippet does not handle this.
Browsers expect the Content-Type headers to be correctly set. To give an example, they won't load a CSS file if the Content-Type header is not set to text/css.
Browsers work best for websites that take the time to provide caching directives. Our little snippet does not care to do so.
Even if my website is rather lightweight20MBytes at the time of writing the first version of this article. , compressing the response of our HTTP server for clients that support it is always a good idea.

This is a gentle reminder of all the things Nginx can do for you with very little configuration.

Handling `index.html` Synonyms

This one is rather simple. For files named index.html, we need 3 handers, not just one. We can achieve this with an additional helper make_handler_remove_suffix.

let make_handler_remove_suffix ~content path suffix
    =
  if String.ends_with ~suffix path then
    let alt_path =
      String.sub path 0 (String.length path - String.length suffix)
    in
    [ make_handler ~content alt_path ]
  else []

Updating the website_route definition to use make_handler_remove_suffix is quite easy as well.

 let website_route =
   Dream.scope "~lthms" []
-  @@ List.map
+  @@ List.concat_map
        (fun path ->
          let content = Option.get (Website_content.read path) in
-         make_handler ~content path)
+         if path = "index.html" then
+           (* Special case to deal with "index.html" which needs to be
+              recognized by the route "/" *)
+           [
+             make_handler ~content "/";
+             make_handler ~content "";
+             make_handler ~content "index.html";
+           ]
+         else
+           make_handler_remove_suffix ~content path
+             "/index.html"
+           @ make_handler_remove_suffix ~content
+               path "index.html"
+           @ [ make_handler ~content path ])
        Website_content.file_list

With that, https://soap.coffee/~lthms/posts/index.html returns the same pages as https://soap.coffee/~lthms/posts or https://soap.coffee/~lthms/posts. Check.

Supporting `Content-Type`

Content-Type is an HTTP header which is used by the receiver of the HTTP message (whether it is a request or a response) to interpret its content.

For instance, when building a RPC API, Content-Type is used by the server to know how to parse the request body (Content-Type: application/json or Content-Type: application/octet-stream being two popular choices, for JSON or binary encoding, respectively).

In our case, the Content-Type header is used by the HTTP server to communicate the nature of the content to browsers. For my website, I can just use the file extensions to infer the correct header to set. First, we list the extensions that are actually used.

let content_types =
  [
    (".html", "text/html");
    (".css", "text/css");
    (".xml", "text/xml");
    (".png", "image/png");
    (".svg", "image/svg+xml");
    (".gz", "application/gzip");
    (".pub", "text/plain");
  ]

A header in Dream is encoded as a string * string value, with the first string being the header name and the second being the header value.

let content_type_header path =
  List.filter_map
    (fun (ext, content_type) ->
      if String.ends_with ~suffix:ext path then
        Some ("Content-Type", content_type)
      else None)
    content_types
  |> assert_f
       ~error_msg:Format.(sprintf "Unsupported file type %s" path)
       (( <> ) [])

with assert_f being defined as follows.

let assert_f ~error_msg f v =
  if f v then v else failwith error_msg

assert_f is used to enforce that I don’t deploy a website which contains route lacking a Content-Type header. For instance, if I remove the "html" entry of the content_type list, I get this error when I try to execute the server.

Fatal error: exception Failure("Unsupported file type index.html")

This is because the headers are only computed once, when each route are defined. This is a key principle of this project: compute once, serve many timeI would love to get a compilation error instead (considering there are no runtime values involved), but have not looked into this just yet. .

 let website_route =
   Dream.scope "~lthms" []
   @@ List.concat_map
        (fun path ->
          let content = Option.get (Website_content.read path) in
+         let headers = content_type_header path in
          if path = "index.html" then
            (* Special case to deal with "index.html" which needs to be
               recognized by the route "/" *)
            [
-             make_handler ~content "/";
-             make_handler ~content "";
-             make_handler ~content "index.html";
+             make_handler ~headers ~content "/";
+             make_handler ~headers ~content "";
+             make_handler ~headers ~content "index.html";
            ]
          else
-           make_handler_remove_suffix ~content path
+           make_handler_remove_suffix ~headers ~content path
              "/index.html"
-           @ make_handler_remove_suffix ~content
+           @ make_handler_remove_suffix ~headers ~content
                path "index.html"
-           @ [ make_handler ~content path ])
+           @ [ make_handler ~headers ~content path ])
        Website_content.file_list

(The changes in make_handler and make_handler_remove_prefix are left as an exercise to enthusiast readers)

Compressing if Requested

Nowadays, computations are cheap, while downloading data costs time (and sometimes money). As a consequence, it is often a good idea for a server to compress a large HTTP response, and browsers do ask them to do so, by setting the Accept-Encoding header of their requests.

The value of the Accept-Encoding header is a comma-separated list of supported compression algorithms, optionally ordered with a priority value q.

For instance, Accept-Encoding: gzip;q=0.5, deflate;q=0.3, identity tells you that the browser supports three encoding methods: gzip, deflate and identity (no compression), and the browser prefers gzip over deflate. Besides, the request can provide several Accept-Encoding headers instead of just one, so we can have

Accept-Encoding: gzip;q=0.5
Accept-Encoding: deflate;q=0.3
Accept-Encoding: identity

The String module provides everything we need to check if a browser supports gzip as an encoding methodSpoiler: they do. I was even wondering at some point if I could just always return GZIP-compressed values, ignoring the Accept-Encoding header altogether. If you do that, though, curl becomes annoying to use (it does not uncompress the response automatically, and instead complains about being about to write binary to the standard output). .

(* For [method(; q=val)?], returns [method], except if
   [q=0]. *)
let to_directive str =
  match String.split_on_char ';' str |> List.map String.trim with
  | [ x ] -> Some x
  | [ x; y ] -> (
      match String.split_on_char '=' y |> List.map String.trim with
      | [ "q"; "0" ] -> None
      | [ "q"; _ ] -> Some x
      | _ -> None)
  | _ -> None
(* [contains ~value:v header] returns [true] if [v] is a
supported method listed in [header]. *)
let contains ~value header =
String.split_on_char ',' header
|> List.to_seq |> Seq.map String.trim
|> Seq.filter_map to_directive
|> Seq.exists (( = ) value)

We use contains to tell us if we can return a compressed response, which leaves us with one final question: how to compress said response?

The OCaml ecosystem seems to have picked camlzip library when GZIP is involvedYou know it is a legitimate OCaml library when one of the top-level modules is not documented at all . . What is surprising with this library is that it does not support in-memory compression: the functions expect channels, not bytes. That is quite annoying, because we are specifically doing this not to use files.

The Internet is helpful here, and quickly suggests using pipes. It works when you remember –or figure out– that pipes are a blocking mechanism: one does not just write a buffer of arbitrary size in a pipe, because after something like 4KBytes, writing becomes blocking until a read happens to free some space. That’s not a big problem: we can read and write concurrently to the pipe using threads, and OCaml 5 makes it quite easy to do so with the Domain module.

let gzip content =
  let inc, ouc = Unix.pipe () in
  let ouc = Gzip.open_out_chan ~level:6 Unix.(out_channel_of_descr ouc) in
  let _writer =
    Domain.spawn (fun () ->
        Gzip.output_substring ouc content 0 String.(length content);
        Gzip.close_out ouc)
  in
  let res = In_channel.input_all Unix.(in_channel_of_descr inc) in
  Unix.close inc;
  res

We know how to decide whether to compress or not, and how to compress. The next step is to modify make_handler accordingly.

-let make_handler ~headers ~content path =
+let make_handler ~headers ~gzip_content ~content path =
+  let gzip_headers = ("Content-Encoding", "gzip") :: headers in
   Dream.get path (fun req ->
-    Lwt.return (Dream.response content)))
+      match Dream.headers req "Accept-Encoding" with
+      | accepted_encodings
+        when List.exists (contains ~value:"gzip") accepted_encodings ->
+          Lwt.return @@ Dream.response ~headers:gzip_headers gzip_content
+      | _ -> Lwt.return @@ Dream.response ~headers content)

gzip_content is computed only once (using our gzip function), and passed to make_handler. This way, the only computation the handler needs to do is to “parse” Accept-Encoding.

Caching

Compressing a page to reduce the number of bytes a browser needs to download is fine. Letting the browser know it does not need to download anything because it's previous version is still accurate is better.

This is achieved through two complementary mechanisms: entity tags (ETag)My first encounter with entity tags was around the time GDPR was a hot topic, because you can use them as a cheap replacement for cookies to track your users . I remained at the surface level at the time, it was fun learning more about them through this little project. , and cache policies (Cache-Control).

Entity tags are used to identify a resource, and are expected to change every time the resource is updated. The general workflow goes like this: the first time a browser requests https://soap.coffee/~lthms/index.html, it caches the result along with the value of the ETag header. The next time it needs the page, it adds the header If-None-Match, with the ETag as its value. For such requests, the server is expected to return an empty response with HTTP code 304 (Not Modified).

I decided to use the sha256 hash algorithm to compute the entity tag of each resource of my website. The sha OCaml library looked like a good enough candidate.

(* in `website_route` *)
let etag = Sha256.(string content |> to_hex) in

Interestingly, one question I had to answer was whether the entity tag of a page needed to be different whether it was compressed or not. Internet almost unanimously answered yes. So be it. We just need to keep in mind ETag values are expected to be surrounded by quotes, and we are good to go. It is just a matter of suffixing the ETag with +gzip in the compressed case.

-let make_handler ~headers ~gzip_content ~content path =
-  let gzip_headers = ("Content-Encoding", "gzip") :: headers in
+let make_handler ~headers ~etag ~gzip_content ~content path =
+  let etag_gzip = Format.sprintf "\"%s+gzip\"" etag in
+  let etag = Format.sprintf "\"%s\"" etag in
+  let gzip_headers =
+    ("Content-Encoding", "gzip") :: ("ETag", etag_gzip) :: headers in
+  let identity_headers = ("ETag", etag) :: headers in
   Dream.get path (fun req ->
-      match Dream.headers req "Accept-Encoding" with
-      | accepted_encodings
-        when List.exists (contains ~value:"gzip") accepted_encodings ->
-          Lwt.return @@ Dream.response ~headers:gzip_headers gzip_content
-      | _ -> Lwt.return @@ Dream.response ~headers content)
+      match Dream.headers req "If-None-Match" with
+      | [ previous_etag ] when previous_etag = etag || previous_etag = etag_gzip
+        ->
+          Lwt.return
+          @@ Dream.response
+               ~headers:(("ETag", previous_etag) :: headers)
+               ~code:304 ""
+      | _ -> (
+          match Dream.headers req "Accept-Encoding" with
+          | accepted_encodings
+            when List.exists (contains ~value:"gzip") accepted_encodings ->
+              Lwt.return @@ Dream.response ~headers:gzip_headers gzip_content
+          | _ -> Lwt.return @@ Dream.response ~headers:identity_headers content))

Entity tags are useful, but you still need the browser to make an HTTP request every single time you visit the website. By setting a cache policy, we can remove even remove the need for this request most of the time. The Cache-Control header is used to set a number of parameters, including the max-age value (in seconds).

In my Nginx configuration, I had set max-age to a year for images. I did the same thing here. Besides, I decided to set max-age for other resources to 5 minutes. This seems like a good compromise: since my website does not change very often, it is very unlikely that you happen to visit it when I publish new content. Setting a 5-minute cache policy should let my readers download each resource only once, yet get the freshest version at their next visitDealing with the Cache-Control header is basically the same exercise as setting the correct Content-Type header, and this article is already long enough, which is why there is no diff or snippet in this section. .

And with this, we are done. We get a standalone library to server our website in a browser-friendly manner, which I can theoretically use to replace my current Nginx-powered setup. Although… is it that simple?

Building and Deploying the Website

Files are quite easy to share and deploy. As I mentioned earlier in this article, you just need to rsync them and be done with it. Binaries, on the other hand… One cannot just assume a binary build on a machine X will work on another machine Y. Some additional works need to be done.

The most straightforward solution I know is to rely on static binaries. I have already written about how to generate static binaries for OCaml projects, so I had a pretty strong head start, but one drawback of the approach I’ve described there (and that I have been using for Spatial Shell’s releases Not that there were many of them lately. ) is that it is rather slow (I have a script creating a new local switch each time) and requires static libraries to be installed (which Arch Linux does not provide).

And so, I figured, why not build the static binary in Docker? This allows me to use Alpine to get a static version of my system dependencies, and can be quite fast thanks to Docker build cache . The Dockerfile is quite simple: one stage for building the system and OCaml dependencies, and one stage for building the static binary.

FROM alpine:3.21 AS build_environment
# Use alpine /bin/ash and set shell options
# See https://docs.docker.com/build/building/best-practices/#using-pipes
SHELL ["/bin/ash", "-euo", "pipefail", "-c"]
USER root
WORKDIR /root
RUN apk add autoconf automake bash build-base ca-certificates opam gcc \ 
git rsync gmp-dev libev-dev openssl-libs-static pkgconf zlib-static 

openssl-dev zlib-dev
RUN opam init --bare --yes --disable-sandboxing
COPY makefile dune-project .
RUN make _opam/.init OCAML="ocaml-option-static,ocaml-option-no-compression,ocaml.5.2.1"
RUN eval $(opam env) && make server-deps
FROM build_environment AS builder
COPY server ./server
COPY out ./out
COPY dune .
RUN eval $(opam env) && dune build server/main.exe --profile=static
FROM alpine:3.21 AS soap.coffee
COPY --from=builder /root/_build/default/server/main.exe /bin/soap.coffee

Then, building my static binary becomes as simple as:

docker build . -f ./build.Dockerfile \
  --target soap.coffee \
  -t soap.coffee:latest
docker create --name soap-coffee-build soap.coffee:latest
docker cp soap-coffee-build:/bin/soap.coffee .
docker rm -f soap-coffee-build

docker cp does not work on an image, but on a container, so we need to create one which can be destroyed shortly after.

This little binary weights 38MBytes, which seems relatively reasonable to me, considering my website weights 20MBytes. I guess it could be easy to reduce this size by embedding the compressed version of my articles and images, instead of the uncompressed one. But really, for my website, I’m not really interested in investing the extra effort.

Conclusion

I would not recommend anyone to use this in production for anything remotely important, but from my perspective, it was both fun and insightful. I was able to refresh my memories about HTTP “internal,” among other things. Again, as far as I can tell, deploying my website this way did not bring me any benefit, performance-wise; even worse, I am pretty sure the Dream server will not behave as well as Nginx when it comes to handling the load (since it is limited to one core, instead of several with Nginx).

That being said, I am way too invested now… which is why, yes, you are reading a blog post served to you directly from memory 🎉.

Thomas Letan’s Blog

2024-12-25T00:00:00-00:00

    <h1>Serving This Article from RAM for Fun and No Real Benefit</h1><div><span class="icon"><svg><use href="/~lthms/img/icons.svg#tag"></use></svg></span>&nbsp;<a href="https://soap.coffee/~lthms/tags/ocaml.html" marked="" class="tag">ocaml</a> <span class="icon"><svg><use href="/~lthms/img/icons.svg#tag"></use></svg></span>&nbsp;<a href="https://soap.coffee/~lthms/tags/meta.html" marked="" class="tag">meta</a> </div>

Embedding My Website in a Binary

The first step of my little toy project is to actually embed the output of soupault into an OCaml program.

That’s where ocaml-crunch comes in handy. It is a MirageOS project, whose only job is to generate an OCaml module from a file system directory. It is straightforward to use it from Dune.

; file: out/dune
(rule
 (target website_content.ml)
 (deps (source_tree ~lthms))
 (action
  (run ocaml-crunch -m plain -o %{target} -s ~lthms)))

This snippet generates the website_content.ml module, which we can then expose through a library with the library stanza.

; file: out/dune
(library
 (name website_content))

And we are basically done. Excluding an Internal module, the signature of Website_content is pretty straightforward.

val file_list : string list
val read : string -> string option
val hash : string -> string option
val size : string -> int option

Serving the content with Dream

Dream is a cool project, and provides a straightforward API that we can leverage to turn our list of in-memory files into an HTTP server.

Naive Approach

Our goal now is to create a Dream.handler for each item in file_list. Done naively (as was my first attempt), it gives you something of the form:

let make_handler ~content path =
  Dream.get path (fun req ->
    Lwt.return (Dream.response content)))

Which we can use to build the main route we will then pass to Dream.router.

let website_route =
  Dream.scope "~lthms" []
  @@ List.map
       (fun path ->
         let content = Option.get (Website_content.read path) in
         make_handler ~content path)
       Website_content.file_list

It does not take an extensive QA process to figure out that this approach is far from being enough. To name a few things:

My website assumes http://path/index.html can be accessed with http://path/ or http://path. Our little snippet does not handle this.
Browsers expect the Content-Type headers to be correctly set. To give an example, they won't load a CSS file if the Content-Type header is not set to text/css.
Browsers work best for websites that take the time to provide caching directives. Our little snippet does not care to do so.
Even if my website is rather lightweight20MBytes at the time of writing the first version of this article. , compressing the response of our HTTP server for clients that support it is always a good idea.

This is a gentle reminder of all the things Nginx can do for you with very little configuration.

Handling `index.html` Synonyms

This one is rather simple. For files named index.html, we need 3 handers, not just one. We can achieve this with an additional helper make_handler_remove_suffix.

let make_handler_remove_suffix ~content path suffix
    =
  if String.ends_with ~suffix path then
    let alt_path =
      String.sub path 0 (String.length path - String.length suffix)
    in
    [ make_handler ~content alt_path ]
  else []

Updating the website_route definition to use make_handler_remove_suffix is quite easy as well.

 let website_route =
   Dream.scope "~lthms" []
-  @@ List.map
+  @@ List.concat_map
        (fun path ->
          let content = Option.get (Website_content.read path) in
-         make_handler ~content path)
+         if path = "index.html" then
+           (* Special case to deal with "index.html" which needs to be
+              recognized by the route "/" *)
+           [
+             make_handler ~content "/";
+             make_handler ~content "";
+             make_handler ~content "index.html";
+           ]
+         else
+           make_handler_remove_suffix ~content path
+             "/index.html"
+           @ make_handler_remove_suffix ~content
+               path "index.html"
+           @ [ make_handler ~content path ])
        Website_content.file_list

With that, https://soap.coffee/~lthms/posts/index.html returns the same pages as https://soap.coffee/~lthms/posts or https://soap.coffee/~lthms/posts. Check.

Supporting `Content-Type`

Content-Type is an HTTP header which is used by the receiver of the HTTP message (whether it is a request or a response) to interpret its content.

let content_types =
  [
    (".html", "text/html");
    (".css", "text/css");
    (".xml", "text/xml");
    (".png", "image/png");
    (".svg", "image/svg+xml");
    (".gz", "application/gzip");
    (".pub", "text/plain");
  ]

A header in Dream is encoded as a string * string value, with the first string being the header name and the second being the header value.

let content_type_header path =
  List.filter_map
    (fun (ext, content_type) ->
      if String.ends_with ~suffix:ext path then
        Some ("Content-Type", content_type)
      else None)
    content_types
  |> assert_f
       ~error_msg:Format.(sprintf "Unsupported file type %s" path)
       (( <> ) [])

with assert_f being defined as follows.

let assert_f ~error_msg f v =
  if f v then v else failwith error_msg

Fatal error: exception Failure("Unsupported file type index.html")

 let website_route =
   Dream.scope "~lthms" []
   @@ List.concat_map
        (fun path ->
          let content = Option.get (Website_content.read path) in
+         let headers = content_type_header path in
          if path = "index.html" then
            (* Special case to deal with "index.html" which needs to be
               recognized by the route "/" *)
            [
-             make_handler ~content "/";
-             make_handler ~content "";
-             make_handler ~content "index.html";
+             make_handler ~headers ~content "/";
+             make_handler ~headers ~content "";
+             make_handler ~headers ~content "index.html";
            ]
          else
-           make_handler_remove_suffix ~content path
+           make_handler_remove_suffix ~headers ~content path
              "/index.html"
-           @ make_handler_remove_suffix ~content
+           @ make_handler_remove_suffix ~headers ~content
                path "index.html"
-           @ [ make_handler ~content path ])
+           @ [ make_handler ~headers ~content path ])
        Website_content.file_list

(The changes in make_handler and make_handler_remove_prefix are left as an exercise to enthusiast readers)

Compressing if Requested

The value of the Accept-Encoding header is a comma-separated list of supported compression algorithms, optionally ordered with a priority value q.

Accept-Encoding: gzip;q=0.5
Accept-Encoding: deflate;q=0.3
Accept-Encoding: identity

(* For [method(; q=val)?], returns [method], except if
   [q=0]. *)
let to_directive str =
  match String.split_on_char ';' str |> List.map String.trim with
  | [ x ] -> Some x
  | [ x; y ] -> (
      match String.split_on_char '=' y |> List.map String.trim with
      | [ "q"; "0" ] -> None
      | [ "q"; _ ] -> Some x
      | _ -> None)
  | _ -> None
(* [contains ~value:v header] returns [true] if [v] is a
supported method listed in [header]. *)
let contains ~value header =
String.split_on_char ',' header
|> List.to_seq |> Seq.map String.trim
|> Seq.filter_map to_directive
|> Seq.exists (( = ) value)

We use contains to tell us if we can return a compressed response, which leaves us with one final question: how to compress said response?

let gzip content =
  let inc, ouc = Unix.pipe () in
  let ouc = Gzip.open_out_chan ~level:6 Unix.(out_channel_of_descr ouc) in
  let _writer =
    Domain.spawn (fun () ->
        Gzip.output_substring ouc content 0 String.(length content);
        Gzip.close_out ouc)
  in
  let res = In_channel.input_all Unix.(in_channel_of_descr inc) in
  Unix.close inc;
  res

Caution

As rightfully pointed out by Daniel Bünzli , the gzip function presented in this article is full of shortcomings. To quote the message, the function can leak fds in case of errors and domains are not meant to be used that way (it’s rather spawn one long running domain per CPU you have). It’s not necessarily more complicated to correct it to use Fun.protect invocations to make sure all your fds get closed even if the function blows up and use Thread.create so that the netizens cut and paste correct code.

In my opinion, this implementation is “good enough” for my use case, which is compressing arbitrary strings before the HTTP server is even started. If it were to be called in the handlers themselves, then definitely, it would not be suitable.

Keep that in mind if you want to borrow this code.

We know how to decide whether to compress or not, and how to compress. The next step is to modify make_handler accordingly.

-let make_handler ~headers ~content path =
+let make_handler ~headers ~gzip_content ~content path =
+  let gzip_headers = ("Content-Encoding", "gzip") :: headers in
   Dream.get path (fun req ->
-    Lwt.return (Dream.response content)))
+      match Dream.headers req "Accept-Encoding" with
+      | accepted_encodings
+        when List.exists (contains ~value:"gzip") accepted_encodings ->
+          Lwt.return @@ Dream.response ~headers:gzip_headers gzip_content
+      | _ -> Lwt.return @@ Dream.response ~headers content)

gzip_content is computed only once (using our gzip function), and passed to make_handler. This way, the only computation the handler needs to do is to “parse” Accept-Encoding.

Caching

I decided to use the sha256 hash algorithm to compute the entity tag of each resource of my website. The sha OCaml library looked like a good enough candidate.

(* in `website_route` *)
let etag = Sha256.(string content |> to_hex) in

-let make_handler ~headers ~gzip_content ~content path =
-  let gzip_headers = ("Content-Encoding", "gzip") :: headers in
+let make_handler ~headers ~etag ~gzip_content ~content path =
+  let etag_gzip = Format.sprintf "\"%s+gzip\"" etag in
+  let etag = Format.sprintf "\"%s\"" etag in
+  let gzip_headers =
+    ("Content-Encoding", "gzip") :: ("ETag", etag_gzip) :: headers in
+  let identity_headers = ("ETag", etag) :: headers in
   Dream.get path (fun req ->
-      match Dream.headers req "Accept-Encoding" with
-      | accepted_encodings
-        when List.exists (contains ~value:"gzip") accepted_encodings ->
-          Lwt.return @@ Dream.response ~headers:gzip_headers gzip_content
-      | _ -> Lwt.return @@ Dream.response ~headers content)
+      match Dream.headers req "If-None-Match" with
+      | [ previous_etag ] when previous_etag = etag || previous_etag = etag_gzip
+        ->
+          Lwt.return
+          @@ Dream.response
+               ~headers:(("ETag", previous_etag) :: headers)
+               ~code:304 ""
+      | _ -> (
+          match Dream.headers req "Accept-Encoding" with
+          | accepted_encodings
+            when List.exists (contains ~value:"gzip") accepted_encodings ->
+              Lwt.return @@ Dream.response ~headers:gzip_headers gzip_content
+          | _ -> Lwt.return @@ Dream.response ~headers:identity_headers content))

Building and Deploying the Website

FROM alpine:3.21 AS build_environment
# Use alpine /bin/ash and set shell options
# See https://docs.docker.com/build/building/best-practices/#using-pipes
SHELL ["/bin/ash", "-euo", "pipefail", "-c"]
USER root
WORKDIR /root
RUN apk add autoconf automake bash build-base ca-certificates opam gcc \ 
git rsync gmp-dev libev-dev openssl-libs-static pkgconf zlib-static 

openssl-dev zlib-dev
RUN opam init --bare --yes --disable-sandboxing
COPY makefile dune-project .
RUN make _opam/.init OCAML="ocaml-option-static,ocaml-option-no-compression,ocaml.5.2.1"
RUN eval $(opam env) && make server-deps
FROM build_environment AS builder
COPY server ./server
COPY out ./out
COPY dune .
RUN eval $(opam env) && dune build server/main.exe --profile=static
FROM alpine:3.21 AS soap.coffee
COPY --from=builder /root/_build/default/server/main.exe /bin/soap.coffee

Then, building my static binary becomes as simple as:

docker build . -f ./build.Dockerfile \
  --target soap.coffee \
  -t soap.coffee:latest
docker create --name soap-coffee-build soap.coffee:latest
docker cp soap-coffee-build:/bin/soap.coffee .
docker rm -f soap-coffee-build

docker cp does not work on an image, but on a container, so we need to create one which can be destroyed shortly after.

Conclusion

That being said, I am way too invested now… which is why, yes, you are reading a blog post served to you directly from memory 🎉.

Caml Weekly News

2024-12-24T12:00:00-00:00

Tarides

2024-12-23T00:00:00-00:00

In a previous post, we discussed how we have developed property-based tests (PBTs) to stress test the new runtime system in OCaml 5, and gave concrete examples of such tests. In this second part, we discuss some of the challenges and the lessons learned from that effort.

Testing APIs With Hidden or Uncontrolled State

In part 1, we saw how STM and Lin were useful to test stateful module interfaces, like Float.Array. The OCaml standard library and runtime however also expose modules that are stateful, but for which the state is hidden or outside the full control of a black-box testing process. We are nevertheless interested in helping ensure the correctness of such modules.

For example, Ephemerons depend on the state of OCaml's heap, meaning that a garbage collection (generally out of control of the test driver) may unsuspectingly trigger and cause changes in the test outcome. As a result, we ended up abandoning Lin tests of Ephemerons, as they would run multiple observations of the same system under test - one parallel observation and several sequential ones - most likely with different results, because of different garbage collection schedulings.

In addition, the Lin tests could cause excessive shrinking searches, in trying to find a minimal example causing Ephemeron differences between such runs. Instead we favoured replacing them with STM tests, that perform only one observation of the system under test. We furthermore experimented with inserting an explicit call to Gc.full_major in between our run of the sequential and parallel tests to increase the chance of starting the latter on a relatively stable heap basis. We encountered similar problems with tests of the Weak module and recently also in the STM tests of the Gc module.

Despite these challenges, the above tests successfully found several issues, e.g.

racing Weak functions could in some cases produce strange values,
certain combinations of Weak Hashset functions could cause the runtime to abort or segfault,
the Ephemeron tests could trigger an assertion failure and abort, and
out-of-date documentation for Gc.quick_stat and Gc.set

Cygwin Challenges

As Cygwin support was being restored up in the 5.1 release, we wanted to test this platform as well to help ensure its correctness. However, we found that a test-suite run took an excessively long time, often not completing within a 6-hour timeout! We solved this initially by splitting a test-suite run into two separate workflows and later by rephrasing it as two separate CI jobs, belonging to the same workflow. Thanks to general improvements to the OCaml 5 runtime system, we have since been able to merge this split back into just one job like the remaining platforms, and eventually also reduce the timeout for this single Cygwin workflow to 4 hours, like the remaining platforms.

Another Cygwin challenge early on was the relatively old opam.2.0.7 version it includes. This made it harder to test the various OCaml compiler versions on Cygwin (and later MinGW). As a workaround, we set up a custom opam repository with appropriate opam files for each compiler version.

Keeping Dependencies Minimal

Initially we happily used ppxlib to generate boilerplate code for QCheck generators using ppx_deriving_qcheck and show functions for the tested cmds. However, ppxlib depends on the OCaml compiler's AST which means that it occasionally breaks on the compiler's trunk development branch whenever its AST changes. As a consequence, testing trunk would halt until ppxlib was fixed again - an unfortunate situation when trying to help ensure its correctness! After helping keep a branch of ppxlib continuously working with trunk, at some point we instead decided to eliminate the test suite's ppxlib dependency. We therefore wrote the corresponding definitions by hand and by utilising a dedicated printing library Pp in qcheck-multicoretests-util. Since then, testing has not been blocked by such breakages, and the dependencies are down to just the qcheck-core package, and (transitively) dune.

Testing an increasing number of platforms such as the MinGW and Cygwin ports over the past 2-3 years has been challenging, as much of that effort predates the more recent Windows support brought by opam-2.2. As we additionally wanted to test OCaml 5's now restored MSVC port, also during its development, we ended up abandoning opam and setup-ocaml in favour of just building the tested compiler and our dependencies (qcheck-core and dune) from source in our CI workflows. As a result, we have gained a uniform CI workflow setup across platforms that also allows us to kick-off tests of feature branches, and thereby eliminate the need for the above-mentioned custom opam repository.

Testing in the Presence of Misbehaviour

Overall, when having found, investigated, reported, and sometimes also fixed an error, we would like to continue testing despite the existence of such known issues. A test-suite rerun is however likely to trigger and report the same bug again, temporarily hindering the discovery and fixing of other issues. Something similar has been observed by others, e.g. in Quviq's property-based testing of AUTOSAR for Volvo.

To be able to continue testing we can, in the simple cases, (temporarily) adjust the tested property to accept the observed (mis)behaviour. This was the case, for example, for Gc.quick_stat which would return non-zero entries for four record fields, where the documentation was out-of-date and specifying that zeros should be returned.

However we have also had to (temporarily) adjust the generator to avoid triggering a particular buggy cmd. This was the case, for example, for Gc.counters which had an independently found-and-fixed issue with improper C interfacing with the GC. Our new Gc test would nevertheless trigger it and cause a crash on 5.2.0, until we adjusted the generator to skip generating Gc.counters calls on tests of versions up to 5.2.0. Since then the fix has been included in the 5.2.1 bugfix release.

Finally, we have also (temporarily) disabled a test on a platform. This is the case, for example, for the parallel test of Dynlink which is unsafe on Windows, due to an underlying flexdll issue. Whereas the flexdll issue is now fixed in the flexdll repository, we are awaiting a new release before proceeding to re-enable the parallel test on Windows.

Crashes Take Down the Test Runner Too

Since the test-driver is running in OCaml too – and in the same process – when the SUT crashes, so does the entire QCheck test-driver process. In our experience this happens more often than not, as runtime issues tend to lead to memory corruption and typically a segmentation fault. Running the test in a forked child process may guard against a crash in the child taking down the parent process, with the caveat that OCaml 5 prevents forking child processes after the first Domain.spawn. In the spirit of functional programming, this option is available as a reusable combinator Util.fork_prop_with_timeout : int -> ('a -> bool) -> 'a -> bool in qcheck-multicoretests-util, thus allowing us to easily wrap the property of a crash-triggering QCheck test.

Despite not designed with any of the above in mind, we have arrived at a test layout where most tests are run as separate executables, which lets us identify crashes relatively easily and simultaneously lets a test-suite run continue despite encountering a crash underway.

Positive Testing, Negative Testing, and Stress Testing

While developing the STM and Lin libraries it became clear that we should guard against changes mistakenly affecting their error-finding behaviour. For this reason, we added negative tests of e.g. parallel modifications of an unprotected ref cell that are expected to fail, and then extended QCheck with a Test.make_neg function to construct a negative test. In the CI we then use this negative testing ability e.g. to test that Hashtbl is unsafe to use in parallel as mentioned in part 1 and that the generator can find a counterexample illustrating it.

Often, such a parallel negative test triggers quickly, e.g. on one of the first 10 test inputs generated – effectively stress-testing a module under parallel usage for only a very limited amount of time. We therefore added stress test properties, in the form of stress_test for Lin and stress_test_par for STM, both offering a weaker, more forgiving property that only fails on unexpected exceptions or outright crashes, thus strengthening our belief in the runtime - even under longer, continued parallel misusage.

Effectively, we have arrived at PBT variants of classical test concepts:

positive tests - expected to hold across many random test inputs
negative tests - expected not to hold and produce a counterexample
stress tests - expected not to misbehave by raising an exception or crashing

False Alarms

As more and more CI target platforms have been added, we have also seen a variation in behaviour across them: Some of the above negative tests are not triggered as consistently across platforms and some of the tests take a long time or cause timeouts on some platforms.

These add noise and still require us to check whether a failure was genuine or not. We have therefore focused on reducing the noise from false positives. To better understand this effort here is a plot of CI workflow outcomes for merged PRs (361-486) spanning a period of ~1.5 years from early June 2023 to early December 2024:

First of all, this covers a period of testing a mix of OCaml versions, starting with workflows targeting 5.1 in June 2023, then 5.2, and now 5.3 and 5.4/trunk (the current compiler development version). Note that this only plots the outcome for Jan's PRs to main for a fair comparison, as these would originally trigger twice as many workflow runs (push and PR). It furthermore only includes the last run for each PR, in case there were more because of PR revisions. Each CI run may in principle trigger several errors (in different categories even!). This is a rare incident however.

Further notes:

On 370 a ppc64 workflow was added (workflow number increase)
391 added framepointer workflows (workflow number increase)
392-393 revealed a cmi-file lookup regression that made several workflows fail, hence the spike
There were CI and network issue around 395-398
On 396 a first FreeBSD workflow was added (workflow number increase)
On 398 a second FreeBSD workflow and extra opam install workflows were added (workflow number increase)
On 420 the 2-split Cygwin workflows were merged (workflow number decrease)
On 429 (merged before 431) we eliminated duplicate CI runs for both push and the PR
On 438 we retired the 5.1 workflows to only run weekly (not on every PR)
On 449 the older MSVC PR (399) adding 2 additional MSVC workflows had just been merged
On 453 the parallel Dynlink tests were disabled on Windows (since a fix had been developed and offered on the FlexDLL repo)
On 458 2 macOS ARM64 workflows were temporarily duplicated while moving them from multicoretests-ci to GitHub actions
On 471 5.4.0+trunk workflows were added and removed three 5.1 multicoretests-ci workflows
On 481 multicoretests-ci stopped running the 4 remaining 5.2 workflows
On 482 the ten 5.2 workflows were disabled

The smaller sub-bars are harder to distinguish with the dominant OK bars in the above figure. Below we therefore zoom in and display the same plot, including only the different kinds of failures:

There are multiple competing efforts and aspects behind the amount of 'genuine' errors triggered in the above:

First, OCaml developers have fixed a number of defects and released 5.1.0, 5.2.0, 5.2.1, and 5.3.0~beta2 over this time period
Second, as new compiler features are added and merged they may accidentally introduce new errors
Third, we have added tests and 'sharpened the axe' of the existing ones
Finally, a fix of a bug in 5.1 merged into 5.2 doesn't prevent the bug from continuing to show up on 5.1 CI runs

It is clear from the plot that both genuine issues (categorised as 'genuine') and false alarms (categorised as 'other') have decreased over this period. The workflow alarms were initially dominated by false ones, then started being dominated by genuine (and 'ci') ones, and have now settled on a level with zero alarms being a common outcome. Such a noise-free test-suite signal is also central for OCaml compiler developers to utilise the test suite in their own development workflow. Here a test-suite red light should ideally signal a problem with a proposed runtime change, rather than (a) false alarms adding needless noise ("oh, never mind that red light!") and (b) true alarms adding genuine noise ("actually that's an existing unfixed issue, not your fault").

To understand the failures triggering on specific versions, here's first a plot of the outcomes for the weekly CI runs of the upcoming 5.3 release:

The one failure on October 21 happened during Cygwin installation, and was hence not related to the test suite. In comparison, weekly runs of the 5.0 workflows tend to trigger issues. This below plot covers 8 workflows in contrast to 5.3's 12 workflows, as Cygwin, frame-pointer, and MSVC byte/native workflows were added since. Since the 5.0 release is older than 5.3, we also started recording weekly workflow outcomes for it sooner:

The plot clearly indicates that issues are triggered on 5.0 workflows more often that not. Ever since starting to note the cause of such failures in late August 2024, the triggered issues are genuine, and typically include crashes due to parallel access to Buffer, pinpointing a case in Buffer.add_string that flew under the radar of the first Buffer fix included in the 5.0.0 release.

Hidden Costs

We have worked to reach zero false alarms and now generally achieve it across an array of 31 CI workflows. We apply due diligence and have thus developed a workflow of going over the CI red lights to understand and summarise the failures – both the genuine ones and the false alarms. We then keep track of them as issues in the multicoretests repo. This allows us to easily refer to them and spot trends, such as starting to see new failures or noticing that a particular failure no longer occurs.

When we spot new errors, we work to reproduce them locally to make sure the issue is genuine. If so, we then report the issue upstream to ocaml/ocaml along with a description of the required steps needed to reproduce it. When understanding the problem well enough, we also contribute with a compiler fix PR. Out of the 40 issues currently identified, Tarides engineers have filed PRs to fix 28 of these, 10 issues have been fixed by others (typically Inria or Jane Street engineers), and 2 issues remain open. Tarides have thus put in a significant effort to resolve errors.

Some Issues are Still Hard to Reproduce

Despite all our efforts to amplify problems and increase reproducibility, some issues are still hard to trigger. One such case was ocaml/ocaml#12707 in which we were able to trigger the assertion failure, albeit rarely. This one took some head-scratching until we realised the problem was caused by a small time window between reading the same atomic field twice in an assertion: di->backup_thread_msg == BT_INIT || di->backup_thread_msg == BT_TERMINATE. This was carried out in parallel with a backup-thread transitioning from BT_TERMINATE to BT_INIT by an atomic write atomic_store_release(&di->backup_thread_msg, BT_INIT), thus creating a tiny chance of neither of the conditions to be true, if the write would happen just in between the two reads. We could then manually insert a call to sleep to confirm, and develop an appropriate fix.

We are currently investigating an even rarer issue triggered by the Gc tests, that causes a rare crash on macOS running on an ARM64 processor and seems to require just the right OCaml heap conditions to trigger. In both cases, despite not triggering on every PBT run, the randomised tests have nevertheless highlighted genuine issues that would otherwise only show up even more rarely on the ocaml/ocaml test suite or – worse – to end users of OCaml.

Lowering the Barrier to Entry for `multicoretests` for Compiler Engineers

To offer compiler engineers the ability to run the test suite easily, we have made it possible to do so by labeling a PR with a run-multicoretests tag. Recently a PR to improve major GC performance with mark-delay kicked off using the run-multicoretests tag and is already making good use of the test suite as it detected an issue with the GC marking of Ephemerons. With the test suite finding an issue in another GC improving PR, we are confident in the value addition that the test suite brings to compiler developer in helping quality-assure runtime-related PRs.

Usage Outside `multicoretests`

The usage of STM has spread outside of multicoretests. In Saturn, a library of lock-free data structures for OCaml 5, both existing and new data structures come with STM tests to help ensure their correctness. This started in connection with moving experimental tests of ws_deque out of the multicoretests suite and has continued since. In Picos, a library for composing effect-based schedulers, STM tests also help ensure correctness of its underlying data structures.

For the Gospel specification language for OCaml, Tarides has worked to develop Ortac-QCheck-STM, as a plugin for Ortac. This is a tool to extract sequential STM tests from a Gospel specification, thereby putting the strength of PBT in the hands of OCaml developers willing to annotate their interfaces with Gospel specifications. This effort is paying off, as the tool is starting to find genuine issues.

Conclusion

Using PBT to test OCaml 5 started as a blue-sky project at Tarides. Despite a range of challenges, the test suite has nevertheless reached a point where a run yields a clear signal free of false alarms and worthy of a confidence increase in a compiler code change. Getting here was a team effort with contributions from Charlène Gros, Samuel Hym, Olivier Nicole, Nicolas Osborne, and Naomi Spargo, along with patience and effort from OCaml compiler engineers working to fix our reported findings.

The PBT approach has successfully pinpointed a range of issues across the Stdlib, the rewritten OCaml 5 runtime system, and the restored backends. We hope that the test suite can continue to do so and thereby help maintain OCaml's reputation as a rock solid and safe platform.

Dmitrii Kovanikov

2024-12-20T00:00:00-00:00

Understanding why associativity matters

Tarides

2024-12-18T00:00:00-00:00

OCaml is a valuable and powerful tool, combining performance, security, and reliability. Joining the ecosystem, community, and learning the language can help you make the most of its capabilities if you're not already familiar with it. But how can we make learning OCaml easier? As with other functional programming languages, its visibility can be limited by factors such as your educational and professional environment, or the programmers you follow. Nevertheless, the world of software development is increasingly recognising the strengths of functional programming in general and OCaml in particular.

To meet the growing interest in OCaml, we aim to make learning the language as straightforward as possible. For individuals, we’re prioritising tutorials, books, and events centred around OCaml. For teams and organisations, we offer OCaml courses for both basic and advanced training.

The Best Ways to Learn OCaml: For the Independent Learner

OCaml.org

Whether you’re looking to write a ‘hello world’ or delve into the details of the OCaml compiler, the central resource is OCaml.org. The Learn area provides a range of resources, including tutorials, exercises, and a directory of books and academic papers, catering to different skill levels. The page also offers ways to install the latest version of OCaml, access the manual, and explore OCaml’s documentation. The team has taken inspiration from other languages like Python, Rust, and the functional language Haskell to improve the resources available to learn OCaml.

These resources are tailored to the independent learner, using their free time to pursue knowledge, or the teacher needing materials for their students. We support the continuous improvement of tutorials and documentation on OCaml.org, and the accessibility of books, papers, and other learning materials. The pages are also open-source, meaning that everyone is welcome to contribute!

Books

If you prefer learning from books, there are several available to choose from. For beginners, OCaml Programming: Correct + Efficient + Beautiful and OCaml From the Very Beginning are excellent choices. The former is a textbook used at Cornell University, complete with a video playlist. Intermediate learners may find Real World OCaml and More OCaml: Algorithms, Methods, & Diversions beneficial. Real World OCaml is available for free download thanks to our sponsorship of its gold open access release.

Community

OCaml’s robust open-source ecosystem significantly aids newcomers by making source code freely available for study, and offering access to experts and feedback through the GitHub repository and Discuss forum.

There are many more ways to engage with the OCaml community online:

Social media: head over to the Community page on OCaml.org for an overview of the different places online people meet to discuss, develop, and blog about OCaml. Resources include more or less formal forums, educational text- or video-based content, entertainment, social networking, and live chats.
Advent of Code: An advent calendar of programming puzzles, suitable for a variety of skill levels and compatible with all languages, popular with programmers as a way to challenge themselves and learn new languages. This year, Sabine has organised a leaderboard for people solving the advent of code in OCaml.
The OCaml Planet: An RSS feed aggregator hosted on OCaml.org which shares articles and blog posts on everything OCaml. Follow it to stay up-to-date on the latest OCaml news, technical deep dives, open-source project updates, and developer insights.

Events

We also host and sponsor community events like hacking days and conferences to further support users who want to learn more about OCaml. Upcoming events include:

The OCaml Workshop: Continuing in a well-established tradition, the OCaml Workshop will be held at the 2025 ICFP Conference. With a more academic focus, talks start out as papers which are then presented as part of the OCaml track.
The FUN OCaml Conference: Returning for a second time in 2025, the new OCaml conference is organised by developers for developers and covers topics big and small.
Reason OCaml India Meetups: A community of Reason and OCaml enthusiasts that organise regular in person and online meet-ups.

There are of course many more events organised outside of those we host and sponsor, and some notable examples are Functional Conf and OUPS. We recommend that you check out the Events page to get an overview of all the great OCaml events you could join!

The Best Ways to Learn OCaml: Tailored Training in OCaml

In large organisational teams, relying on individual learning or the dissemination of knowledge from a few experts can be time-consuming and inconsistent. Tailored courses enable simultaneous training for the entire group, fostering collaboration in tackling new challenges. To make learning OCaml easier for groups, we’ve created courses to enhance team competence in the language. Here’s what our OCaml courses offer.

Our Foundational Course Starting with OCaml: An Introduction

This course teaches core OCaml programming concepts through theory and practice, and has been designed to facilitate a smooth transition for engineers from other languages, including imperative programming. It covers OCaml’s history, design choices, and key functional programming concepts such as functions, recursion, and higher-order functions alongside critical concepts like tuples, the OCaml type system, type inference, pattern matching, and polymorphism.

The bulk of the course comprises several modules that introduce the practical building blocks of programming with OCaml. This includes methods: imperative and modular programming, data structures, error handling, interfacing with C, and command-line parsing. It also offers guidance on tooling: opam, Dune, VSCode, debugging and profiling, concurrent programming with lwt, and testing using expect tests and QuickCheck.

Finally, the course would not be complete without hands-on experience writing OCaml code. The modules on setting up the OCaml environment, building, documenting, and releasing an OCaml project culminate in an entire day spent on building an OCaml application from scratch. Participants will leave with a solid understanding of OCaml, the necessary tools for development, and practical application-building experience. Following this foundational course, your entire team will be familiar with the OCaml language and workflow.

Our Higher-Level Course Mastering OCaml: Advanced Techniques

The advanced course is designed for experienced users and teams looking to enhance their skills. It consists of three sections: advanced OCaml techniques, advanced methods for common OCaml tools, and advanced tooling.

Advanced OCaml techniques make up the bulk of the course, including generalised algebraic data types (GADTs), multicore programming with OCaml 5, and transitioning from OCaml 4 to OCaml 5. Advanced methods for common OCaml tools cover using the Dune build system, and PPX preprocessors.

Topics in advanced tooling include: the MirageOS library operating system for secure, high-performance, network applications; web development with JS_of_ocaml, Wasm_of_ocaml, and Dream; concurrent programming with [Eio], and using property-based testing to evaluate programs. This course aims to equip teams already using OCaml with new skills and tools. An advanced course with deep topics, the course is tailored to focus on relevant areas for your team, taking them to the next level.

Finally, one of the biggest benefits we offer with our courses is customisation. We understand that teams have unique challenges and goals, and offer you the option to create a course that fits your needs. Not only can you pick a selection of topics from the advanced OCaml modules, but you can completely tailor the training with a customised itinerary. This is negotiated on a per-instance basis, and you can schedule a free consultation to discuss what that could look like for your teams.

Get in Touch

Need help learning OCaml? Reach out to the community on Mastodon, X, or LinkedIn to get helpful pointers or ask questions on Discuss. If you’re interested in our courses for your teams, you can get a free consultation to discuss whether the curriculum is right for you and sign up to our mailing list to receive the latest news.

We look forward to hearing from you, please don’t hesitate to get in touch!

Caml Weekly News

2024-12-17T12:00:00-00:00

Tarides

2024-12-11T00:00:00-00:00

The first version of the Saturn library is out! Saturn is a new OCaml 5 library available on opam, which offers a collection of well-tested, benchmarked, and efficient concurrent data structures ready to be used with OCaml Multicore. Access to concurrent-safe data structures saves developers from the time-consuming and often error-prone process of designing their own.

This post will give you an overview of the library, its main features, and some use cases. The team encourages you to try the data structures and share your feedback in the repo and on Discuss. After you're finished with this post, I also recommend you read the paper on Saturn from the OCaml workshop at ICFP 2024 if you want more details. Let’s dig in!

What is Saturn?

The Saturn repository is made up of one package, saturn, containing all lock-free data structures. You can use Saturn with OCaml 5.2 or later, and it can be installed from opam with the command opam install saturn.

Saturn covers many use cases, from simple stacks and queues to more complex structures, including skip lists, hash tables, work-stealing deques, and more. All of them have been adapted to be compatible with and take advantage of the OCaml 5 memory model. For example, the team behind Saturn had to rework the Michael-Scott queue to avoid memory leaks.

To improve performance, they introduced several micro-optimisations, including preventing false sharing, adding fenceless atomic reads where possible (improving performance on ARM processors), and avoiding extra indirection in arrays and atomics to reduce memory consumption. While optimising the library, their work highlighted some missing features in OCaml 5 and led to upstreamed improvements to the language, such as padded atomics and fixing a CSE bug.

Why Saturn?

Sharing data between multiple threads or cores is a well-known problem in computer science. The most obvious solution is to use a sequential data structure protected by a lock, but this approach can introduce performance overhead due to contention between locks. Liveness issues like deadlock, starvation, and priority inversion are also associated with locks in parallel programming.

The opposite approach is to use a lock-free implementation, relying on fine-grained synchronisation instead of locks. This approach benefits from higher performance and guarantees system-wide progress. However, it does come with its own set of bugs, including the ABA problem (which is largely mitigated in garbage-collected languages), data races, and unexpected behaviours as a result of non-linearisability.

In this quagmire of pitfalls, Saturn stands out as a source of reliable, tested data structures that the user can adopt for their own projects. Developers can pick from a variety of structures knowing that they are suitable and safe, saving them time and making OCaml 5 easier to use.

Benchmarks

The library is still relatively new, so more benchmarks are still to come, but there are some numbers to give you an idea of performance. You can find tables showing the throughput of different queues and stacks on single and multiple domains in the paper from the OCaml Workshop at ICFP 2024 about Saturn. The tests reveal that Saturn implementations either outperform or, in the case of simpler implementations, match the performance of non-Saturn structures. However, the benefit of using Saturn, even in cases where another simple implementation exists, is that the Saturn data structures have built-in optimisations and synchronisation safeguards, and it’s up to each individual developer to decide what that’s worth to them.

The library also has a benchmarking command make bench that can be run from the root of the repository to run a standard set of benchmarks. It outputs in JSON and is intended to be consumed by current-bench.

Tests

Testing Saturn’s structures was a high priority for the team and crucial to ensure the safety, linearisability, and lock-freedom where expected. Saturn was tested using two primary tools: DSCheck and STM. STM is used for both unit testing and linearisability. It can automatically generate random full programs using the API provided, and for Saturn, this is a data structure. It then executes the programs in parallel with two domains and checks all the results against the post-conditions of each function, providing unit testing. STM performs a sequence of random commands in parallel, records the results, and checks whether the observed results can be linearised and reconciled with some sequential execution.

DSCheck is a model checker designed to compute all the possible interleavings of instructions between multiple domains and verify that each one returns the expected result. This method is useful for detecting bugs that only occur in rare circumstances which would otherwise be hard to catch. DSCheck can also be used to verify that a program is lock-free, as it will fail to terminate if any form of blocking is present in the program. The DSCheck implementation has been optimised to make the tests efficient even on Saturn’s more complex data structures.

Formal Verification

The verification work for Saturn is done at Inria, forming part of Clément Allain's PhD work there. Due to the notoriously finicky behaviours of lock-free algorithms, they have formally verified part of Saturn’s data structures and aim to keep going until they’ve covered the entire library. The main criterion for correctness in concurrent data structures is linearisability, which requires each operation in a data structure to appear to take effect instantaneously at some point during its execution (called the linearisation point) so that all linearisation points from all operations form a coherent, sequential history. The team at Inria are formally verifying that this linearisability is correctly present in Saturn's data structures.

To verify linearisability, they use the mechanised concurrent separation logic IRIS, which has been used in the past to verify realistic data structures. All proofs are formalised in Coq and available on GitHub. They manually translated the original code from Saturn to a deeply embedded language in Coq, hoping to potentially automate the process in the future. Work continues to verify data structures for Saturn, ensuring safe and predictable behaviours.

Stay in Touch!

To delve deeper into everything Saturn, I recommend you watch the ICFP 2024 presentation from the OCaml Workshop. You can try Saturn for your projects via the repo, and don’t forget to share your feedback with the team!

Tarides

2024-12-04T00:00:00-00:00

In part one of our series on Notafs, we outlined the motivations and challenges behind creating a file system for MirageOS tailored to the SpaceOS use case of storing large files on disk in satellites. Furthermore, Notafs can be used to run the irmin-pack backend of Irmin, which, in turn, provides users with an amazing file system for their MirageOS projects.

In this post, we delve into more detail concerning the design choices behind Notafs and provide some benchmarks and visualisations to give you a better understanding of the file system. Let's dig in!

Big Design Decisions: Copy-on-Write

The Limits of File Systems

In our previous post, we mentioned some of the expectations of file systems to resist corruption and data loss. Ensuring these conditions are met can be challenging since hardware exhibits many failure modes. For example, it is impossible to check if a disk has physically persisted a write operation. In order to improve performance, disks use RAM buffering to quickly store the write requests before committing them persistently (which is a much slower process). Short of unplugging the disk and waiting for its internal battery and RAM to clear before rebooting, we can't know that the writes were executed and must be ready for the worst-case scenario: unwritten, partially written, or corrupted data regions on the disk! Writes could also be reordered, leading to consistency issues.

Furthermore, the way that disks are structured presents its challenges. Disks are organised into blocks of fixed sizes, like 512 bytes, 1 kb, or 4kb, depending on the model. A block is the smallest unit we can read or write, so even if we only want to update a few bytes on the disk, we still need to re-write the full surrounding block data. Different disk models can have additional constraints, such as:

Hard drives that favour contiguous block reads and writes to ensure high performance since spinning their head to random locations is expensive.
SSD exhibits faster wear if one block is updated more often than the others, leading to a shorter device life than what's expected under uniform usage.

All About Copy-on-Write

Considering these factors, Notafs was designed as a copy-on-write file system. This strong design principle means that when we update the file system, we never override a block in place but rather write the new data block in a new, unused location on the disk. Preserving the old block means that Notafs maintains access to a backup until the block is reused. If a write operation fails, for example, in case of a power outage in the middle of multiple block updates, Notafs can recover information from the old block.

This strategy does not impact performance since writing the new data in a new location or in place is virtually indistinguishable from the disk's perspective. From an implementation standpoint, copy-on-write is very similar to how purely functional data structures work to provide fast updates. Only modified data needs to be copied, so fast updates are still possible, as sharing unmodified content between revisions is free.

To implement the copy-on-write design, we needed to keep track of the unused free blocks where writes can be performed. This is akin to the challenge of implementing a memory allocator, but it is made slightly simpler because we only need to allocate fixed-size blocks. Our block allocator is backed by a queue containing the list of free block identifiers. After formatting a disk, the queue is (virtually) initialised to contain all block locations except a couple of reserved blocks. When a new block needs to be written to the disk, we pick a free block location from the queue where the write should take place. Filesystem operations are carefully implemented to free old blocks which aren't reachable anymore by pushing their block identifier location back onto the queue. Once the disk is full, these free blocks will eventually be reused to store new data. At the same time, old data is kept intact and can be used as a backup if needed. To avoid complications in determining if and when a block can be freed, we enforce strict linear ownership of blocks so that we do not need any reference counting or advanced garbage collection.

The queue of free blocks is part of the file system state and must be recoverable when we boot it. This is why the queue state itself is backed by a copy-on-write data structure, which is stored on disk! To ensure good performance, the block locations that have been freed during a single file system operation are sorted before being pushed to the queue. This process favours the allocation of contiguous block locations, leading to faster writes on hard drives. Consecutive block locations are also compressed in the queue by representing them as intervals instead of listing all the intermediate block locations.

When the disk gets full, the previously freed blocks will eventually be reused, storing new data. When that happens, the queue guarantees that we use the blocks that have been free for the longest time so that only the oldest backups become unrecoverable. However, since reusing blocks can lead to data consistency issues if some writes fail to be committed on disk, we needed a way to detect whether a block contains the expected latest data or the old partially-written/corrupted data. To this end, we use a checksum of the stored data to validate each block. We could have stored the checksum in the block next to the data it validates, but it would only have allowed for the detection of corrupted data and not for detecting whether the block's data is old since the old checksum remains valid. So, instead, we store the checksum in the parent block next to the block location it validates. When a parent block needs to read one of its child block's data, it uses its checksum to verify that its child data is as expected.

Final Considerations

There was one last issue for our team to tackle regarding the design: if everything is copy-on-write and the new filesystem root is written to a new block every time, how do we find the location of the latest root when the unikernel boots? We wanted to avoid scanning the whole disk in search of the latest root. Hence, when formatting a new disk, we reserved a fixed number of blocks at known locations. After each file system update, one of these blocks is updated in place to point to the latest root allocation in a round-robin fashion, with a generation counter to help our application determine which one was written last.

When the filesystem starts, it identifies the latest root from the reserved blocks and traverses all of its reachable data to check each block checksum and validate that the whole filesystem is in a consistent state. If an issue is detected, implying the disk wasn't shut down properly, Notafs will attempt to use the previous filesystem root found in the previous reserved block. This procedure effectively rolls back the failed write operation until a valid filesystem state is found (or the disk is deemed unrecoverable!). To avoid SSD wear from updating the reserved root blocks too often compared to other blocks, we have added an extra, dynamically allocated level of indirection to the previous explanations, which amortises the number of updates to the reserved blocks.

Finally, we implemented several optimisations to the above scheme to best use runtime resources. For example, we save disk space by representing block locations with as few bytes as possible, depending on the disk size. A 'Least-Recently-Used (LRU) cache keeps track of the latest reads and bufferises the incoming writes. This LRU enables Notafs to run with very low memory overhead, even when a file system operation would have required more RAM if another system had performed it.

We are very happy with this design because it combines simplicity with high data consistency guarantees. We used the setup to implement a copy-on-write rope data structure to represent files of any size with fast updates and read operations. We added a rudimentary hierarchical structure of folders and files on top to enable multiple file usage. This last part has not been optimised and is only suitable for applications with a low number of files. It will get slower the more files you have, but there are no hard limits.

Benchmarks and Tests

Since filesystem correctness is critical, we tested all steps of the project to validate our solution. Of special interest is the final verification that our team realised with the Gospel specification language using Ortac to translate the specification into a QCheck test suite from the formal spec. By comparing the behaviour of our Notafs file system with an obviously correct in-memory reference model, we could stress-test our system by simulating a huge number of tests to detect discrepancies. Our experience with Gospel made a strong case for it being the future of documentation, with high-level specifications that are both mechanically verifiable and readable by end-users.

Ultimately, correctness alone was not enough to justify using Notafs in production if it did not meet our performance goals. We benchmarked Notafs in comparison to existing MirageOS file systems for the features that were critical for SpaceOS and Irmin support. Those features were the ability to write large files, to read back those files, and to read only a sub-range in those files (a feature required by irmin-pack).

While the TAR file format will always be the fastest thanks to its simplicity, it comes with the limitation that files can't be removed or updated (hence eventually running out of disk space!). In general, the benchmarks confirm that Notafs meets its goal of handling large files efficiently. In particular, its careful use of memory with LRU enables Notafs to run the benchmarks on larger files than the alternative – for unikernels with restricted max memory usage. Note that benchmarks are always subject to interpretation; we did not benchmark the scenarios where we expect Notafs to perform the worst, e.g. with a large number of small files!

Our next image is actually a video showing Irmin running on top of a small Notafs disk. It combines everything we have explained so far: (epilepsy warning: the video displays flashing lights)

The disk is divided into blocks of 1024-sized bytes, conveniently represented by 32x 32px squares. The blocks in colour contain live data used by the Irmin store, while the grey crossed-out ones are the free blocks. The colourful blocks create distinctive patterns that indicate the different types of data that Irmin uses to store the database.
As more Irmin operations occur during the test's execution, you can see the Notafs block allocator in action, circling around the disk to write new data to the oldest freed block.
At the bottom of the video, two histograms show the distribution of the reads and writes on each block to check that fair usage of each block is achieved.
Finally, we can see the Irmin garbage collector clearing its old history on a frequent basis to avoid running out of disk space.

This visualisation was made possible because MirageOS libraries can also be used on standard operating systems without requiring special compilation by a unikernel. In other words, Notafs also enables Irmin to be used as a single-file database (like sqlite, which might be of interest for some applications' workflows. We want to do more work on this front to lift the limitation that the database size must currently be fixed in advance when formatting the disk.

Conclusion

Developing a new filesystem has been a very exciting project, and we hope that you'll enjoy the new options when developing unikernels with MirageOS:

Notafs: for applications which require a limited number of very large files (e.g. SpaceOS satellite pictures).
Irmin on Notafs: for a general-purpose file system, including optimisations to support a large number of arbitrarily sized files and many advanced features (with the caveat that this version of Irmin requires OCaml 5, which is still experimental on MirageOS).

We invite you to participate and try Notafs yourself! Please consult the open-source Notafs repository for further information, and share your experience (good and bad!) with the team. You can stay in touch with us on X, Mastodon, Threads, and LinkedIn.

Caml Weekly News

2024-12-03T12:00:00-00:00

Tarides

2024-11-27T00:00:00-00:00

We are pleased to announce one (or two) new filesystems for MirageOS! The motivation behind creating them is an exciting new use case requiring the system to store data on disk.

MirageOS allows you to compile OCaml applications into unikernels. By selecting the operating system functionalities required, a unikernel can be constructed for your application using only the necessary components. This process reduces the attack surface and increases the hardware efficiency of your final application. MirageOS unikernels can be deployed to various cloud and mobile platforms.

As a case in point, Tarides is developing SpaceOS to run unikernels on satellites, an industry where both security and performance are critical. While MirageOS has a long history of cloud usage and comes with advanced network capabilities, SpaceOS provides an interesting use case for disk storage. A possible use case for satellites is to take pictures from space and send them to Earth for analysis, and since satellites may not be able to send the pictures right away (due to their location, for example), storing high-resolution images on a disk is a must.

So, naturally, the question we posed ourselves was what MirageOS file system could we use for SpaceOS? And that started us down the journey to the new file system Notafs, that even lets you use Irmin on top!

Why Create a New Filesystem?

At the start, there were a couple of existing filesystems available for MirageOS that we needed to evaluate:

The Chamelon filesystem is designed to efficiently support a large number of very small files on a disk.
The Docteur filesystem which provides read-only file compression.
The FAT filesystem, which provides support for the (old) standard FAT16 filesystem with restrictions on file sizes.
The TAR file format, which has the limitation that users can only add new files: deletion or modification of existing files is not supported.

All of the above libraries are well-tested and highly recommended if they fit your application needs! But, as we will illustrate in part 2 with our benchmarks, none of these systems satisfied the SpaceOS requirement to store large files.

Furthermore, using a conventional file system like the ones available on traditional operating systems like Linux and Windows was also not an option since they are closely tied to large operating system functionalities. This is because to ensure higher security, MirageOS libraries are built using the OCaml programming language, which provides strong memory-safety guarantees. While this is a fantastic design choice for the future of computing, it is harder to support a conventional filesystem (programmed in C/C++) natively on MirageOS. A unikernel based on that kind of technology would lose the appeal of a small software stack with high-security guarantees. In comparison, the Notafs solution fits into four thousand lines, which is a much more ‘human-sized’ project to review.

The Challenges With Filesystems

Implementing a filesystem is a complex task requiring compromises, which explains the lack of options for storing large files with the SpaceOS project. For example, even though file systems may appear simple, with an interface that everyone is familiar with, their critical main function is to protect applications from (recoverable) hardware defects. Unless the disk dies, the user should never lose or see corrupted data, even if a power outage was to interrupt the filesystem in the middle of an operation. In other words, a filesystem update should have transactional semantics: either the operation succeeds, or it does not, but applications should never observe an in-between broken state. Without this property, software built on top of the file system would be vulnerable to experiencing faults.

Tarides is well aware of the challenges involved with implementing a filesystem – we maintain the Irmin database, which provides a hierarchical key-value store with high data consistency guarantees, git-inspired history for rollbacks, and distributed replication over the network. While the Irmin API resembles a filesystem, it is a full-blown database and provides many more functionalities. So rather than starting a new filesystem implementation from scratch, we asked ourselves whether we could reuse the Irmin database as a filesystem for MirageOS.

Irmin as a Filesystem?

Using Irmin for this purpose is not a new idea, and Irmin already provides multiple backends that allow users to run its databases on various platforms with different constraints. The irmin-pack backend was of particular interest for MirageOS support. We have spent years optimising its performance since this backend is notably used to store the Tezos blockchain, and it enables the freeing of disk space by truncating the database history to get rid of unnecessary old backups.

Finally, the low-level implementation of irmin-pack is especially suited for a port to MirageOS, as it only requires support for a few large (append-only) files from the operating system.

This last technical requirement was especially relevant to the SpaceOS use case, where the satellite needs to be able to store high-resolution pictures on disk. If our custom-made file system supported large files, then we would be able to support SpaceOS and enable the general-purpose use of Irmin for MirageOS unikernels. This realisation still left us with the task of developing the foundations of a file system, but even just the bare functionalities would satisfy our use cases.

Notafs is Born

We called this new file system ‘Notafs’, which, as the name suggests, is not a general-purpose file system due to its focus on handling a few large files. It is designed to handle a small number of large files for Mirage block devices. It can, however, be used to run the irmin-pack backend, which gives users all the benefits of an Irmin filesystem for MirageOS. Together, running irmin-pack on Notafs lifts the limitations of the latter, and supports many file names, is optimised for small and large files, and includes a git-like history with branching and merging just to name a few features!

Navigating design restrictions is an excellent way to focus the efforts of a project. In our case, that meant directing them towards the correctness and performance of the few selected operations of Notafs. We didn’t need to implement advanced filesystem operations since missing functionalities could be provided by irmin-pack, including optimised management of many small files. As long as our underlying file system could provide fast operations on large files, the rest was taken care of!

From a unikernel developer’s perspective, Notafs provides an implementation of the Mirage_kv interfaces. This is the standard API for filesystem usage on MirageOS, so any existing unikernel can use it without needing to change its application code. We also provide a command-line interface to simplify the formatting of disks and to allow for external inspection of file system contents of the unikernel. Check out this example of a minimal setup of Notafs in MirageOS.

While Notafs has restricted functionalities, it provides a nice developer experience and a useful alternative in the file system design space for MirageOS. We are especially proud of the safety guarantees that Notafs provides for your data stored on disk.

Using Irmin on Notafs

To enable users to run Irmin on MirageOS, we provide an implementation of the syscalls required by the irmin-pack backend on top of Notafs. This was straightforward as Notafs provides the functionalities required to run irmin-pack. Thanks to OCaml functors, the irmin-pack codebase was already structured for alternative syscall implementations.

One OCaml subtlety we encountered was handling asynchronous I/O: MirageOS and Notafs use the Lwt monad, while the Irmin syscalls expected a direct-style I/O implementation. This would have been an issue in the past, but we could bridge that gap using the effect handlers that came with OCaml 5. Note that OCaml 5 support is still experimental on MirageOS at the time of writing!

From a user perspective, using Irmin addresses the design limitations of Notafs (while keeping its desirable consistency properties) by adding efficient support for managing a large number of small files. Irmin is a much more general-purpose file system; on top of that, a user can build a unikernel application with many additional features. We’re hopeful that applications already using Irmin will consider this new bridge for targeting MirageOS unikernels.

Until Next Time!

Thank you for checking out the first part of this two-part series where we’ve introduced Notafs, the motivation behind the file system, the challenges, and its use cases. Look out for part two where we delve into the details behind Notafs’ design alongside benchmarks and visualisations of the file system in action.

Connect with us online on X, Mastodon, Threads, and LinkedIn to stay updated on our latest projects. We also invite you to consult the open-source Notafs repository for more information or to test out the file system yourself!

2024-11-20T00:00:00-00:00

1. Introduction

The Language Server Protocol (LSP) defines a standardised protocol that facilitates communication between an editor (client) and a language server. It is developed by Microsoft and was designed to simplify the process of adding language support to different code editors by abstracting away the most common implementation details of the programming language.

OCaml-LSP, which relies on Merlin's amazing engine, is an implementation of the LSP protocol for the OCaml programming language.

2. Code Navigation in LSP

When it comes to navigating through code, the LSP protocol provides four ways to do this:

Goto Declaration Request: Resolves the declaration location of a symbol.
Goto Definition Request: Resolves the definition location of a symbol.
Goto Type Definition Request: Resolves the type definition location of a symbol.
Goto Implementation Request: Resolves the implementation location of a symbol. Not supported in ocaml-lsp.

While these goto requests provide general navigation for most programming languages, it falls short when dealing with complex syntax such as OCaml's modules, functors, etc., and being able to navigate to specific sections.

3. Code Navigation in Merlin

Merlin offers a range of commands that allow precise code navigation, including the ability to jump to specific constructs like functions, module definitions, and match cases. As earlier seen, generalised navigation in LSP is insufficient for precise movements. To solve this, Merlin introduces a specialised Jump command that allows users to jump to specific targets within their OCaml code.

The key targets include:

fun: Jumps to a function definition.
let: Jumps to a let binding.
module: Jumps to a module.
module-type: Jumps to a module type definition.
match: Jumps to a match construct.
- match-next-case: Jumps to the next case in a match statement.
- match-prev-case: Jumps to the previous case in a match statement.

4. Custom Requests in LSP

When standard LSP requests are insufficient, we have the possibility of writing custom requests to implement the functionality we want. The downside to this approach is that we lose native support in the various editors or clients and will also have to write the resulting client implementations for each custom request. Implementing precise navigation in OCaml-LSP using Merlin's Jump is a typical use for a custom request.

For this implementation, our focus was on being able to use the requests already available on LSP and use them in non-typical, innovative ways. With this in mind, our solution works by using a CodeAction in combination with a ShowDocument request to move the cursor to our desired position.

5. Implementing Merlin's Jump in OCaml-LSP

Code action requests are used to execute commands for a given text document and range. These commands are typically used to change the state of a document, such as code fixes and beautifying or refactoring code. In general, we use code actions to perform quick edits in code. It's a bit unusual to think of code actions as a navigation tool, but with some smart workarounds, we can indeed use code actions to move through code. Here's how the feature works:

a) Document Detection: When a source file is opened in the editor, the OCaml-LSP server checks if it is an OCaml file supported by Merlin. If the document is incompatible, the "Jump to Target" functionality is not provided.
b) Client Capability Check: The LSP allows servers to query what features a client (the editor) supports. For "Jump to Target" to work, the server checks if the client supports the ShowDocument capability, which is necessary to move the cursor to the correct location.
c) Generating CodeActions: The client begins by sending a request for all valid code actions:

[Trace - 06:21:32] Sending request 'textDocument/codeAction - (71)'.
Params: {
    "textDocument": {
        "uri": "file:///.../test.ml"
    },
    "range": {
        "start": {
            "line": 5,
            "character": 12
        },
        "end": {
            "line": 5,
            "character": 12
        }
    },
    "context": {
        "diagnostics": [],
        "triggerKind": 2
    }
}

The server receives this request, and for each target (fun, let, match, module, etc.), it asynchronously queries Merlin's Jump command. It assembles the possible jumps into a list of code actions and returns this list back to the client.

[Trace - 06:21:32] Received response 'textDocument/codeAction - (71)' in 7ms.
Result: [
    ...,
    {
        "command": {
            "arguments": [
                "file:///.../test.ml",
                {
                    "end": {
                        "character": 2,
                        "line": 9
                    },
                    "start": {
                        "character": 2,
                        "line": 9
                    }
                }
            ],
            "command": "ocamllsp/merlin-jump-to-target",
            "title": "Let jump"
        },
        "kind": "merlin-jump-let",
        "title": "Let jump"
    }
]

Given that code actions are a standard LSP functionality, we benefit from an already existing pool of client implementations, so the clients display the codeactions returned by the server without any coding or customisation. Each CodeAction includes a:

Title (e.g., "Jump to function").
Command (ocamllsp/merlin-jump-to-target): A command that is executed when the code action is selected.
Kind: This parameter distinguishes various code actions and can be used to group similar code actions or differentiate them especially for use with keybindings.
d) Executing CodeAction Commands: When a user clicks or selects a specific code action, the command associated with that code action is executed. The client sends an executeCommand request to the server.

[Trace - 06:37:10] Sending request 'workspace/executeCommand - (73)'.
Params: {
    "command": "ocamllsp/merlin-jump-to-target",
    "arguments": [
        "file:///.../test.ml",
        {
            "end": {
                "character": 2,
                "line": 9
            },
            "start": {
                "character": 2,
                "line": 9
            }
        }
    ]
}

This executeCommand request triggers the server to ask the client to perform a showDocument request using the document URI and location range received from the code action.

[Trace - 06:37:10] Received request 'window/showDocument - (6)'.
Params: {
    "selection": {
        "end": {
            "character": 2,
            "line": 9
        },
        "start": {
            "character": 2,
            "line": 9
        }
    },
    "takeFocus": true,
    "uri": "file:///.../test.ml"
}

Both executeCommand and showDocument are standard LSP requests, meaning all clients that already have LSP support can automatically perform this jump.

e) Error Handling: In cases where Merlin fails to find a valid target (e.g., the target does not exist or the position is invalid), the server gracefully handles the error by omitting the specific code action for that particular target, ensuring a smooth user experience.

6. Feature Demonstration

Below is a demonstration showing the various code actions which perform navigation.

7. Conclusion

Set for release in early December 2024, Merlin's Jump functionality in OCaml-LSP brings precision navigation into the LSP world, enhancing OCaml code navigation capabilities across various editors. By using a combination of existing LSP requests, the feature is implemented in a way that is client-agnostic and fully compatible with LSP, ensuring a smooth and efficient user experience for developers working with OCaml.

For developers working on large or complex OCaml projects, this precise navigation capability will significantly streamline workflows, enabling quick and direct navigation to relevant parts of their code.

While implementing Merlin's Jump as a code action offers a broad client-agnostic solution, there are several drawbacks to this approach. First, it can clutter the code action lists, making them noisy for users who are used to a more streamlined selection. Since code actions are typically used for refactoring or fixing issues, adding navigation-related actions here is not an expected usage of code actions. Additionally, binding these navigation actions to shortcuts becomes harder, as code actions do not naturally lend themselves to precise key bindings for navigation.

To overcome these limitations, one potential solution is to use a custom LSP request, which would allow navigation functionality without overloading the code action list. We have a an open PR on ocaml-lsp repository which introduces this custom request. Alternatively, a client-side tool like tree-sitter could be used to parse the code and generate the necessary jump targets directly in the client, offering a more flexible and efficient solution tailored to the user's editor setup.

8. Resources

Caml Weekly News

2024-11-19T12:00:00-00:00

Frama-C

2024-11-15T00:00:00-00:00

Frama-C

2024-11-15T00:00:00-00:00

OCamlPro

2024-11-13T13:56:36-00:00

Feedback on this post is welcomed on Discuss!

As mentioned in our talk at the OCaml Workshop 2024, we decided to switch to a time-based release cycle (every 6 months), starting with opam 2.3.

As promised, we are very pleased to announce the release of opam 2.3.0, and encourage all users to upgrade. Please read on for installation and upgrade instructions.

Try it!

In case you plan a possible rollback, you may want to first backup your ~/.opam or $env:LOCALAPPDATAopam directory.

The upgrade instructions are unchanged:

Either from binaries: run

For Unix systems

$ bash -c "sh <(curl -fsSL https://opam.ocaml.org/install.sh)"

or from PowerShell for Windows systems

Invoke-Expression "& { $(Invoke-RestMethod https://opam.ocaml.org/install.ps1) }"

or download manually from the Github "Releases" page to your PATH.

Or from source, manually: see the instructions in the README.

You should then run:

opam init --reinit -ni

Major breaking change: extra-files

When loading a repository, opam now ignores files in packages' files/ directories which aren't listed in the extra-files field of the opam file. This was done to simplify the opam specification where we hope the opam file to be the only thing that you have to look at when reading a package specification. It being optional to list all files in the extra-files: field went against that principle. This change also reduces the surface area for potential file corruption as all extra-files must have checksums.

This is a breaking change and means that if you are using the files/ directory without listing them in the extra-files: field, you need to make sure that all files in that directory are included in the extra-files field. The resulting opam file remains compatible with all previous opam 2.x releases.

If you have an opam repository, you should make sure all files are listed so every packages continues to work without any issue, which can be done automatically using the opam admin update-extrafiles command.

Major changes

Packages requiring an unsupported version of opam are now marked unavailable, instead of causing a repository error. This means an opam repository can now allow smoother upgrade in the future where some packages can require a newer version of opam without having to fork the repository to upgrade every package to that version as was done for the upgrade from opam 1.2 to 2.0
Add a new opam list --latests-only option to list only the latest versions of packages. Note that this option respects the order options were given on the command line. For example: --available --latests-only will first list all the available packages, then choose only the latest packages in that set; while --latests-only --available will first list all the latest packages, then only show the ones that are available in that set
Fix and improve opam install --check, which now checks if the whole dependency tree of the package is installed instead of only the root dependencies
Add a new --verbose-on option to enable verbose output for specified package names. Thanks to @desumn for this contribution
Add a new opam switch import --deps-only option to install only the dependencies of the root packages listed in the opam switch export file
opam switch list-available no longer displays compilers flagged with avoid-version/deprecated unless --all is given, meaning that pre-release or unreleased OCaml packages no longer appear to be the latest version
opam switch create --repositories now correctly infers --kind=git for URLs ending with .git rather than requiring the git+https:// protocol. This is consistant with other commands such as opam repository add. Thanks to @Keryan-dev for this contribution
opam switch set-invariant now displays the switch invariant using the same syntax as the --formula flag
The builtin-0install solver was improved and should now be capable of being your default solver instead of builtin-mccs+glpk. It was previously mostly only suited for automated tasks such as Continuous Integration. If you wish to give it a try, simply calling opam option solver=builtin-0install (call opam option solver= restores the default)
Most of the unhelpful conflict messages were fixed. (#4373)
Fix an opam 2.1 regression where the initial pin of a local VCS directory would store untracked and ignored files. Those files would usually be cleaned before building the package, however git submodules would not be cleaned and would cause issues when paired with the new behaviour added in 2.3.0~alpha1 which makes opam error when git submodules fail to update (it was previously a warning). (#5809)
Fix the value of the arch variable when the current OS is 32bit on a 64bit machine (e.g. Raspberry Pi OS). (#5949)
opam now fails when git submodules fail to update instead of ignoring the error and just showing a warning
opam's libraries now compile with OCaml >= 5.0 on Windows
Fix the installed packages internal cache, which was storing the wrong version of the opam file after a build failure. This could be triggered easily for users with custom repositories with non-populated extra-files. (#6213)
Several improvements to the pre-built release binaries were made:
- The Linux binaries are now built on Alpine 3.20
- The FreeBSD binary is now built on FreeBSD 14.1
- The OpenBSD binary is now built on OpenBSD 7.6 and loses support for OpenBSD 7.5 and earlier
- Linux/riscv64 and NetBSD/x86_64 binaries are now available

And many other general, performance and UI improvements were made and bugs were fixed. You can take a look to previous blog posts. API changes and a more detailed description of the changes are listed in:

This release also includes PRs improving the documentation and improving and extending the tests.

Please report any issues to the bug-tracker.

We hope you will enjoy the new features of opam 2.3!

Tarides

2024-11-13T00:00:00-00:00

FUN OCaml describes itself as a "two day open-source hacking event dedicated to OCaml enthusiasts and developers around the globe". This past September, 70+ developers descended upon a coworking space in Berlin, Germany, to learn, socialise, and hack together. Some had travelled from as far away as Brazil and the United States to join the event in person, and some combined it with React Alicante and ICFP for a summer filled with functional programming.

As a key sponsor - alongside Ahrefs and Jane Street - Tarides is striving to make OCaml more accessible. I have had the pleasure of interviewing one of the head organisers Sabine Schmaltz about her experience at the conference, the aim of the event, and the future of FUN OCaml. Let's dive in!

Why FUN OCaml

FUN OCaml began as a small idea that grew thanks to great teamwork; as Sabine recalled, "David Sancho from Ahrefs was thinking of organising an online conference and had already done a lot of work to plan the talks, etc. I reached out to explain that I was planning an in-person conference and asked whether we could join forces – and he immediately said yes!" It's great to see collaboration across companies benefiting the community, and several additional volunteers would join the organising committee as the conference drew nearer.

I was curious to know what motivated Sabine to organise the event, and she explained that:

"I felt that OCaml was missing a bigger event that was targeted uniquely at developers. FUN OCaml gives them a space to teach each other things and build connections, bringing maintainers and contributors together. FUN OCaml is organised by developers for developers, covering things they are interested in on a peer-to-peer basis".

She was inspired by ZuriHac, a yearly Haskell event in Zurich that she has attended before. "It has a great spirit; people go there to hack and hang out by the lake. It's a very social event with many different rooms and discussions". Inspired by ZuriHac, Sabine adopted an alternative approach to scheduling for FUN OCaml in comparison to traditional conferences.

"We didn't stop at 6 pm and send everyone away! We stayed open until midnight on the first day and at 10 pm on the second. I wanted to provide a space in the evenings for activities, whether OCaml-related or not. Letting people put faces to names and form relationships was important because it's crucial for collaboration". The evenings were filled with board games, hacking, hanging out, and karaoke – creating a space for socialisation and mingling. Sabine pointed out that when conferences close after their programs end, people generally stick to groups of people they already know to hang out with in the evening, which prevents them from making new connections. Consequently, creating a social space for everyone was a high priority for the organisers and a core motivation behind the conference.

Talks, Workshops, and More!

So, what was a day at FUN OCaml like? Both days ran a track for talks and a track for workshops. The talks were live streamed and are [available online for you to watch]. Recording them meant that participants who went to a workshop could catch up on the ones they missed after the fact. The talks covered MirageOS, GADTs, odoc, type engineering, OCANNL, and learning OCaml with Tiny Code Xmas, just to name a few! It was a great mix of more and less technical topics. Sabine recalled the software engineer and popular live streamer Dillon Mulroy's presentation: "Dillon gave a great talk on his journey to OCaml, like a personal account of how he came to OCaml and the OCaml community. It was great to see a Typescript developer and public figure in the developer community talk about OCaml".

Dillon commented on his time at FUN OCaml:

“FUN OCaml was my favorite conference I've ever attended. I finally got to meet so many friends and peers that I've met online over the past year or two and got so much value out of collaborating with them in person. I can't wait for next year 🐫”

The workshop track was a valuable addition that gave participants an alternative way of engaging with OCaml: "People really enjoy the hands-on approach and the rare opportunity to hack on projects together with their maintainers". The workshops covered a range of topics such as a beginner's introduction to OCaml, building reliable actor systems, concurrency and parallelism, MirageOS, creating 2D games in OCaml, and web and mobile app development. "I heard from the host of the game engine workshop, Emile, that people built many cool things, including a minesweeper game!"

FUN OCaml 2025 and Running Your own OCaml Meet-up or Conference

To wrap up, I was curious about what Sabine had planned for the future, including whether there were any plans for another FUN OCaml. I am pleased to report that FUN OCaml is on track for 2025!

"This year, we had to schedule FUN OCaml for September to make it work with ICFP and REACT Alicante, but that meant it fell outside of the semester break for many students. Next time, we would like to schedule FUN OCaml between semesters so as many students as possible can come. They would benefit from hacking on open-source projects and getting feedback from maintainers. Software in the wild!"

We want to see more OCaml events around the world! You can organise your own conference or meet-up to bring more OCaml to your neck of the woods. Sabine is happy to help any member of the OCaml community looking to create events, and you can contact Sabine on X, on Bluesky, or by email.```

Until Next Time!

Connect with us online on X, Mastodon, Threads, and LinkedIn. We look forward to hearing from you and hope to see you around the OCaml universe!

Caml Weekly News

2024-11-12T12:00:00-00:00

Frama-C

2024-11-07T00:00:00-00:00

OCamlPro

2024-11-06T16:50:25-00:00

Un bonsaï sous sa cloche de verre. De nos jours, l'accès à la généalogie grand public est préservé surtout grâce à la maintenance de codes patrimoniaux.

L’équipe d’OCamlPro a récemment été sollicitée par l’association Roglo, une association française de généalogie qui gère une base de plus de 10 millions de personnes connectées dans un même arbre généalogique, et dont la base s'accroît d’environ 500 000 nouvelles contributions tous les ans. L’association s’appuie sur le logiciel libre Geneweb, l’un des plus puissants logiciels du domaine, créé en 1997 à l’Inria, permettant de partager sur le web des arbres généalogiques, et utilisé aussi bien par des particuliers que par des leaders du secteur, comme la société française Geneanet, acquise en 2021 par l’Américain Ancestry.

Notre mission s’est d’abord concentrée sur l'optimisation des performances, pour ramener le traitement de certaines requêtes sur la base gargantuesque de Roglo à des temps raisonnables. Après avoir rapidement survolé le code de plus de 80 000 lignes et profilé les requêtes les plus coûteuses, nous avons pu proposer une solution, l’implanter et l’intégrer dans la branche principale. Pour l’une des requêtes, le temps passe ainsi de 77s à 4s, soit 18 fois plus rapide ! Nous travaillons maintenant à enrichir Geneweb de nouvelles fonctionnalités pour ces utilisateurs, mais aussi pour ses contributeurs et les mainteneurs de la plateforme !

Cette mission, fragmentée en sprints de développement, s'inscrit dans une démarche continue visant à faire évoluer Geneweb pour qu'il puisse gérer des volumes de données encore plus importants. Nous sommes ravis de contribuer à cette évolution, en apportant notre expertise en optimisation et en développement logiciel pour faire grandir cette plateforme de référence.

Tarides

2024-11-06T00:00:00-00:00

We are steadfast OCaml advocates, providing core maintenance, hosting community events, and bringing groundbreaking new features to the language. We choose OCaml because it is a programming language with a unique combination of strengths, combining an expressive syntax and strong type system with memory safety and the power of multicore programming.

OCaml’s success is the combined result of the innovative efforts, passion, and dedication of the people who contribute to its development. The open-source community behind OCaml works collaboratively and transparently to make OCaml faster, safer, and easier. This community is made up of individual developers, research groups, and companies all working together. Tarides is one of the companies that contribute extensively to OCaml, and several of our team members are core OCaml developers.

You can support our open-source work by becoming a sponsor on our GitHub page. Your contribution will have a direct impact on the OCaml ecosystem by helping us maintain, develop, and improve its libraries, features, and tools. Let’s show you what we’re working on and what your support will help us achieve for OCaml!

What Does Tarides do?

At Tarides, we’re passionate about making OCaml even more powerful and accessible for developers. Our work focuses on three key areas: Enhancements, maintenance, and accessibility. Your contribution will support our work in all these areas.

Enhancing the Language: We bring new features to OCaml by resolving developer pain points and testing performance on different platforms. We want OCaml to be a competitive programming language with developer experience a high priority.
Maintaining Core Tools and Libraries: We ensure that OCaml developers have a reliable foundation for their projects by keeping the tools and libraries they depend on up to date.
Community Support and Outreach: We prioritise clear documentation and tutorials to improve accessibility and regularly organise events that allow community members to meet and exchange ideas.

What Projects Does Tarides Work on?

We work on a broad range of projects targeting different parts of the OCaml ecosystem: Compiler and language tools, development tools, community and infrastructure, and advanced projects.

Compiler and Language Tools

OCaml Compiler: Long-term maintenance of the compiler alongside several collaborators, as well as feature development and enhancements.
OCaml 5: We’re driving the OCaml 5 release with Multicore support and effect handlers.
Js_of_ocaml and wasm_of_ocaml: Run OCaml code in your browser!

Development Tools

OCaml Platform: Core OCaml tools, ensuring availability and compatibility with new compiler releases.
VSCode extension: Support and development of the OCaml Platform VSCode editor extension.
opam: The OCaml package manager, its tools, and plugins.
Dune and Dune Developer Preview: The OCaml build system.
Merlin and OCaml-LSP: A modern IDE for OCaml.
odoc: A documentation generator for OCaml.
OCamlFormat: Formatting for OCaml code.

Community and Infrastructure

OCaml.org: OCaml’s home on the web, the central knowledge base where the community can connect, access resources, and get the latest OCaml news.
OCaml Infrastructure: Maintenance of various parts of the OCaml.org and opam infrastructures.
OCaml Package Ecosystem: Maintenance of the growth and quality of the OCaml package ecosystem, including opam-repository and related opam-repo-ci, OCaml-CI as well as opam-health-check for Linux, FreeBSD and Windows as public services.

Advanced Projects

Eio: A modern, effect-based I/O library for OCaml designed to provide a high-level, structured concurrency model.
MirageOS: An operating system that constructs unikernels for secure, high-performance applications across various cloud computing and mobile platforms.
Irmin: Distributed data stores based on distributed version-control systems.

Contribute to the Future of OCaml Today!

By sponsoring us, you’ll support the maintenance of these essential OCaml tools and libraries and contribute to the growth of a diverse, dynamic community. Your contribution will have a direct impact on our projects and the OCaml ecosystem as a whole.

Please reach out to us on X, Mastodon, Threads, and LinkedIn. We look forward to hearing from you!

Caml Weekly News

2024-11-05T12:00:00-00:00

Tarides

2024-10-30T00:00:00-00:00

The world of cryptocurrency and internet transactions is constantly evolving, and the changing landscape of technologies that support this growing industry often means that legislation struggles to keep up, allowing cybercriminals to exploit loopholes and take advantage of the lack of oversight. Concerted effort across the sector is necessary to address this vulnerability.

To that end, we are thrilled to announce the ARGOS project, which stands for Analyse et Représentation des Graphes des Opérations Suspectes or the analysis and graphical representation of suspicious operations. Together with Functori, the LMF lab at Paris Saclay, and the Lip6 lab at Sorbonne University, Tarides is creating a platform for blockchain analysis. ARGOS will facilitate the monitoring and tracing of individual transactions on the blockchain, helping both nations enforce the law and organisations comply with regulation.

The Current Problem

The financial crimes perpetrated using crypto and the blockchain include money laundering, extortion, fraud, and funnelling funding for terrorist activities. Cybercriminals take advantage of the same traits that make crypto attractive to legitimate users: its decentralisation, dynamism, and versatility. Further exacerbating the problem, these same traits make it difficult for nations to legislate and enforce rules like they do for traditional financial institutions like banks.

This is because, rather than employing the traditional methods of identification, authentication, and document verification by a central authority, the blockchain uses an 'authorisation by default' approach to validate the exchange itself. Even though existing safeguards exist, which make it possible for police and the judicial system to determine the individuals associated with a transaction, they are not enough to effectively tackle cybercrime. What is needed is a way to identify, analyse, and track suspicious transactions and their sources at scale.

The ARGOS Solution

Several proposed solutions to the situation exist, but many of them can't address the breadth and complexity of the challenge or/and are not designed for the European financial system. Existing regulations empower nations to pose specific requirements on cryptocurrency providers and traders, and in Europe, that includes PACTE for France. Regulations impose rules on how the blockchain interacts with traditional financial institutions and the obligations that providers have including authenticating clients, freezing accounts, and identifying cryptographic addresses.

ARGOS will be a French solution custom-created to comply with and enforce French and European financial market regulations. It is designed to be lightweight enough not to place an undue burden on the existing ecosystem of blockchain developers, cryptocurrency providers, and traders, efficient enough to handle large codebases and complex systems, and robust enough to catch criminal activity reliably.

What is ARGOS?

The success of ARGOS rests on the key characteristics of the project, which are as follows:

Data Sovereignty: ARGOS will be designed and developed in France in compliance with the best practices and regulations of the European market. This provides additional security and privacy guarantees for European users, who do not have to share sensitive information with non-European parties.
Open Source: Being open-source is a fundamental aspect of the ARGOS project, providing the high levels of transparency required to facilitate collaboration across organisations. The project will be published under an open-source licence, ensuring the entire platform (including the source code, tests, etc.) is available to the community for validation and scrutiny. This transparency is important for users to feel confident in the platform and to encourage innovation in its ecosystem.
Large-Scale Data Management: We will use and adapt large-scale data management techniques to handle the big datasets characteristic of the blockchain sector. Our focus will be on efficiency and performance, and we will undertake R&D work to develop new solutions that suit our use case.
Suspicious-Pattern Detection: We plan to implement data processing mechanisms like taint analysis and pattern detection to track the movement of digital assets. Machine learning should also greatly improve the quality of the results we obtain.
In-Depth Analysis of Smart Contracts: Our system will be able to analyse smart contracts and understand the paths of nested interactions to determine whether a series of operations is suspicious.
Blockchain Bridge Traceability: Our solution will account for transactions across blockchain bridges, ensuring that ARGOS doesn't lose track of them and remains global (not limited to one blockchain).
Customisable User Interface: ARGOS will provide an intuitive and customisable user interface that can be adapted to suit each user's individual needs. It's important that our solution is accessible, easy to use, and integrates with existing tools. For example, a Digital Asset Service Provider (DASP) (so-called Prestataire de Service sur Actifs Numériques or PSAN in French) should be able to use it in combination with the monitoring and cryptographic tools they already employ with their own users.
Adapted Analysis Reports: ARGOS will offer several analytics report tools tailored to the specific needs of different users.

What Does the Future Hold?

The project is in its early stages, and we are collaborating closely with our partners to research and develop new approaches and technologies. Look out for more updates on the project coming in the future!

Connect with us online on X, Mastodon, Threads, and LinkedIn to stay updated on our latest projects.

Tarides champions open-source development. We create and maintain key features of the OCaml language in collaboration with the OCaml community. To learn more about how you can support our open-source work, discover our page on GitHub.

We are always happy to discuss commercial opportunities around OCaml. We provide core services, including training, tailor-made tools, and secure solutions. Contact us today to learn more about how Tarides can help your team.

Caml Weekly News

2024-10-29T12:00:00-00:00

Robur Cooperative

2024-10-29T00:00:00-00:00

An update of our email stack

Robur Cooperative

2024-10-28T00:00:00-00:00

libarchive vs hybrid GUID partition table and GNU tar volume header

Robur Cooperative

2024-10-25T00:00:00-00:00

    The NGI-funded DNSvizor provides core network services on your network; DNS resolution and DHCP.

Tarides

2024-10-23T00:00:00-00:00

It has been a while since the biggest functional programming event of the year: ICFP 2024 in Milan, Italy. The annual conference, sponsored by ACM SIGPLAN, combines world-class talks with workshops on some of the biggest functional programming languages, including ML, Haskell, OCaml, Scheme and Erlang.

Tarides are co-sponsors of ICFP, and several of our team members attend each year. This year, we had six talks given by Tarides team members and many of our colleagues chose to go as participants. I have asked several of them to share their experience at this year's ICFP to give you a taste of what the conference is all about.

The Tarides Talks

Before we get started, let's recap the talks created or co-created by Tarides engineers and where you can watch them after the fact.

First-Class Windows: Building a Roadmap for OCaml on Windows: A talk introducing the project seeking to make the developer experience with OCaml on Windows as good as it is on macOS and Linux. Watch the first-class Windows talk here.
Opam 2.2 and Beyond: Gives background to the recent opam release from the perspective of its core maintainers. Watch the opam 2.2 talk here.
Picos – Interoperable Effects-Based Concurrency: Covers the ongoing project to create an interface between effects-based schedulers and concurrency abstractions. Watch the Picos talk here.
Project-Wide Occurrences for OCaml, a Progress Report: Describes the design behind the first iteration of improved search features available with editor tooling. Watch the project-wide occurrences talk here.
Saturn: A Library of Verified Concurrent Data Structures for OCaml 5: This talk covers Saturn, a new library of well-tested, benchmarked, partially verified concurrent data structures. Watch the Saturn talk here.
Wasm_of_ocaml: Presents the work done on the Js_of_ocaml fork which translates OCaml bytecode to Webassembly. Watch the Wasm_of_OCaml talk here

Experience Reports

Curious about ICFP or missing the action already? I've asked several of my fellow Tarides team members who attended ICFP this year to share their thoughts and experiences. Let's dive in!

Why ICFP?

ICFP stands out to functional programming enthusiasts for gathering a large community of like-minded people in one place. As Jan Midtgaard puts it, "it's a wonderful mix of academics and industry people gathering due to a common interest in functional programming". KC Sivaramakrishnan has been attending ICFP since 2009 when he was a PhD student, and "ICFP is now the place where I meet friends and collaborators as well as the future generation of functional programming researchers".

Another reason behind the conference's enduring popularity is, of course, the packed schedule full of high-quality talks. The different tracks offer a variety of opportunities for participants to explore topics that interest them. Ambre Suhamy comments, "I know for sure that I will learn something new and come back from ICFP with more knowledge".

Stand-Out Talks

On the topic of talks, my colleagues attended several across different days and tracks. I asked them to share some of their impressions from the presentations, and while there were far too many to include them all, let's check out some of what they had to share!

ICFP Paper: Functional Programming in Financial Markets and OCaml Workshop: Recursion schemes in OCaml: Tim McGilchrist really enjoyed these two talks, the first focussing on Standard Chartered Bank's use of typed functional programming (Haskell) across their entire tech stack, and the second on Bloomberg's use of OCaml modelling bilateral financial contracts. "An interesting experience report for recursion schemes at Bloomberg. I'd previously only seen this implemented in Haskell!"
ICFP Papers: Oxidising OCaml: To KC Sivaramakrishnan, this was a "technical highlight". The talk centred around the potential for optimising heap allocations in OCaml without compromising on safety guarantees. "The modal types work aims to get the best of Rust into OCaml without letting go of what OCaml is good for – and the Rust-like features are opt-in!"
OCaml Workshop: Opam 2.2: Both Riku Silvola and Jan were impressed by this presentation on the new opam 2.2 release which notably brought native Windows support among other features. "That Raja Boujbel from OCamlPro, Kate Deplaix from Ahrefs (and the OCaml Software Foundation), and David Allsopp from Tarides gave a joint opam 2.2 talk sent a wonderful signal that three OCaml companies have worked together to do good for the community", said Jan. Riku agreed, commenting "David, Kate, and Raja presenting the long-awaited opam 2.2 release with its plethora of new features highlighted a longstanding and fruitful open-source collaboration across companies". They also announced opam's new time-based release cycle of updates every six months, check out the opam 2.3.0 blog post to learn more!
ML Track: Pattern Matching on Mutable Values: This one stood out in Ambre's mind for its memorable demonstration of an obscure corner case where the pattern-matching compiler would generate incorrect code. "It was insane; here's this five-line snippet of OCaml code; you can run it today, and it segfaults!" The original bug was found in 2016, and work has been ongoing to narrow down and fix the problem. Even though it was a rare edge case, the community rallied to address it, and a bug fix will be included in OCaml 5.3.
OCaml Workshop: Project-Wide Occurrences: Riku highlighted the work presented by Ulysse Gérard, introducing a new feature of editor tooling that lets users query all occurrences of a selected value anywhere in their project. "This work also highlights the important part the compiler plays in the overall platform vision, as the feature required work to be done not only in ocaml-lsp-server and merlin but also in dune and the compiler", Riku said.

Memorable Moments

My colleagues' week in Milan created several lasting memories. Ambre chaired her first session, the 3rd OCaml Workshop session, which included presentations on Priodomainslib, Saturn, Picos, and Distributed Actors. She also attended FARM, the international workshop on Functional Art, Music, Modelling, and Design. Ambre remembers the workshop as "people using music to make programs, or programming to make music, and understanding music through programming".

Tim recalls his many conversations between events with a variety of people in the hallways. It might not be the first thing that springs to mind, but the conversations that strike up organically at ICFP can be enlightening. For example, "people generally didn't know that you could use GBD/LLDB on OCaml binaries and, once they knew that, they were very excited about using those tools on their OCaml programs".

KC is focussed on the future, highlighting the feedback he received from participants and how it will help Tarides going forward. "I got some really nice feedback from folks, building on OCaml, about the work that Tarides is doing. I also got lots of honest feedback on what's not working. At the end of the day, our user community matters, and we need to solve their pain points so that OCaml becomes an advantage and not a technical risk for their engineering teams." He also wants to keep contributing to the future of ICFP: "I'm on the ICFP steering committee and will be a DEI co-chair for the next ICFP. I want to ensure that ICFP remains a thriving, friendly, and inclusive place for all of our attendees".

Finally, we can't discuss a Milan conference without mentioning the food! All the participants enjoyed sampling the local Italian fare and just look at this delicious pizza!

Join us Next Year!

The good news is that ICFP happens every year, so if you didn't attend this year, you can always set your sights on next year. The conference also moves around, alternating locations to make it easier for participants around the globe to join. We look forward to seeing you at ICFP 2025 in Singapore, so come find us if you're going!

You can reach out to us on X, Mastodon, Threads, and LinkedIn. Stay in touch!

Tarides champions open-source development. We create and maintain key features of the OCaml language in collaboration with the OCaml community. To learn more about how you can support our open-source work, discover our page on GitHub.

We are always happy to discuss commercial opportunities around OCaml. We provide core services, including training, tailor-made tools, and secure solutions. Contact us today to learn more about how Tarides can help your teams realise their vision.

Caml Weekly News

2024-10-22T12:00:00-00:00

Robur Cooperative

2024-10-22T00:00:00-00:00

The history of runtime arguments to a MirageOS unikernel

Robur Cooperative

2024-10-21T00:00:00-00:00

How we organise as a collective, and why we're doing that.

Tarides

2024-10-16T00:00:00-00:00

We’re excited to share a significant update to Dune's Package Management system, particularly one that will be of great interest to OCaml developers. For those who have been exploring Dune’s experimental package management capabilities over the past six months, you’ll be pleased to know that we've recently merged a feature allowing OCaml compiler packages to be installed directly through Dune.

Until now, Dune’s Package Management has supported the installation of various packages from the opam repository. However, it lacked the ability to install a functioning OCaml compiler — a crucial component for most Dune projects. This limitation meant that early adopters of Dune’s Package Management had to rely on external tools to manage their OCaml compiler installations, which wasn’t ideal. The good news is that this obstacle has been removed, making Dune Package Management far more robust and ready for testing by early adopters.

The challenge we faced in integrating the OCaml compiler installation stemmed from a conflict between the compiler’s build system and the way Dune handles package builds. Dune builds packages in what’s known as a “sandbox” — a temporary directory where a package is initially constructed and installed. Once the build is complete, the package's installed components are moved to their final destination within Dune’s build directory. However, the OCaml compiler assumes that its installation location will be its permanent home. Moving the installed files after installation caused the compiler to malfunction, making this an intractable problem for Dune’s package management.

While work is underway to make the OCaml compiler more flexible in terms of its installation location, we didn’t want to delay Dune’s package management features until this work was completed. Instead, we’ve introduced a workaround that allows compiler packages to be installed in a way that maintains their functionality.

The solution involves installing OCaml compiler packages to a global, per-user directory, rather than within the project’s sandbox. By default, the compiler is installed in a directory such as ~/.cache/dune/toolchains/ocaml-base-compiler.<version>-<hash>. This ensures that the compiler remains in its expected location and operates correctly without the need for relocation.

Moreover, this approach offers an added benefit: compilers installed through Dune can be shared across multiple projects. If two projects use the same version of the compiler, the installation step can be skipped in the second project, significantly speeding up the build process. Given that installing an OCaml compiler can take several minutes, this optimisation will save developers considerable time.

In summary, this new feature represents a substantial improvement to Dune’s Package Management system, making it easier and faster for developers to set up and manage their OCaml projects. By enabling direct installation of OCaml compilers through Dune, we’re removing a major barrier to adoption and enhancing the overall development experience.

Since it works transparently whenever you build a project with the Developer Preview, we'd love for you to test it out! Try out the new Dune Developer Preview today, and let us know how it goes on Discuss. We’re eager to see how the community leverages this new capability and look forward to your feedback as we continue to refine and expand Dune’s Package Management features.

Tarides is an open-source company with a strong focus on building and supporting the OCaml community. We're equally committed to the ongoing development of the OCaml language, collaborating with industry partners and engineers to enhance its performance and features.

We invite you to join the OCaml community, explore the language and tools, and contribute to its evolution.

Tarides is also open to discussing commercial opportunities related to OCaml. We offer services such as training, support, and custom development to help businesses adopt OCaml 5 efficiently. Please reach out to us to explore how we can assist with your needs.

Caml Weekly News

2024-10-15T12:00:00-00:00

The OCaml Planet

Caml Weekly News

Dmitrii Kovanikov

Thomas Letan’s Blog

Free and Open Source Software

Blog posts

$WORK

Talks

Sport

Final Notes

Caml Weekly News

Debajyati's Blog

Summary of the Content Prior to Reading the Article

What is Cmdliner?

Create a Dune Project

Setting Up the Dune Project

Writing The Actual Code

Our Internal library

Our Executable Program

Building & Running the Project

Building the Project

Ending

Thomas Letan’s Blog

Embedding My Website in a Binary

Serving the content with Dream

Naive Approach

Handling index.html Synonyms

Supporting Content-Type

Compressing if Requested

Caching

Building and Deploying the Website

Conclusion

Thomas Letan’s Blog

Embedding My Website in a Binary

Serving the content with Dream

Naive Approach

Handling index.html Synonyms

Supporting Content-Type

Compressing if Requested

Caching

Building and Deploying the Website

Conclusion

Caml Weekly News

Tarides

Testing APIs With Hidden or Uncontrolled State

Cygwin Challenges

Keeping Dependencies Minimal

Testing in the Presence of Misbehaviour

Crashes Take Down the Test Runner Too

Positive Testing, Negative Testing, and Stress Testing

False Alarms

Hidden Costs

Some Issues are Still Hard to Reproduce

Lowering the Barrier to Entry for multicoretests for Compiler Engineers

Usage Outside multicoretests

Conclusion

Dmitrii Kovanikov

Tarides

The Best Ways to Learn OCaml: For the Independent Learner

OCaml.org

Books

Community

Events

The Best Ways to Learn OCaml: Tailored Training in OCaml

Our Foundational Course Starting with OCaml: An Introduction

Our Higher-Level Course Mastering OCaml: Advanced Techniques

Get in Touch

Caml Weekly News

Tarides

What is Saturn?

Why Saturn?

Benchmarks

Tests

Formal Verification

Stay in Touch!

Frama-C

Caml Weekly News

Frama-C

Frama-C

Robur Cooperative

`$WORK`

Handling `index.html` Synonyms

Supporting `Content-Type`

Handling `index.html` Synonyms

Supporting `Content-Type`

Lowering the Barrier to Entry for `multicoretests` for Compiler Engineers

Usage Outside `multicoretests`