OCaml Planet

Shayne Fletcher

2017-05-11T20:18:34-00:00

Here are two well known "classic" functions over polymorphic lists.

map f l computes a new list from l by applying f to each of its elements.


        let rec map (f : α → β) : α list → β list = function
        | [] → []
        | h :: t → f h :: map f t
        ;;

mem x l returns true is x is an element of l and returns false if it is not.


        let rec mem (a : α) : α list → bool  = function
        | [] → false
        | x :: l → a = x || mem a l
        ;;

If y is an element of the list obtained by mapping f over l then there must be an element x in l such that f x = y. Conversely, if there exists an x in l such that y = f x, then y must be a member of the list obtained by mapping f over l.

We attempt a proof of correctness of the given definitions with respect to this property.

Lemma mem_map_iff:


    ∀ (f : α → β) (l : α list) (y : β),
        mem y (map f l) ⇔ ∃(x : α), f x = y ∧ mem x l.

Proof:

We first treat the forward implication
```
    ∀ (f : α → β) (l : α list) (y : β), 
      mem y (map f l) ⇒ ∃(x : α), f x = y ∧ mem x l
        
```
and proceed by induction on l.
- l = []:
  - Show mem y (map f []) ⇒ ∃(x : α), f x = y ∧ mem x [].
  - mem y (map f []) ≡ False.
  - Proof follows (ex falso quodlibet).
- l has form x' :: l (use l now to refer to the tail):
  - Assume the induction hypothesis:
    - mem y (map f l) ⇒ ∃x, f x = y ∧ mem x l.
  - We are required to show for an arbitrary (x' : α):
    - mem y (map f (x' :: l)) ⇒ ∃(x : α), f x = y ∧ mem x (x' :: l).
  - By simplification, we can rewrite the above to:
    - f x' = y ∨ mem y (map f l) ⇒ ∃(x : α), f x = y ∧ (x' = x ∨ mem x l).
  - We assume then an (x' : α) and a (y : β) such that:
    1. f x' = y ∨ mem y (map f l).
    2. mem y (map f l) ⇒ ∃(x : α), f x = y ∧ mem x l.
  - Show ∃(x : α), f x = y ∧ (x' = x ∨ mem x l).
    - First consider f x' = y in (1).
      - Take x = x' in the goal.
      - Then by (1) f x = y ∧ x = x'.
      - So x' is a witness.
    - Now consider mem y (map f l) in (1).
      - ∃(x^* : α), f x^* = y ∧ mem x^* l by (2).
      - Take x = x^* in the goal.
      - By the above f x^* = y ∧ mem x^* l
      - So x^* is a witness.
We now work on the reverse implication. We want to show that
```
    ∀ (f : α → β) (l : α list) (y : β),
       ∃(x : α), f x = y ∧ mem x l ⇒ mem y (map f l)
      
```
and proceed by induction on l.
- l = []:
  - Assume x, y with f x = y ∧ mem x [].
  - Show mem y (map f []):
- l has form x' :: l (use l now to refer to the tail):
  - Assume the induction hypothesis:
    - ∃(x : α), f x = y ∧ mem x l ⇒ mem y (map f l).
  - We are required to show for an arbitrary (x' : α):
    - ∃ (x : α), f x = y ∧ mem x (x' :: l) ⇒ mem y (map f (x' :: l))
  - By simplification, we can rewrite the above to:
    - ∃ (x : α), f x = y ∧ x = x' ∨ mem x l ⇒ f x' = y ∨ mem y (map f l).
  - Assume the goal and induction hypotheses:
    - There is (x : α) and (y : β) such that:
      1. f x = y ∧ (x = x' ∨ mem x l)
      2. f x = y ∧ mem x l ⇒ mem y (map f l)
  - Show f x' = y ∨ mem y (map f l).
    - Assume x = x' in (1) and show f x' = y:
      - Since, f x = y is given by (1.), f x' = y.
    - Assume mem x l in (1) and show mem y (map f l):
      - Rewrite mem y (map f l) via (2) to f x = y ∧ mem x l.
      - f x = y by (1) so mem y (map f l).

∎

References:
"Sofware Foundations" -- Pierce et. al.

OCamlPro

2017-05-11T13:20:05-00:00

This blog will cover yet another aspect of the improvements opam 2.0 has over opam 1.2. It may be a little more technical than previous issues, as it covers a feature directed specifically at packagers and repository maintainers, and regarding the package definition format.

Specifying dependencies in opam 1.2

Opam 1.2 already has an advanced way of specifying package dependencies, using formulas on packages and versions, with the following syntax:

depends: [
  "foo" {>= "3.0" & < "4.0~"}
  ( "bar" | "baz" {>= "1.0"} )
]

meaning that the package being defined depends on both package foo, within the 3.x series, and one of bar or baz, the latter with version at least 1.0. See here for a complete documentation.

This only allows, however, dependencies that are static for a given package.

Opam 1.2 introduced build, test and doc “dependency flags” that could provide some specifics for dependencies (e.g. test dependencies would only be needed when tests were requested for the package). These were constrained to appear before the version constraints, e.g. "foo" {build & doc & >= "3.0"}.

Extensions in opam 2.0

Opam 2.0 generalises the dependency flags, and makes the dependencies specification more expressive by allowing to mix filters, i.e. formulas based on opam variables, with the version constraints. If that formula holds, the dependency is enforced, if not, it is discarded.

This is documented in more detail in the opam 2.0 manual.

Note also that, since the compilers are now packages, the required OCaml version is now expressed using this mechanism as well, through a dependency to the (virtual) package ocaml, e.g. depends: [ "ocaml" {>= "4.03.0"} ]. This replaces uses of the available: field and ocaml-version switch variable.

Conditional dependencies

This makes it trivial to add, for example, a condition on the OS to a given dependency, using the built-in variable os:

depends: [ "foo" {>= "3.0" & < "4.0~" & os = "linux"} ]

here, foo is simply not needed if the OS isn't Linux. We could also be more specific about other OSes using more complex formulas:

depends: [
  "foo" { "1.0+linux" & os = "linux" |
          "1.0+osx" & os = "darwin" }
  "bar" { os != "osx" & os != "darwin" }
]

Meaning that Linux and OSX require foo, respectively versions 1.0+linux and 1.0+osx, while other systems require bar, any version.

Dependency flags

Dependency flags, as used in 1.2, are no longer needed, and are replaced by variables that can appear anywhere in the version specification. The following variables are typically useful there:

with-test, with-doc: replace the test and doc dependency flags, and are true when the package's tests or documentation have been requested
likewise, build behaves similarly as before, limiting the dependency to a “build-dependency”, implying that the package won't need to be rebuilt if the dependency changes
dev: this boolean variable holds true on “development” packages, that is, packages that are bound to a non-stable source (a version control system, or if the package is pinned to an archive without known checksum). dev sources often happen to need an additional preliminary step (e.g. autoconf), which may have its own dependencies.

Use opam config list for a list of pre-defined variables. Note that the with-test, with-doc and build variables are not available everywhere: the first two are allowed only in the depends:, depopts:, build: and install: fields, and the latter is specific to the depends: and depopts: fields.

For example, the datakit.0.9.0 package has:

depends: [
  ...
  "datakit-server" {>= "0.9.0"}
  "datakit-client" {with-test & >= "0.9.0"}
  "datakit-github" {with-test & >= "0.9.0"}
  "alcotest" {with-test & >= "0.7.0"}
]

When running opam install datakit.0.9.0, the with-test variable is set to false, and the datakit-client, datakit-github and alcotest dependencies are filtered out: they won't be required. With opam install datakit.0.9.0 --with-test, the with-test variable is true (for that package only, tests on packages not listed on the command-line are not enabled!). In this case, the dependencies resolve to:

depends: [
  ...
  "datakit-server" {>= "0.9.0"}
  "datakit-client" {>= "0.9.0"}
  "datakit-github" {>= "0.9.0"}
  "alcotest" {>= "0.7.0"}
]

which is treated normally.

Computed versions

It is also possible to use variables, not only as conditions, but to compute the version values: "foo" {= var} is allowed and will require the version of package foo corresponding to the value of variable var.

This is useful, for example, to define a family of packages, which are released together with the same version number: instead of having to update the dependencies of each package to match the common version at each release, you can leverage the version package-variable to mean “that other package, at the same version as current package”. For example, foo-client could have the following:

depends: [ "foo-core" {= version} ]

It is even possible to use variable interpolations within versions, e.g. specifying an os-specific version differently than above:

depends: [ "foo" {= "1.0+%{os}%"} ]

this will expand the os variable, resolving to 1.0+linux, 1.0+darwin, etc.

Getting back to our datakit example, we could leverage this and rewrite it to the more generic:

depends: [
  ...
  "datakit-server" {>= version}
  "datakit-client" {with-test & >= version}
  "datakit-github" {with-test & >= version}
  "alcotest" {with-test & >= "0.7.0"}
]

Since the datakit-* packages follow the same versioning, this avoids having to rewrite the opam file on every new version, with a risk of error each time.

As a side note, these variables are consistent with what is now used in the build: field, and the build-test: field is now deprecated. So this other part of the same datakit opam file:

build:
  ["ocaml" "pkg/pkg.ml" "build" "--pinned" "%{pinned}%" "--tests" "false"]
build-test: [
  ["ocaml" "pkg/pkg.ml" "build" "--pinned" "%{pinned}%" "--tests" "true"]
  ["ocaml" "pkg/pkg.ml" "test"]
]

would now be preferably written as:

build:
  ["ocaml" "pkg/pkg.ml" "build" "--pinned" "%{pinned}%" "--tests" "%{with-test}%"]
run-test:
  ["ocaml" "pkg/pkg.ml" "test"]

which avoids building twice just to change the options.

Conclusion

Hopefully this extension to expressivity in dependencies will make the life of packagers easier; feedback is welcome on your personal use-cases.

Note that the official repository is still in 1.2 format (served as 2.0 at https://opam.ocaml.org/2.0, through automatic conversion), and will only be migrated a little while after opam 2.0 is finally released. You are welcome to experiment on custom repositories or pinned packages already, but will need a little more patience before you can contribute package definitions making use of the above to the official repository.

NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!

OCaml Weekly News

2017-05-09T12:00:00-00:00

GitHub Jobs matching ocaml

2017-05-08T15:23:58-00:00

Software Developer

Jane Street is a proprietary quantitative trading firm, focusing primarily on trading equities and equity derivatives. We use innovative technology, a scientific approach, and a deep understanding of markets to stay successful in our highly competitive field. We operate around the clock and around the globe, employing over 400 people in offices in New York, London and Hong Kong.

The markets in which we trade change rapidly, but our intellectual approach changes faster still. Every day, we have new problems to solve and new theories to test. Our entrepreneurial culture is driven by our talented team of traders and programmers. At Jane Street, we don't come to work wanting to leave. We come to work excited to test new theories, have thought-provoking discussions, and maybe sneak in a game of ping-pong or two. Keeping our culture casual and our employees happy is of paramount importance to us.

We are looking to hire great software developers with an interest in functional programming. OCaml, a statically typed functional programming language with similarities to Haskell, Scheme, Erlang, F# and SML, is our language of choice. We've got the largest team of OCaml developers in any industrial setting, and probably the world's largest OCaml codebase. We use OCaml for running our entire business, supporting everything from research to systems administration to trading systems. If you're interested in seeing how functional programming plays out in the real world, there's no better place.

The atmosphere is informal and intellectual. There is a focus on education, and people learn about software and trading, both through formal classes and on the job. The work is challenging, and you get to see the practical impact of your efforts in quick and dramatic terms. Jane Street is also small enough that people have the freedom to get involved in many different areas of the business. Compensation is highly competitive, and there's a lot of room for growth.

You can learn more about Jane Street and our technology from our main site, janestreet.com. You can also look at a a talk given at CMU about why Jane Street uses functional programming (http://ocaml.janestreet.com/?q=node/61), and our programming blog (http://ocaml.janestreet.com).

We also have extensive benefits, including:

90% book reimbursement for work-related books
90% tuition reimbursement for continuing education
Excellent, zero-premium medical and dental insurance
Free lunch delivered daily from a selection of restaurants
Catered breakfasts and fresh brewed Peet's coffee
An on-site, private gym in New York with towel service
Kitchens fully stocked with a variety of snack choices
Full company 401(k) match up to 6% of salary, vests immediately
Three weeks of paid vacation for new hires in the US
16 weeks fully paid maternity/paternity leave for primary caregivers, plus additional unpaid leave

More information at http://janestreet.com/culture/benefits/

Shayne Fletcher

2017-05-11T20:18:34-00:00

PPX

Preprocessor extensions for code generation

"A Guide to Extension Points in OCaml"[1] provides a great "quick-start" on using the OCaml extension points API to implement preprocessor extensions for abstract syntax tree rewrites. This post picks up where that tutorial leaves off by showing how to write a ppx that does code generation.

The problem treated here is one posed in Whitequark's blog : "Implement a syntax extension that would accept type declarations of the form type t = A [@id 1] | B of int [@id 4] [@@id_of] to generate a function mapping a value of type t to its integer representation."

Implementing the "`id_of`" ppx

The basic strategy

In the OCaml parse tree, structures are lists of structure items. Type declarations are structure items as are let-bindings to functions.

In this program, analysis of an inductive type declaration t may result in the production of a new structure item, the AST of an of_id function to be appended to the structure containing t.

Now the general strategy in writing a ppx is to provide a record of type Ast_mapper.mapper. That record is based on the Ast_mapper.default_mapper record but selectively overriding those fields for those sytactic categories that the ppx is intending to transform.

Now, as we determined above, the effect of the ppx is to provide a function from a structure to a new structure. Accordingly, at a minimum then we'll want to override the structure field of the default mapper. Schematically then our ppx code will take on the following shape.


  open Ast_mapper
  open Ast_helper
  open Asttypes
  open Parsetree
  open Longident

  let structure_mapper mapper structure =
    ...

   let id_of_mapper =  {
     default_mapper with structure = structure_mapper
   }

   let () = register "id_of" id_of_mapper

This program goes just a little bit further though. Any @id or @@id_of attributes that get as far as the OCaml compiler would be ignored. So, it's not neccessary that they be removed by our ppx once they've been acted upon but it seems tidy to do so. Accordingly, there are two more syntactic constructs that this ppx operates on.


  open Ast_mapper
  open Ast_helper
  open Asttypes
  open Parsetree
  open Longident

  let structure_mapper mapper structure =
    ...

  let type_declaration_mapper mapper decl =
    ...

  let constructor_declaration_mapper mapper decl =
    ...

  let id_of_mapper argv = {
    default_mapper with
      structure = structure_mapper;
      type_declaration = type_declaration_mapper;
      constructor_declaration = constructor_declaration_mapper
  }

Implementing the mappings

To warm up, lets start with the easy mappers.

The role of type_declaration_mapper is a function from a type_declaration argument to a type_declaration result that is the argument in all but that any @@id_of attribute has been removed.


  let type_declaration_mapper 
      (mapper : mapper) 
      (decl : type_declaration) : type_declaration  =
    match decl with
      (*Case of an inductive type "t"*)
    | {ptype_name = {txt = "t"; _};
       ptype_kind = Ptype_variant constructor_declarations;
       ptype_attributes;_} ->
      let (_, attrs) = 
        List.partition (fun ({txt;_},_) ->txt="id_of") ptype_attributes in
      {(default_mapper.type_declaration mapper decl) 
      with ptype_attributes=attrs}
    (*Not an inductive type named "t"*)
    | _ -> default_mapper.type_declaration mapper decl

constructor_declaration_mapper is analogous to type_declaration_mapper above but this time its @id attributes that are removed.


  let constructor_declaration_mapper 
      (mapper : mapper) 
      (decl : constructor_declaration) : constructor_declaration =
    match decl with
    | {pcd_attributes; _} ->
      let (_, attrs) = 
        List.partition (fun ({txt;_}, _) -> txt="id") pcd_attributes  in
      {(default_mapper.constructor_declaration mapper decl) 
      with pcd_attributes=attrs}

Now to the raison d'etre of the ppx, structure_mapper.

First, a utility function that computes from a constructor_declaration with an @id attribute, a (function) case for it. For example, suppose "Bar of int [@id 4]" is the constructor declaration, then the case to be computed is the AST corresponding to the code "| Bar _ -> 4".


  let case_of_constructor_declaration : 
      constructor_declaration -> case =  function
    | {pcd_name={txt;loc};pcd_args;pcd_attributes; _} ->
      match List.filter (fun ({txt;_}, _) -> txt="id") pcd_attributes with
      (*No "@id"*)
      | [] -> 
        raise (Location.Error (Location.error ~loc "[@id] : Missing"))
      (*Single "@id"*)
      | [(_, payload)] -> 
        begin match payload with 
          | PStr [{pstr_desc=Pstr_eval ({pexp_desc=
              Pexp_constant (Pconst_integer (id, None)); _}, _)
            }] ->
            Exp.case 
              (Pat.construct 
                 {txt=Lident txt; loc=(!default_loc)}  
                 (match pcd_args with 
                 | Pcstr_tuple [] -> None | _ -> Some (Pat.any ())))
              (Exp.constant (Pconst_integer (id, None)))
          | _ -> 
            raise (Location.Error (Location.error ~loc 
            "[@id] : Bad (or missing) argument (should be int e.g. [@id 4])"))
        end
      (*Many "@id"s*)
      | (_ :: _) -> 
        raise (Location.Error (Location.error ~loc 
        "[@id] : Multiple occurences"))

One more utility function is required.

eval_structure_item item acc computes structure items to push on the front of acc. If item is a single declaration of an inductive type t attributed with @@id_of, then two structure items will be produced : one for t and one synthesized for t's of_id function. In all other cases, just one structure item will be pushed onto acc.


  let eval_structure_item
      (mapper : mapper) 
      (item : structure_item) 
      (acc : structure) : structure =
    match item with
    (*Case of a single inductive type declaration*)
    | { pstr_desc = Pstr_type (_, [type_decl]); pstr_loc} ->
      begin
        match type_decl with
        (*Case where the type identifer is [t]*)
        | {ptype_name = {txt = "t"; _};
           ptype_kind = Ptype_variant constructor_declarations;
           ptype_attributes;
           _} ->
          begin
            match List.filter (fun ({txt;_},_) ->txt="id_of") 
              ptype_attributes 
            with
            (*No [@@id_of]*)
            | [] -> default_mapper.structure_item mapper item :: acc
  
            (*At least one [@@id_of] (treat multiple occurences as if
              one)*)
            | _ ->
              (*Cases of an [id_of] function for [t], one for each
                of its constructors*)
              let cases= 
                List.fold_right 
                  (fun x acc -> 
                    case_of_constructor_declaration x :: acc) 
                  constructor_declarations [] in
              (*The [id_of] function itself*)
              let id_of : structure_item = 
                Str.value Nonrecursive [
                  Vb.mk 
                    (Pat.var {txt="id_of"; loc=(!default_loc)}) 
                    (Exp.function_ cases)] in
  
              default_mapper.structure_item mapper item :: id_of :: acc
          end
        (*Case the type identifier is something other than [t]*)
        | _ -> default_mapper.structure_item mapper item :: acc
      end
    (*Case this structure item is something other than a single type
      declaration*)
    | _ -> default_mapper.structure_item mapper item :: acc

Finally we can write structure_mapper itself as a simple fold over a structure.


  let structure_mapper 
      (mapper : mapper) 
      (structure : structure) : structure =
    List.fold_right (eval_structure_item mapper) structure []

Building and testing

So that's it, this preprocessor extension is complete. Assuming the code is contained in a file called ppx_id_of.ml it can be compiled with a command along the lines of the following.


  ocamlc -o ppx_id_of.exe  -I +compiler-libs ocamlcommon.cma ppx_id_of.ml

When built, it can be tested with a command like ocamlc -dsource -ppx ppx_id_of.exe test.ml.

For example, when invoked on the following program,


  type t = 
    | A [@id 2] 
    | B of int [@id 4] [@@id_of]

  module M = struct
    type t =
    | Foo of int [@id 42]
    | Bar [@id 43] [@@id_of]

    module N = struct
       type t = 
       | Baz [@id 8] 
       | Quux of string * int [@id 7] [@@id_of]

      module Q = struct
        type t = 
          | U [@id 0] [@@id_of]
      end
    end
  end

the resulting output is,


  type t =
    | A
    | B of int
  let id_of = function | A  -> 2 | B _ -> 4
  module M =
    struct
      type t =
        | Foo of int
        | Bar
      let id_of = function | Foo _ -> 42 | Bar  -> 43
      module N =
        struct
          type t =
            | Baz
            | Quux of string * int
          let id_of = function | Baz  -> 8 | Quux _ -> 7
          module Q = struct type t =
                              | U
                            let id_of = function | U  -> 0  end
        end
    end

References:
[1] "A Guide to Extension Points in OCaml" -- Whitequark (blog post 2014)

OCamlPro

2017-05-11T13:20:05-00:00

After the opam build feature was announced followed a lot of discussions, mainly having to do with its interface, and misleading name. The base features it offered, though, were still widely asked for:

a way to work directly with the project in the current directory, assuming it contains definitions for one or more packages
a way to copy the installed files of a package below a specified destdir
an easier way to get started hacking on a project, even without an initialised opam

Status of `opam build`

opam build, as described in a previous post has been dropped. It will be absent from the next Beta, where the following replaces it.

Handling a local project

Consistently with what was done with local switches, it was decided, where meaningful, to overload the <packages> arguments of the commands, allowing directory names instead, and meaning “all packages defined there”, with some side-effects.

For example, the following command is now allowed, and I believe it will be extra convenient to many:

opam install . --deps-only

What this does is find opam (or <pkgname>.opam) files in the current directory (.), resolve their installations, and install all required packages. That should be the single step before running the source build by hand.

The following is a little bit more complex:

opam install .

This also retrieves the packages defined at ., pins them to the current source (using version-control if present), and installs them. Note that subsequent runs actually synchronise the pinnings, so that packages removed or renamed in the source tree are tracked properly (i.e. removed ones are unpinned, new ones pinned, the other ones upgraded as necessary).

opam upgrade, opam reinstall, and opam remove have also been updated to handle directories as arguments, and will work on “all packages pinned to that target”, i.e. the packages pinned by the previous call to opam install <dir>. In addition, opam remove <dir> unpins the packages, consistently reverting the converse install operation.

opam show already had a --file option, but has also been extended in the same way, for consistency and convenience.

This all, of course, works well with a local switch at ./, but the two features can be used completely independently. Note also that the directory name must be made unambiguous with a possible package name, so make sure to use ./foo rather than just foo for a local project in subdirectory foo.

Specifying a destdir

This relies on installed files tracking, but was actually independent from the other opam build features. It is now simply a new option to opam install:

opam install foo --destdir ~/local/

will install foo normally (if it isn’t installed already) and copy all its installed files, following the same hierarchy, into ~/local. opam remove --destdir is also supported, to remove these files.

Initialising

Automatic initialisation has been dropped for the moment. It was only saving one command (opam init, that opam will kindly print out for you if you forget it), and had two drawbacks:

some important details (like shell setup for opam) were skipped
the initialisation options were much reduced, so you would often have to go back to opam init anyway. The other possibility being to duplicate init options to all commands, adding lots of noise. Keeping things separate has its merits.

Granted, another command, opam switch create ., was made implicit. But using a local switch is a user choice, and worse, in contradiction with the previous de facto opam default, so not creating one automatically seems safer: having to specify --no-autoinit to opam build in order to get the more simple behaviour was inconvenient and error-prone.

One thing is provided to help with initialisation, though: opam switch create <dir> has been improved to handle package definitions at <dir>, and will use them to choose a compatible compiler, as opam build did. This avoids the frustration of creating a switch, then finding out that the package wasn’t compatible with the chosen compiler version, and having to start over with an explicit choice of a different compiler.

If you would really like automatic initialisation, and have a better interface to propose, your feedback is welcome!

More related options

A few other new options have been added to opam install and related commands, to improve the project-local workflows:

opam install --keep-build-dir is now complemented with --reuse-build-dir, for incremental builds within opam (assuming your build-system supports it correctly). At the moment, you should specify both on every upgrade of the concerned packages, or you could set the OPAMKEEPBUILDDIR and OPAMREUSEBUILDDIR environment variables.
opam install --inplace-build runs the scripts directly within the source dir instead of a dedicated copy. If multiple packages are pinned to the same directory, this disables parallel builds of these packages.
opam install --working-dir uses the working directory state of your project, instead of the state registered in the version control system. Don’t worry, opam will warn you if you have uncommitted changes and forgot to specify --working-dir.

NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!

OCaml Weekly News

2017-05-09T12:00:00-00:00

OCaml – Jane Street Tech Blogs

2017-05-11T19:02:45-00:00

Jane Street is looking to hire a technical writer. If you're interested, or know someone who you think would be a good match, here's the application link.

We've always believed that developers should spend time and effort documenting their own code, but at the same time, a great writer with a feel for the technology can raise the level of quality in a way that few developers can. And as we've grown, having someone dedicated to writing makes a ton of sense.

Here are the kinds of things we'd like to have a technical writer work on:

Training material. We have a training program that many new hires go through, including most new developers and all new traders. In that program, they learn about OCaml, our base libraries, our build system, the UNIX shell, Emacs, and our dev tools. Part of the job would be to help make the course better, both by improving what we have, and by adding new material.
Improving library documentation. While we expect developers to do a reasonable job of documenting their code, our most important libraries deserve the time and care to make them really shine. This is aimed both internally and externally, since a lot of these libraries, like Async, Core and Incremental, are open source.
Writing longer pieces. We need more tutorials and overviews on a variety of topics. Part of the work would be to create great new documentation, and part of it is to serve as an example for others as to what good documentation looks like. And where possible, we want to do this so that the documentation effectively compiles against our current APIs, preventing it from just drifting out of date.

In terms of skills, we want someone who is both a clear and effective written communicator, and who is good enough at programming to navigate our codebase, work through our tutorials, and write up examples. An interest in functional programming and expressive type systems is a plus, but you don't need to know any OCaml (the language we use). That's something we're happy to teach you here.

m3ga blog

2017-04-30T02:22:00-00:00

Disclaimer: I work at Ambiata (our Github presence) probably the biggest Haskell shop in the southern hemisphere. Although I mention some of Ambiata's coding practices, in this blog post I am speaking for myself and not for Ambiata. However, the way I'm using ExceptT and handling exceptions in this post is something I learned from my colleagues at Ambiata.

At work, I've been spending some time tracking down exceptions in some of our Haskell code that have been bubbling up to the top level an killing a complex multi-threaded program. On Friday I posted a somewhat flippant comment to Google Plus:

Using exceptions for control flow is the root of many evils in software.ï»¿

Lennart Kolmodin who I remember from my very earliest days of using Haskell in 2008 and who I met for the first time at ICFP in Copenhagen in 2011 responded:

Yet what to do if you want composable code? Currently I have
type Rpc a = ExceptT RpcError IO a
which is terrible

But what do we mean by "composable"? I like the wikipedia definition:

ï»¿Composability is a system design principle that deals with the inter-relationships of components. A highly composable system provides recombinant components that can be selected and assembled in various combinations to satisfy specific user requirements.

The ensuing discussion, which also included Sean Leather, suggested that these two experienced Haskellers were not aware that with the help of some combinator functions, ExceptT composes very nicely and results in more readable and more reliable code.

At Ambiata, our coding guidelines strongly discourage the use of partial functions. Since the type signature of a function doesn't include information about the exceptions it might throw, the use of exceptions is strongly discouraged. When using library functions that may throw exceptions, we try to catch those exceptions as close as possible to their source and turn them into errors that are explicit in the type signatures of the code we write. Finally, we avoid using String to hold errors. Instead we construct data types to carry error messages and render functions to convert them to Text.

In order to properly demonstrate the ideas, I've written some demo code and made it available in this GitHub repo. It compiles and even runs (providing you give it the required number of command line arguments) and hopefully does a good job demonstrating how the bits fit together.

So lets look at the naive version of a program that doesn't do any exception handling at all.


  import Data.ByteString.Char8 (readFile, writeFile)

  import Naive.Cat (Cat, parseCat)
  import Naive.Db (Result, processWithDb, renderResult, withDatabaseConnection)
  import Naive.Dog (Dog, parseDog)

  import Prelude hiding (readFile, writeFile)

  import System.Environment (getArgs)
  import System.Exit (exitFailure)

  main :: IO ()
  main = do
    args <- getArgs
    case args of
      [inFile1, infile2, outFile] -> processFiles inFile1 infile2 outFile
      _ -> putStrLn "Expected three file names." >> exitFailure

  readCatFile :: FilePath -> IO Cat
  readCatFile fpath = do
    putStrLn "Reading Cat file."
    parseCat <$> readFile fpath

  readDogFile :: FilePath -> IO Dog
  readDogFile fpath = do
    putStrLn "Reading Dog file."
    parseDog <$> readFile fpath

  writeResultFile :: FilePath -> Result -> IO ()
  writeResultFile fpath result = do
    putStrLn "Writing Result file."
    writeFile fpath $ renderResult result

  processFiles :: FilePath -> FilePath -> FilePath -> IO ()
  processFiles infile1 infile2 outfile = do
    cat <- readCatFile infile1
    dog <- readDogFile infile2
    result <- withDatabaseConnection $ \ db ->
                 processWithDb db cat dog
    writeResultFile outfile result

Once built as per the instructions in the repo, it can be run with:


  dist/build/improved/improved Naive/Cat.hs Naive/Dog.hs /dev/null
  Reading Cat file 'Naive/Cat.hs'
  Reading Dog file 'Naive/Dog.hs'.
  Writing Result file '/dev/null'.

The above code is pretty naive and there is zero indication of what can and cannot fail or how it can fail. Here's a list of some of the obvious failures that may result in an exception being thrown:

Either of the two readFile calls.
The writeFile call.
The parsing functions parseCat and parseDog.
Opening the database connection.
The database connection could terminate during the processing stage.

So lets see how the use of the standard Either type, ExceptT from the transformers package and combinators from Gabriel Gonzales' errors package can improve things.

Firstly the types of parseCat and parseDog were ridiculous. Parsers can fail with parse errors, so these should both return an Either type. Just about everything else should be in the ExceptT e IO monad. Lets see what that looks like:


  {-# LANGUAGE OverloadedStrings #-}
  import           Control.Exception (SomeException)
  import           Control.Monad.IO.Class (liftIO)
  import           Control.Error (ExceptT, fmapL, fmapLT, handleExceptT
                                 , hoistEither, runExceptT)

  import           Data.ByteString.Char8 (readFile, writeFile)
  import           Data.Monoid ((<>))
  import           Data.Text (Text)
  import qualified Data.Text as T
  import qualified Data.Text.IO as T

  import           Improved.Cat (Cat, CatParseError, parseCat, renderCatParseError)
  import           Improved.Db (DbError, Result, processWithDb, renderDbError
                               , renderResult, withDatabaseConnection)
  import           Improved.Dog (Dog, DogParseError, parseDog, renderDogParseError)

  import           Prelude hiding (readFile, writeFile)

  import           System.Environment (getArgs)
  import           System.Exit (exitFailure)

  data ProcessError
    = ECat CatParseError
    | EDog DogParseError
    | EReadFile FilePath Text
    | EWriteFile FilePath Text
    | EDb DbError

  main :: IO ()
  main = do
    args <- getArgs
    case args of
      [inFile1, infile2, outFile] ->
              report =<< runExceptT (processFiles inFile1 infile2 outFile)
      _ -> do
          putStrLn "Expected three file names, the first two are input, the last output."
          exitFailure

  report :: Either ProcessError () -> IO ()
  report (Right _) = pure ()
  report (Left e) = T.putStrLn $ renderProcessError e


  renderProcessError :: ProcessError -> Text
  renderProcessError pe =
    case pe of
      ECat ec -> renderCatParseError ec
      EDog ed -> renderDogParseError ed
      EReadFile fpath msg -> "Error reading '" <> T.pack fpath <> "' : " <> msg
      EWriteFile fpath msg -> "Error writing '" <> T.pack fpath <> "' : " <> msg
      EDb dbe -> renderDbError dbe


  readCatFile :: FilePath -> ExceptT ProcessError IO Cat
  readCatFile fpath = do
    liftIO $ putStrLn "Reading Cat file."
    bs <- handleExceptT handler $ readFile fpath
    hoistEither . fmapL ECat $ parseCat bs
    where
      handler :: SomeException -> ProcessError
      handler e = EReadFile fpath (T.pack $ show e)

  readDogFile :: FilePath -> ExceptT ProcessError IO Dog
  readDogFile fpath = do
    liftIO $ putStrLn "Reading Dog file."
    bs <- handleExceptT handler $ readFile fpath
    hoistEither . fmapL EDog $ parseDog bs
    where
      handler :: SomeException -> ProcessError
      handler e = EReadFile fpath (T.pack $ show e)

  writeResultFile :: FilePath -> Result -> ExceptT ProcessError IO ()
  writeResultFile fpath result = do
    liftIO $ putStrLn "Writing Result file."
    handleExceptT handler . writeFile fpath $ renderResult result
    where
      handler :: SomeException -> ProcessError
      handler e = EWriteFile fpath (T.pack $ show e)

  processFiles :: FilePath -> FilePath -> FilePath -> ExceptT ProcessError IO ()
  processFiles infile1 infile2 outfile = do
    cat <- readCatFile infile1
    dog <- readDogFile infile2
    result <- fmapLT EDb . withDatabaseConnection $ \ db ->
                 processWithDb db cat dog
    writeResultFile outfile result

The first thing to notice is that changes to the structure of the main processing function processFiles are minor but all errors are now handled explicitly. In addition, all possible exceptions are caught as close as possible to the source and turned into errors that are explicit in the function return types. Sceptical? Try replacing one of the readFile calls with an error call or a throw and see it get caught and turned into an error as specified by the type of the function.

We also see that despite having many different error types (which happens when code is split up into many packages and modules), a constructor for an error type higher in the stack can encapsulate error types lower in the stack. For example, this value of type ProcessError:


  EDb (DbError3 ResultError1)

contains a DbError which in turn contains a ResultError. Nesting error types like this aids composition, as does the separation of error rendering (turning an error data type into text to be printed) from printing.

We also see that with the use of combinators like fmapLT, and the nested error types of the previous paragraph, means that ExceptT monad transformers do compose.

Using ExceptT with the combinators from the errors package to catch exceptions as close as possible to their source and converting them to errors has numerous benefits including:

Errors are explicit in the types of the functions, making the code easier to reason about.
Its easier to provide better error messages and more context than what is normally provided by the Show instance of most exceptions.
The programmer spends less time chasing the source of exceptions in large complex code bases.
More robust code, because the programmer is forced to think about and write code to handle errors instead of error handling being and optional afterthought.

Want to discuss this? Try reddit.

OCamlPro

2017-05-11T13:20:05-00:00

Among the areas we wanted to improve on for opam 2.0 was the handling of switches. In opam 1.2, they are simply accessed by a name (the OCaml version by default), and are always stored into ~/.opam/<name>. This is fine, but can get a bit cumbersome when many switches are in presence, as there is no way to sort them or associate them with a given project.

A reminder about switches

For those unfamiliar with it, switches, in opam, are independent prefixes with their own compiler and set of installed packages. The opam switch command allows to create and remove switches, as well as select the currently active one, where operations like opam install will operate.

Their uses include easily juggling between versions of OCaml, or of a library, having incompatible packages installed separately but at the same time, running tests without damaging your “main” environment, and, quite often, separation of environment for working on different projects.

You can also select a specific switch for a single command, with
opam install foo --switch other
or even for a single shell session, with
eval $(opam env --switch other)

What opam 2.0 adds to this is the possibility to create so-called local switches, stored below a directory of your choice. This gets users back in control of how switches are organised, and wiping the directory is a safe way to get rid of the switch.

Using within projects

This is the main intended use: the user can define a switch within the source of a project, for use specifically in that project. One nice side-effect to help with this is that, if a “local switch” is detected in the current directory or a parent, opam will select it automatically. Just don’t forget to run eval $(opam env) to make the environment up-to-date before running make.

Interface

The interface simply overloads the switch-name arguments, wherever they were present, allowing directory names instead. So for example:

cd ~/src/project
opam switch create ./

will create a local switch in the directory ~/src/project. Then, it is for example equivalent to run opam list from that directory, or opam list --switch=~/src/project from anywhere.

Note that you can bypass the automatic local-switch selection if needed by using the --switch argument, by defining the variable OPAMSWITCH or by using eval $(opam env --switch <name>)

Implementation

In practice, the switch contents are placed in a _opam/ subdirectory. So if you create the switch ~/src/project, you can browse its contents at ~/src/project/_opam. This is the direct prefix for the switch, so e.g. binaries can be found directly at _opam/bin/: easier than searching the opam root! The opam metadata is placed below that directory, in a .opam-switch/ subdirectory.

Local switches still share the opam root, and in particular depend on the repositories defined and cached there. It is now possible, however, to select different repositories for different switches, but that is a subject for another post.

Finally, note that removing that _opam directory is handled transparently by opam, and that if you want to share a local switch between projects, symlinking the _opam directory is allowed.

Current status

This feature has been present in our dev builds for a while, and you can already use it in the current beta.

Limitations and future extensions

It is not, at the moment, possible to move a local switch directory around, mainly due to issues related to relocating the OCaml compiler.

Creating a new switch still implies to recompile all the packages, and even the compiler itself (unless you rely on a system installation). The projected solution is to add a build cache, avoiding the need to recompile the same package with the same dependencies. This should actually be possible with the current opam 2.0 code, by leveraging the new hooks that are made available. Note that relocation of OCaml is also an issue for this, though.

Editing tools like ocp-indent or merlin can also become an annoyance with the multiplication of switches, because they are not automatically found if not installed in the current switch. But the user-setup plugin (run opam user-setup install) already handles this well, and will access ocp-indent or tuareg from their initial switch, if not found in the current one. You will still need to install tools that are tightly bound to a compiler version, like merlin and ocp-index, in the switches where you need them, though.

NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!

OCaml – Jane Street Tech Blogs

2017-05-11T19:02:45-00:00

We have a new tech talk coming up on May 17th, from our very own Dominick LoBraico. This one is about how to represent configurations with programs. In some sense, this is an obvious idea. Lots of programmers have experienced the dysphoria that comes from watching your elegant little configuration format metamorphize into a badly constructed programming language with miserable tools. This happens because, as you try to make your configs clearer and more concise, you often end up walking down the primrose path of making your config format ever more language-like. But you never really have the time to make it into a proper language.

The obvious alternative is to just use a real language, one that comes with decent tooling and well-designed abstractions. (And ideally, a functional language, because they tend to be better at writing clear, declarative code.)

This talk discusses what happens when you try to put this obvious idea into practice, which is less, well, obvious. I like this kind of topic because it represents the kind of hard-won knowledge you can only get by trying something and screwing it up a few times.

So please join us! You can register here.

OCaml Weekly News

2017-05-09T12:00:00-00:00

GitHub Jobs matching ocaml

2017-05-08T15:23:58-00:00

Fulltime, Copenhagen

issuu is the world's fastest-growing digital publishing platform. We are looking for a new member to join our fantastic team. With great people, unique ideas and stunning technology, we're changing the future of publishing today. Can you be the best at what you do? Join us!

About this job

As a Front-end Developer at issuu, you will be joining a team of highly skilled web enthusiasts building web applications in an agile environment. We currently develop for desktops, tablets and mobile web. This requires great responsive sites that ensure the best possible user experience on all platforms.

We run a NodeJS server in a production system that serves billions of pages every month. Our client-side applications are a mix of vanilla JavaScript (transpiled with Babel from ES2015), and React / Redux, apps with an in-house developed, modular, BEM-styled styling framework.SASS enables us to keep our CSS well structured.

As our ideal candidate, you are excited about HTML5 features, like SVG and

What we Like

We think browsers are awesome.
We love HTML5, CSS3 and all the new exciting web APIs.
We use React, Redux, ES2015 and CSS Modules.
We draw with Canvas, WebGL and SVGs.
We build modern progressive web apps.
We like declarative and functional programming.
We used to like jQuery and Backbone, but now we sort of grew apart.
We also like it if our front-end developers aren’t afraid of touching our backends, which we make with -
Python, NodeJS, OCaml and Erlang.

Qualifications

Demonstrated ability to deliver results in a fast-paced environment as a self-starter and quick learner
Expert understanding of browsers, web technologies (object-oriented JavaScript, ES2015, HTML, CSS), and a passion for the latest and greatest web standards
Experience in collaborating with UI Designers
Experience architecting and building robust, high-performance production web applications
Used to working with Design and Product teams in an agile fashion
Experience with building web applications for mobile web is a plus
Experience with MVC-based web applications is a plus
Experience with server-side languages like Python is a plus
Solid communications skills in English

What we Offer

You’ll be a part of issuu – an amazing place with room for parents, foodies, geeks, odd birds and the occasional normal person
Informal, inclusive and very flexible workplace
Competitive compensation package
Flat hierarchy – every opinion matters regardless of team and position
Regular hackathons
A sleek Mac and a pretty sweet desk setup
Great offices in Copenhagen, Palo Alto and Berlin

About the Team and Office

You will be joining our DK engineering team in Copenhagen consisting of approx. 27 developers. We are a diverse office with members of all parts of issuu, including Customer Support, Product, Design, Data Analytics and Management.

In a company the size of issuu, code maintainability is just as important as site accessibility. That also means that you are willing to share your code with other people or sit together for pair programming sessions.

Our office is conveniently located next to the Copenhagen Central Station. We provide nice, catered lunches each day, regular outings and monthly Friday bars with drinks, snacks and activities.

About issuu

issuu connects the world’s publishers to an audience of active consumers, on a global scale. Every day millions of people find, read and share billions of pieces of content they find meaningful, from every device. Millions of magazine, newspaper and catalog creators are empowered to distribute, measure and monetize their work through the issuu platform issuu. If you are interested in a sneak peek of who we are as issuuees, have a look at our brandbook.

Details

It’s a prerequisite that you have a valid EU work permit

OCaml Labs compiler hacking

2017-04-18T12:00:00-00:00

Our next OCaml Compiler Hacking event will be on Tuesday 16th May in The Old Library at Pembroke College, Cambridge.

If you're planning to come along, it'd be helpful if you could indicate interest via Doodle and sign up to the mailing list to receive updates.

Where: The Old Library, Pembroke College, Cambridge CB2 1RF

The Old Library is the first building on the left straight after the Porters Lodge.

When: 6:30pm, Tuesday 16th May

Who: anyone interested in improving OCaml. Knowledge of OCaml programming will obviously be helpful, but prior experience of working on OCaml internals isn't necessary.

Refreshments: either the Pembroke buttery between 5:45-6:45pm (cash only) or there will be a finger buffet in The Old Library itself.

What: fixing bugs, implementing new features, learning about OCaml internals

Wiki: https://github.com/ocamllabs/compiler-hacking/wiki

We're defining "compiler" pretty broadly, to include anything that's part of the standard distribution, which means at least the standard library, run time, tools (ocamldep, ocamllex, ocamlyacc, etc.), the debugger, the documentation, and the compiler itself. We'll have suggestions for mini-projects for various levels of experience, but feel free to come along and work on whatever you fancy.

Drup's thingies

2017-04-17T00:00:00-00:00

Here are some news about Tyre, along with release of version 0.3.

OCamlPro

2017-05-11T13:20:05-00:00

As you may have noticed, on the begining of April I have some urge to write something technical about some deeply specific point of OCaml. This time I’d like to tackle that through sudoku.

It appearch that Sudoku is of great importance considering the number of posts explaining how to write a solver. Following that trend I will explain how to write one in OCaml. But with a twist.

We will try to optimize it. I won’t show you anything as obvious as how to micro-optimize your code or some smart heuristc. No we are not aiming for being merely algorithmically good. We will try to make something serious, we are want it to be solved even before the program starts.

Yes really. Before. And I will show you how to use a feature of OCaml 4.03 that is sadly not well known.

—

First of all, as we do like type and safe programs, we will define what a well formed sudoku solution looks like. And by defining of course I mean declaring some GADTs with enough constraints to ensure that only well correct solutions are valid.

I assume tha you know the rules of Sudoku and will refrain from infuriating you by explaining it. But we will still need some vocabulary.

So the aim of sudoku is to fill a ‘grid’ with ‘symbols’ satisfying some ‘row’ ‘column’ and ‘square’ constraints.

To make the code examples readable we will stick to 4*4 sudokus. It’s the smallest size that behaves the same way as 9*9 ones (I considered going for 1*1 ones, but the article ended up being a bit short). Of course everything would still apply to any n^2*n^2 sized one.

So let’s start digging in some types. As we will refine them along the way, I will leave some parts to be filled later. This is represented by ‘…’ .

First there are symbols, just 4 of them befause we reduced the size. Nothing special about that right now.

type ... symbol =
| A : ...
| B : ...
| C : ...
| D : ...

And a grid is 16 symbols. To avoid too much visual clutter in the type I just put them linearly. The comment show how it is supposed to be seen in the 2d representation of the grid:

(* a b c d
   e f g h
   i j k l
   m n o p *)

type grid =
Grid :
... symbol * (* a *)
... symbol * (* b *)
... symbol * (* c *)
... symbol * (* d *)

... symbol * (* e *)
... symbol * (* f *)
... symbol * (* g *)
... symbol * (* h *)

... symbol * (* i *)
... symbol * (* j *)
... symbol * (* k *)
... symbol * (* l *)

... symbol * (* m *)
... symbol * (* n *)
... symbol * (* o *)
... symbol (* p *)
-> solution

Right now grid is a simple 16-uple of symbols, but we will soon start filling those ‘…’ to forbid any set of symbols that is not a valid solution.

Each constraint looks like, ‘among those 4 positions neither 2 symbols are the same’. To express that (in fact something equivalent but a bit simpler to state with our types), we will need to name positions. So let’s introduce some names:

type r1 (* the first position among a row constraint *)
type r2 (* the second position among a row constraint *)
type r3
type r4

type c1 (* the first position among a column constraint *)
type c2
type c3
type c4

type s1
type s2
type s3
type s4

type ('row, 'column, 'square) position

On the 2d grid this is how the various positions will be mapped.

r1 r2 r3 r4
r1 r2 r3 r4
r1 r2 r3 r4
r1 r2 r3 r4

c1 c1 c1 c1
c2 c2 c2 c2
c3 c3 c3 c3
c4 c4 c4 c4

s1 s2 s1 s2
s3 s4 s3 s4
s1 s2 s1 s2
s3 s4 s4 s4

For instance, the position g, in the 2nd row, 3rd column, will at the 3rd position in its row constraint, 2nd in its column constraint, and 3rd in its square constraint:

type g = (r3, c2, s3) position

We could have declare a single constraint position type, but this is slightly more readable. than:

type g = (p3, p2, p3) position

The position type is phantom, we could have provided a representation, but since no value of this type will ever be created, it’s less confusing to state it that way.

type a = (r1, c1, s1) position
type b = (r2, c1, s2) position
type c = (r3, c1, s1) position
type d = (r4, c1, s2) position

type e = (r1, c2, s3) position
type f = (r2, c2, s4) position
type g = (r3, c2, s3) position
type h = (r4, c2, s4) position

type i = (r1, c3, s1) position
type j = (r2, c3, s2) position
type k = (r3, c3, s1) position
type r = (r4, c3, s2) position

type m = (r1, c4, s3) position
type n = (r2, c4, s4) position
type o = (r3, c4, s3) position
type p = (r4, c4, s4) position

It is now possible to state for each symbol in which position it is, so we will start filling a bit those types.

type ('position, ...) symbol =
| A : (('r, 'c, 's) position, ...) symbol
| B : (('r, 'c, 's) position, ...) symbol
| C : (('r, 'c, 's) position, ...) symbol
| D : (('r, 'c, 's) position, ...) symbol

This means that a symbol value is then associated to a single position in each constraint. We will need to state that in the grid type too:

type grid =
Grid :
(a, ...) symbol * (* a *)
(b, ...) symbol * (* b *)
(c, ...) symbol * (* c *)
(d, ...) symbol * (* d *)

(e, ...) symbol * (* e *)
(f, ...) symbol * (* f *)
(g, ...) symbol * (* g *)
(h, ...) symbol * (* h *)

(i, ...) symbol * (* i *)
(j, ...) symbol * (* j *)
(k, ...) symbol * (* k *)
(l, ...) symbol * (* l *)

(m, ...) symbol * (* m *)
(n, ...) symbol * (* n *)
(o, ...) symbol * (* o *)
(p, ...) symbol (* p *)
-> solution

We just need to forbid a symbol to appear in two different positions of a given row/column/square to prevent invalid solutions.

type 'fields row constraint 'fields = < a : 'a; b : 'b; c : 'c; d : 'd >
type 'fields column constraint 'fields = < a : 'a; b : 'b; c : 'c; d : 'd >
type 'fields square constraint 'fields = < a : 'a; b : 'b; c : 'c; d : 'd >

Those types represent the statement ‘in this line/column/square, the symbol a is at the position ‘a, the symbol b is at the position ‘b, …’

For instance, the row ‘A D B C’ will be represented by

< a : l1; b : l3; c : l4; d : l2 > row

Which reads: ‘The symbol A is in first position, B in third position, C in fourth, and D in second’

The object type is used to make things a bit lighter later and allow to state names.

Now the symbols can be a bit more annotated:

type ('position, 'row, 'column, 'square) symbol =
| A : (('r, 'c, 's) position,
< a : 'r; .. > row,
< a : 'c; .. > column,
< a : 's; .. > square)
symbol

| B : (('r, 'c, 's) position,
< b : 'r; .. > row,
< b : 'c; .. > column,
< b : 's; .. > square)
symbol

| C : (('r, 'c, 's) position,
< c : 'r; .. > row,
< c : 'c; .. > column,
< c : 's; .. > square)
symbol

| D : (('r, 'c, 's) position,
< d : 'r; .. > row,
< d : 'c; .. > column,
< d : 's; .. > square)
symbol

Notice that ‘..’ is not ‘…’. Those dots are really part of the OCaml syntax: it means ‘put whatever you want here, I don’t care’. There is nothing more to add to this type.

This type declaration reports the position information. Using the same variable name ‘r in the position and in the row constraint parameter for instance means that both fields will have the same type.

For instance, a symbol ‘B’ in position ‘g’ would be in the 3rd position of its row, 2nd position of its column , and 3rd position of its square:

# let v : (g, _, _, _) symbol = B;;
val v :
(g, < b : r3 > row,
    < b : c2 > column,
    < b : s3 > square)
symbol = B

Those types constraints ensure that this is correctly reported.

The real output of the type checker is a bit more verbose, but I remove the irrelevant part:

val v :
(g, < a : 'a; b : r3; c : 'b; d : 'c > row,
    < a : 'd; b : c2; c : 'e; d : 'f > column,
    < a : 'g; b : s3; c : 'h; d : 'i > square)
symbol = B

We are now quite close from a completely constrained type. We just need to say that the various symbols from the same row/line/column constraint have the same type:

type grid =
Grid :
(a, 'row1, 'column1, 'square1) symbol *
(b, 'row1, 'column2, 'square1) symbol *
(c, 'row1, 'column3, 'square2) symbol *
(d, 'row1, 'column4, 'square2) symbol *

(e, 'row2, 'column1, 'square1) symbol *
(f, 'row2, 'column2, 'square1) symbol *
(g, 'row2, 'column3, 'square2) symbol *
(h, 'row2, 'column4, 'square2) symbol *

(i, 'row3, 'column1, 'square3) symbol *
(j, 'row3, 'column2, 'square3) symbol *
(k, 'row3, 'column3, 'square4) symbol *
(l, 'row3, 'column4, 'square4) symbol *

(m, 'row4, 'column1, 'square3) symbol *
(n, 'row4, 'column2, 'square3) symbol *
(o, 'row4, 'column3, 'square4) symbol *
(p, 'row4, 'column4, 'square4) symbol *

That is two symbols in the same row/column/square will share the same ‘row/’symbol/’square type. For any couple of symbols in say, a row, they must agree on that type, hence, on the position of every symbol.

Let’s look at the ‘A’ symbol for the ‘a’ and ‘c’ position for instance. Both share the same ‘row1 type variable. There are two cases. Either both are ‘A’s ore one is not.

* If one symbol is not a ‘A’, let’s say those are ‘C’ and ‘A’ symbols. Their row type (pun almost intended) will be respectively < c : r1; .. > and < a : r3; .. >. Meaning that ‘C’ does not care about the position of ‘A’ and conversly. Those types are compatible. No problem here.

* If both are ‘A’s then something else happens. Their row types will be < a : r1; .. > and < a : r3; .. > which is certainly not compatible since r1 and r3 are not compatible. This will be rejected.

Now we have a grid type that checks the sudoku constraints !

Let’s try it.

let ok =
Grid
(A, B, C, D,
C, D, A, B,

D, A, B, C,
B, C, D, A)

val ok : grid = Grid (A, B, C, D, C, D, A, B, D, A, B, C, B, C, D, A)

let not_ok =
Grid
(A, B, C, D,
C, D, A, B,

D, A, B, C,
B, C, A, D)

B, C, A, D);;
^
Error: This expression has type
(o, < a : r3; b : r1; c : r2; d : 'a > row,
    < a : c4; b : 'b; c : 'c; d : 'd > column,
    < a : s3; b : 'e; c : 'f; d : 'g > square)
  symbol
but an expression was expected of type
(o, < a : r3; b : r1; c : r2; d : 'a > row,
    < a : c2; b : c3; c : c1; d : 'h > column,
    < a : 'i; b : s1; c : s2; d : 'j > square)
  symbol
Types for method a are incompatible

What it is trying to say is that ‘A’ is both at position ‘2’ and ‘4’ of its column. Well it seems to work.

Solving it

But we are not only interested in checking that a solution is correct, we want to find them !

But with ‘one weird trick’ we will magically transform it into a solver, namely the -> . syntax. It was introduced in OCaml 4.03 for some other purpose. But we will now use its hidden power !

This is the right hand side of a pattern. It explicitely states that a pattern is unreachable. For instance

type _ t =
  | Int : int -> int t
  | Float : float -> float t

let add (type v) (a : v t) (b : v t) : v t =
  match a, b with
  | Int a, Int b -> Int (a + b)
  | Float a, Float b -> Float (a +. b)
  | _ -> .

By writing it here you state that you don’t expect any other pattern to verify the type constraints. This is effectively the case here. In general you won’t need this as the exhaustivity checker will see it. But in some intricate situations it will need some hints to work a bit more. For more information see Jacques Garrigue / Le Normand article

This may be a bit obscure, but this is what we now need. Indeed, we can ask the exhaustivity checker if there exist a value verifying the pattern and the type constraints. For instance to solve a problem, we ask the compiler to check if there is any value verifying a partial solution encoded as a pattern.

 A _ C _
 _ D _ B
 _ A D _
 D _ B _

let test x =
match x with
| Grid
  (A, _, C, _,
   _, D, _, B,

   _, A, D, _,
   D, _, B, _) -> .
| _ -> ()

Error: This match case could not be refuted.
Here is an example of a value that would reach it:
Grid (A, B, C, D, C, D, A, B, B, A, D, C, D, C, B, A)

The checker tells us that there is a solution verifying those constraints, and provides it.

If there were no solution, there would have been no error.

let test x =
match x with
| Grid
  (A, B, C, _,
   _, _, _, D,

   _, _, _, _,
   _, _, _, _) -> .
| _ -> ()

val test : grid -> unit =

And that’s it !

Wrapping it up

Of course that’s a bit cheating since the program is not executable, but who cares really ?
If you want to use it, I made a small (ugly) script generating those types. You can try it on bigger problems, but in fact it is a bit exponential. So you shouldn’t really expect an answer too soon.

OCaml Weekly News

2017-05-09T12:00:00-00:00

Frama-C RSS News

2017-04-05T17:30:30-00:00

OCaml Weekly News

2017-05-09T12:00:00-00:00

Drup's thingies

2017-04-17T00:00:00-00:00

I’m happy to announce the release of Pumping, a library to leverage the OCaml type system to recognize regular languages.

Shayne Fletcher

2017-05-11T20:18:34-00:00

Locations

Writing compilers and interpreters requires rigorous management of source code locations harvested during syntax analysis and associated error handling mechanisms that involve reporting those locations along with details of errors they associate to.

This article does a "deep dive" into the the Location module of the OCaml compiler. The original source can be found in the ocaml/parsing directory of an OCaml distribution (copyright Xavier Leroy).

Location is a masterclass in using the standard library Format module. If you have had difficulties understanding Format and what it provides the OCaml programmer, then this is for you. Furthermore, Location contains invaluable idioms for error reporting & exception handling. Learn them here to be able to apply them in your own programs.

Describing locations

A location corresponds to a range of characters in a source file. Location defines this type and a suite of functions for the production of location values.


type t = { 
  loc_start : Lexing.position;
  loc_end : Lexing.position;
  loc_ghost : bool
}

val none : t
val in_file : string → t
val init : Lexing.lexbuf → string → unit
val curr : Lexing.lexbuf → t

val symbol_rloc : unit → t
val symbol_gloc : unit → t
val rhs_loc : int → t

type 'a loc = { 
  txt : 'a;
  loc : t; 
}

val mkloc : 'a → t → 'a loc
val mknoloc : 'a → 'a loc

A value of the (standard library) type Lexing.position describes a point in a source file.


    type position = {
      pos_fname : string;
      pos_lnum : int;
      pos_bol : int;
      pos_cnum : int
    }

The fields of this record have the following meanings:

pos_fname is the file name
pos_lnum is the line number
pos_bol is the offset of the beginning of the line (the number of characters between the beginning of the lexbuf and the beginning of the line)
pos_cnum is the offset of the position (number of characters between the beginning of the lexbuf (details below) and the position)

The difference between pos_cnum and pos_bol is the character offset within the line (i.e. the column number, assuming each character is one column wide).

A location in a source file is defined by two positions : where the location starts and where the location ends.


type t = {
  loc_start : position;
  loc_end : position;
  loc_ghost : bool
}

The third field loc_ghost is used to disambiguate locations that do not appear explicitly in the source file. A location will not be marked as ghost if it contains a piece of code that is syntactically valid and corresponds to an AST node and will be marked as a ghost location otherwise.

There is a specific value denoting a null position. It is called none and it is defined by the function in_file.


let in_file (name : string) : t =
  let loc : position = {
    pos_fname = name; (*The name of the file*)
    pos_lnum = 1; (*The line number of the position*)
    pos_bol = 0; (*Offset from the beginning of the lexbuf of the line*)
    pos_cnum = -1; (*Offset of the position from the beginning of the lexbuf*)
  } in
  { loc_start = loc; loc_end = loc; loc_ghost = true }

let none : t = in_file "_none_"

Lexing.lexbuf is the (standard library) type of lexer buffers. A lexer buffer is the argument passed to the scanning functions defined by generated scanners (lexical analyzers). The lexer buffer holds the current state of the scanner plus a function to refill the buffer from the input.


type lexbuf = {
  refill_buff : lexbuf → unit;
  mutable lex_buffer : bytes;
  mutable lex_buffer_len : int;
  mutable lex_abs_pos : int;
  mutable lex_start_pos : int;
  mutable lex_curr_pos : int;
  mutable lex_last_pos : int;
  mutable lex_last_action : int;
  mutable lex_eof_reached : bool;
  mutable lex_mem : int array;
  mutable lex_start_p : position;
  mutable lex_curr_p : position;
}

At each token, the lexing engine will copy lex_curr_p to lex_start_p then change the pos_cnum field of lex_curr_p by updating it with the number of characters read since the start of the lexbuf. The other fields are left unchanged by the the lexing engine. In order to keep them accurate, they must be initialized before the first use of the lexbuf and updated by the relevant lexer actions.


(*Set the file name and line number of the [lexbuf] to be the start
   of the named file*)
let init (lexbuf : Lexing.lexbuf) (fname : string) : unit =
  let open Lexing in
  lexbuf.lex_curr_p <- {
    pos_fname = fname;
    pos_lnum = 1;
    pos_bol = 0;
    pos_cnum = 0;
  }

The location of the current token in a lexbuf is computed by the function curr.


(*Get the location of the current token from the [lexbuf]*)
let curr (lexbuf : Lexing.lexbuf) : t = 
  let open Lexing in {
    loc_start = lexbuf.lex_start_p;
    loc_end = lexbuf.lex_curr_p;
    loc_ghost = false;
  }

Parsing is the run-time library for parsers generated by ocamlyacc. The functions symbol_start and symbol_end are to be called in the action part of a grammar rule (only). They return the offset of the string that matches the left-hand-side of the rule : symbol_start returns the offset of the first character; symbol_end returns the offset after the last character. The first character in a file is at offset 0.

symbol_start_pos and symbol_end_pos are like symbol_start and symbol_end but return Lexing.position values instead of offsets.


(*Compute the span of the left-hand-side of the matched rule in the
   program source*)
let symbol_rloc () : t = {
  loc_start = Parsing.symbol_start_pos ();
  loc_end = Parsing.symbol_end_pos ();
  loc_ghost = false
}

(*Same as [symbol_rloc] but designates the span as a ghost range*)
let symbol_gloc () : t =  { 
  loc_start = Parsing.symbol_start_pos ();
  loc_end = Parsing.symbol_end_pos ();
  loc_ghost = true
}

The Parsing functions rhs_start and rhs_end are the same as symbol_start and symbol_end but return the offset of the string matching the n-th item on the right-hand-side of the rule where n is the integer parameter to rhs_start and rhs_end. n is 1 for the leftmost item. rhs_start_pos and rhs_end_pos return a position instead of an offset.


(*Compute the span of the [n]th item of the right-hand-side of the
   matched rule*)
let rhs_loc n = {
  loc_start = Parsing.rhs_start_pos n;
  loc_end = Parsing.rhs_end_pos n;
  loc_ghost = false;
}

The type 'a loc associates a value with a location.


(*A type for the association of a value with a location*)
type 'a loc = { 
  txt : 'a;
  loc : t; 
}

(*Create an ['a loc] value from an ['a] value and location*)
let mkloc (txt : 'a) (loc : t) : 'a loc = { txt ; loc }

(*Create an ['a loc] value bound to the distinguished location called
   [none]*)
let mknoloc (txt : 'a) : 'a loc = mkloc txt none

Error reporting with locations

Location has a framework for error reporting across modules concerned with locations (think lexer, parser, type-checker, etc).


open Format

type error =
{
  loc : t;
  msg : string;
  sub : error list;
}

val error_of_printer : t →  (formatter → 'a → unit) → 'a → error
val errorf_prefixed : ?loc : t → ?sub : error list → ('a, Format.formatter, unit, error) format4 → 'a

So, in the definition of the error record, loc is a location in the source code, msg an explanation of the error and sub a list of related errors. We deal here with the error formatting functions. The utility function print_error_prefix simply writes an error prefix to a formatter.


let error_prefix = "Error"

let warning_prefix = "Warning"

let print_error_prefix (ppf : formatter) () : unit =
  fprintf ppf "@{<error>%s@}:" error_prefix;
  ()

The syntax, "@{<error>%s}@" associates the embedded text with the named tag "error".

Next another utility, pp_ksprintf.


let pp_ksprintf 
    ?(before : (formatter → unit) option) 
    (k : string → 'd)
    (fmt : ('a, formatter, unit, 'd) format4) : 'a =
  let buf = Buffer.create 64 in
  let ppf = Format.formatter_of_buffer buf in
  begin match before with
    | None → ()
    | Some f → f ppf
  end;
  kfprintf
    (fun (_ : formatter) : 'd →
      pp_print_flush ppf ();
      let msg = Buffer.contents buf in
      k msg)
    ppf fmt

It proceeds as follows. A buffer and a formatter over that buffer is created. When presented with all of the arguments of the format operations implied by the fmt argument, if the before argument is non-empty, call it on the formatter. Finally, call kfprintf (from the standard library Format module) which performs the format operations on the buffer before handing control to a function that retrieves the contents of the now formatted buffer and passes them to the user provided continuation k.

With pp_ksprintf at our disposal, one can write the function errorf_prefixed.


let errorf_prefixed 
    ?(loc:t = none) 
    ?(sub : error list = []) 
    (fmt : ('a, Format.formatter, unit, error) format4) : 'a =
  let e : 'a =
    pp_ksprintf
      ~before:(fun ppf → fprintf ppf "%a " print_error_prefix ())
      (fun (msg : string) : error → {loc; msg; sub})
    fmt
  in e

errorf_prefixed computes a function. The function it computes provides the means to produce error values by way of formatting operations to produce the msg field of the error result value. The formatting operations include prefixing the msg field with the error_prefix string. The type of the arguments of the computed function unifies with the type variable 'a. In other words, the type of the computed function is 'a → error. For example, the type of errorf_prefixed "%d %s" is int → string → error.

The groundwork laid with errorf_prefixed above means a polymorphic function error_of_printer can now be produced.


let error_of_printer 
    (loc : t) 
    (printer : formatter → 'error_t → unit) 
    (x : 'error_t) : error =
  let mk_err : 'error_t → error = 
    errorf_prefixed ~loc "%a@?" printer in
  mk_err x

The idea is that error_of_printer is provided a function that can format a value of type 'error. This function is composed with errorf_prefixed thereby producing a function of type 'error → error. For example, we can illustrate how this works by making an error of a simple integer with code like the following:


# error_of_printer none (fun ppf x → Format.fprintf ppf "Code %d" x) 3;;
- : error =
{loc =
  {loc_start =
    {pos_fname = "_none_"; pos_lnum = 1; pos_bol = 0; pos_cnum = -1};
   loc_end = {pos_fname = "_none_"; pos_lnum = 1; pos_bol = 0; pos_cnum = -1};
   loc_ghost = true};
 msg = "Error: Code 3"; sub = []}

So, that's error_of_printer. The following utility function is much simpler - it simply writes a given filename to a formatter.


let print_filename (ppf : formatter) (file : string) : unit =
  fprintf ppf "%s" file

Next, a set of constants for consistent messages that involve locations and a function to get the file, line and column of a position.


let (msg_file, msg_line, msg_chars, msg_to, msg_colon) =
    ("File \"",        (*'msg file'*)
     "\", line ",      (*'msg line'*)
     ", characters ",  (*'msg chars'*)
     "-",              (*'msg to'*)
     ":")              (*'msg colon'*)

let get_pos_info pos = (pos.pos_fname, pos.pos_lnum, pos.pos_cnum - pos.pos_bol)

Making use of the above we have now print_loc : a function to print a location on a formatter in terms of file, line and character numbers.


let print_loc (ppf : formatter) (loc : t) : unit  =
  let (file, line, startchar) = get_pos_info loc.loc_start in
  let endchar = loc.loc_end.pos_cnum - loc.loc_start.pos_cnum + startchar in
  if file = "//toplevel//" then
    fprintf ppf "Characters %i-%i"
      loc.loc_start.pos_cnum loc.loc_end.pos_cnum
  else begin
    fprintf ppf "%s@{<loc>%a%s%i" msg_file print_filename file msg_line line;
    if startchar >= 0 then
      fprintf ppf "%s%i%s%i" msg_chars startchar msg_to endchar;
    fprintf ppf "@}"
  end

Locations generally speaking come out in a format along the lines of: File "<string>, line 1, characters 0-10:"


let print (ppf : formatter) (loc : t) : unit =
  (* The syntax, [@{<loc>%a@}] associates the embedded text with the
     named tag 'loc'*)
  fprintf ppf "@{<loc>%a@}%s@." print_loc loc msg_colon

That last function, print is just a small wrapper over print_loc that appends a colon to the location.

Exception handling involving errors with locations

This section is concerned with the following section of the Location's signature.


val register_error_of_exn : (exn → error option) → unit
val error_of_exn : exn → error option
val error_reporter : (formatter → error → unit) ref
val report_error : formatter → error → unit
val report_exception : formatter → exn → unit

Location contains a mutable list of exception handlers where an exception handler is a function of type exn → error option.


let error_of_exn : (exn → error option) list ref = ref []

A function is provided that adds an exception handler to the above list.


let register_error_of_exn f = error_of_exn := f :: !error_of_exn

The next function error_of_exn (yes, it is the only remaining function that manipulates the list error_exn previously defined directly) walks the list looking for a handler returning the contents of the result of the first such function that doesn't return a None value.


let error_of_exn exn =
  let rec loop = function
    | [] → None
    | f :: rest →
      match f exn with
      | Some _ as r → r
      | None → loop rest
  in
  loop !error_of_exn

We define now a "default" error reporting function. Given a formatter and an error, write the error location, an explanation of the error to the formatter and the same for any associated "sub" errors.


let rec default_error_reporter 
    (ppf : formatter) ({loc; msg; sub} : error) : unit =
  print ppf loc;
  Format.pp_print_string ppf msg;
  List.iter (Format.fprintf ppf "@\n@[<2>%a@]" default_error_reporter) sub

Now, error_reporter itself is a reference cell with default value default_error_reporter.


let error_reporter = ref default_error_reporter

This next function, print_updating_num_loc_lines looks more complicated than it is but does demonstrate a rather advanced usage of Format by containing calls to the functions pp_get_formatter_out_functions, pp_set_formatter_out_functions to tempoarily replace the default function for writing strings. The semantic of the function is to print an error on a formatter incidentally recording the number of lines required to do so.


(* A mutable line count*)
let num_loc_lines : int ref = ref 0

(*Prints an error on a formatter incidentally recording the number of
  lines required to do so*)
let print_updating_num_loc_lines 
    (ppf : formatter) 
    (f : formatter → error → unit) 
    (arg : error) : unit =
  (*A record of functions of output primitives*)
  let out_functions : formatter_out_functions
      = pp_get_formatter_out_functions ppf () in
  (*The strategoy is to temporarily replace the basic function for
    writing a string with this one*)
  let out_string (str : string) (start : int) (len : int) : unit =
    (*A function for counting line breaks in [str]. [c] is the current
      count, [i] is the current char under consideration*)
    let rec count (i : int) (c : int) : int=
      if i = start + len then c
      else if String.get str i = '\n' then count (succ i) (succ c)
      else count (succ i) c 
    in
    (*Update the count*)
    num_loc_lines := !num_loc_lines + count start 0;
    (*Write the string to the formatter*)
    out_functions.out_string str start len 
  in
  (*Replace the standard string output primitive with the one just
    defined *)
  pp_set_formatter_out_functions ppf 
    {out_functions with out_string} ;
  (*Write the error to the formatter*)
  f ppf arg ;
  pp_print_flush ppf ();
  (*Restore the standard primitive*)
  pp_set_formatter_out_functions ppf out_functions

The function report_error uses the currently installed error reporter to write an error report for a given error and formatter incidentally updating a count indicating the number of lines written.


let report_error (ppf : formatter) (err : error) : unit=
  print_updating_num_loc_lines ppf !error_reporter err

This next function, report_exception_rec tries a number of times to find a handler for a given error and if successful formats it. In the worst case a handler is never found and the exception propogates.


let rec report_exception_rec (n : int) (ppf : formatter) (exn : exn) : unit =
  (*Try to find a handler for the exception*)
  try match error_of_exn exn with
  | Some err → 
    (*Format the resulting error using the current error reporter*)
    fprintf ppf "@[%a@]@." report_error err 
  (*The syntax @[%a@]@ writes function output in a box followed by a
    'cut' break hint*)
  | None → raise exn (*A handler could not be found*)
  with exn when n > 0 →
    (*A handler wasn't found. Try again*)
    report_exception_rec (n - 1) ppf exn

The last piece is report_exception. It attempts to write an error report for the given exception on the provided formatter. The exception can be re-raised if no handler is found.


let report_exception (ppf : formatter) (exn : exn) : unit = 
  report_exception_rec 5 ppf exn

Usage

In this section we see how an example of how the above machinery is used. Consider defining a lexical analyzer as an example. Suppose the scanner is defined by the file lexer.mll (the input file to ocamllex). We can imagine its header containing code like the following.


     {
        (*The cases of lexer errors*)
        type error =
          | Illegal_character of char
          | Unterminated_comment of Location.t

        (*The lexer exception type*)
        exception Error of error * Location.t

        (*This function takes a formatter and an instance of type
          [error] and writes a message to the formatter explaining the
          meaning. This is a "printer"*)
        let report_error (ppf : Format.formatter) : error → unit = function
         | Illegal_character c → 
            Format.fprintf ppf "Illegal character (%s)" (Char.escaped c)
         | Unterminated_comment _ → 
            Format.fprintf ppf "Comment not terminated"

        (*Note that [report_error] is a function that unifies with
          the [formatter → 'a → unit] parameter of
          [error_of_printer]*)

        (*Register an exception handler for the lexer exception type*)
        let () =
         Location.register_error_of_exn
          (function
           | Error (err, loc) →
              Some (Location.error_of_printer loc report_error err)
           | _ →  None
          )
     }

     /*...*/
     rule token = ...

A function to handle errors with attached locations (in a REPL for example) is expressible as an idiom as simple as something like this.


let handle_interpreter_error ?(finally=(fun () → ())) ex =
  finally ();
  Location.report_exception (Format.std_formatter) ex

let safe_proc ?finally f =
  try f ()
  with exn → handle_interpreter_error ?finally exn

OCaml Weekly News

2017-05-09T12:00:00-00:00

Mathematics and Computation

2017-03-28T11:27:59-00:00

We are looking for two PhD students at the Faculty of Mathematics and Physics, University of Ljubljana. The programme starts in October 2017 and lasts three years. The positions will be fully funded (subject to approval by the funding agency). The candidates should have a Master’s degree in mathematics or computer science. No knowledge of Slovene is required.

The first PhD student will be advised by dr. Andrej Bauer. The topic of research is foundations of type theory. The candidate should have interest in mathematical aspects of type theory, and familiarity with proof assistants is desirable.

The second PhD student will be advised by dr. Matija Pretnar. The topic of research is the theory of programming languages with a focus on computational effects. The candidate should have interest in both the mathematical foundations and practical implementation of programming languages.

Candidates should send their applications as soon as possible, but no later than the end of April, to Andrej Bauer andrej.bauer@fmf.uni-lj.si or Matija Pretnar matija.pretnar@fmf.uni-lj.si, depending on their primary interest. Please include a short CV, academic record, and a statement of interest.

Shayne Fletcher

2017-05-11T20:18:34-00:00

Polynomials over rings

This post provides a workout in generic programming using modules & functors.

The program presented here models univariate polynomials over rings based on an exercise in "The Module Language" chapter, of Didier Rémy's book, Using, Understanding and Unraveling the OCaml Lanaguage.

Arithmetics and rings

We begin with a type for modules implementing arithmetic.


    module type ARITH = sig
      type t
      val of_int : int -> t            val to_int : t -> int
      val of_string : string -> t      val to_string : t -> string
      val zero : t                     val one : t
      val add : t -> t -> t            val sub : t -> t -> t
      val mul : t -> t -> t            val div : t -> t -> t
      val compare : t -> t -> int      val equal : t -> t -> bool
    end;;

A ring is a set equipped with two binary operations that generalize the arithmetic operations of addition and multiplication.


    module type RING = sig
      type t
      type extern_t
      val print : t -> unit
      val make : extern_t -> t         val show : t -> extern_t
      val zero : t                     val one : t
      val add : t -> t -> t            val mul : t -> t -> t
      val equal : t -> t -> bool
    end;;

We can build rings over arithmetics with functors. This particular one fixes the external representation of the elements of the ring to int.


    module Ring (A : ARITH) :
      RING  with type t = A.t and type extern_t = int =
    struct
      include A
      type extern_t = int
      let make = of_int                let show = to_int
      let print x = print_int (show x)
    end;;

Thus, here for example are rings over various specific arithmetic modules.


    module Ring_int32 = Ring (Int32);;
    module Ring_int64 = Ring (Int64);;
    module Ring_nativeint = Ring (Nativeint);;
    module Ring_int = Ring (
      struct
        type t = int
        let of_int x = x                   let to_int x = x
        let of_string = int_of_string      let to_string = string_of_int
        let zero = 0                       let one = 1
        let add = ( + )                    let sub = ( - )
        let mul = ( * )                    let div = ( / )
        let compare = Pervasives.compare   let equal = ( = )
      end
    );;

Polynomials

We define now the type of polynomials.


    module type POLYNOMIAL = sig
      type coeff (*Type of coefficients*)
      type coeff_extern_t (*Type of coeff. external rep*)

      (*Polynomials satisfy the ring interface*)
      include RING (*Declares a type [t] and [extern_t]*)

      (*Function to evaluate a polynomial at a point*)
      val eval : t -> coeff -> coeff
    end;;

Given a module implementing a ring, we can generate a module implementing polynomials with coefficients drawn from the ring.


    module Polynomial (R : RING) :
      POLYNOMIAL with type coeff = R.t
      and type coeff_extern_t = R.extern_t
      and type extern_t = (R.extern_t * int) list =
    struct
    
      type coeff = R.t (*Coefficient type*)
      type coeff_extern_t = R.extern_t (*External coeff. rep*)
      type extern_t = (coeff_extern_t * int) list (*External polynomial rep*)
    
      (*List of coefficients and their powers*)
      type t = (coeff * int) list (*Invariant : Ordered by powers,
                                    lower order terms at the front*)

      (* ... *)

    end;;

As the comments indicate, the polynomial data structure is a list of pairs of coefficients and powers, ordered so that lower powers come before higher ones. Here's a simple printing utility to aid visualization.


    let print p =
      List.iter
        (fun (c, k) -> Printf.printf "+ (";
          R.print c;
          Printf.printf ")X^%d " k)
        p

In order that we get a canonical representation, null coefficients are eliminated. In particular, the null monomial is simply the empty list.


    let zero = []

The multiplicative identity one is not really necessary as it is just a particular monomial however, its presence makes polynomials themselves satisfy the interface of rings.


    let one = [R.one, 0]

This helper function constructs monomials.


    let monomial (a : coeff) (k : int) =
      if k < 0 then
        failwith "monomial : negative powers not supported"
      else if R.equal a R.zero then [] else [a, k]

Next up, we define addition of polynomials by the following function. Care is taken to ensure the representation invariant is respected.


    let rec add u v =
      match u, v with
      | [], _ -> v
      | _, [] -> u
      | ((c1, k1) :: r1 as p1), ((c2, k2) :: r2 as p2) ->
        if k1 < k2 then
          (c1, k1) :: (add r1 p2)
        else if k1 = k2 then
          let c = R.add c1 c2 in
          if R.equal c R.zero then add r1 r2
          else (c, k1) :: (add r1 r2)
        else (c2, k2) :: (add p1 r2)

With monomial and add avaialable, we can now write make that computes a polynomial from an external representation. We also give the inverse function show here too.


    let make l =
      List.fold_left (fun acc (c, k) ->
        add (monomial (R.make c) k) acc) zero l

    let show p =
      List.fold_right (fun (c, k) acc -> (R.show c, k) :: acc) p []

The module private function times left-multiplies a polynomial by a monomial.


    let rec times (c, k) = function
      | [] -> []
      | (c1, k1) :: q ->
        let c2 = R.mul c c1 in
        if R.equal c2 R.zero then times (c, k) q
        else (c2, k + k1) :: times (c, k) q

Given the existence of times, polynomial multiplication can be expressed in a "one-liner".


    let mul p = List.fold_left (fun r m -> add r (times m p)) zero

Comparing two polynomials for equality is achieved with the following predicate.


    let rec equal p1 p2 =
      match p1, p2 with
      | [], [] -> true
      | (c1, k1) :: q1, (c2, k2) :: q2 ->
        k1 = k2 && R.equal c1 c2 && equal q1 q2
      | _ -> false

In the course of evaluating polynomials for a specific value of their indeterminate, we'll require a function for computing powers. The following routine uses the exponentiation by squaring technique.


    let rec pow c = function
      | 0 -> R.one
      | 1 -> c
      | k ->
        let l = pow c (k lsr 1) in
        let l2 = R.mul l l in
        if k land 1 = 0 then l2 else R.mul c l2

Finally, the function eval for evaluation of a polynomial at a specific point. The implementation uses Horner's rule for computationally efficient evaluation.


    let eval p c = match List.rev p with
      | [] -> R.zero
      | (h :: t) ->
        let reduce (a, k) (b, l) =
          let n = pow c (k - l) in
          let t = R.add (R.mul a n) b in
          (t, l)  in
        let a, k = List.fold_left reduce h t in
        R.mul (pow c k) a

Testing and example usage

The following interactive session creates a polynomial with integer coefficients module and uses it to confirm the equivalence of $(1 + x)(1 - x)$ with $(1 - x^{2})$.


    # #use "polynomial.ml";;
    # module R = Ring_int;;
    # module P = Polynomial (R);;
    # let p = P.mul (P.make [(1, 0); (1, 1)]) (P.make [(1, 0); (-1, 1)]);;
    # let q = P.make [(1, 0); (-1, 2)];;
    # P.equal p q;;
    - : bool = true

Polynomials in two variables, can be treated as univariate polynomials with polynomial coefficients. For example, the polynomial $(X + Y)$ can be regarded as $(1 \times X^{1})Y^{0} + (1 \times X^{0})Y^{1}$. Similarly we can write $X - Y$ as $(1 \times X^{1})Y^{0} + (-1 \times X^{0}) Y^1$ and now check the equivalence $(X + Y)(X - Y) = (X^{2} - Y^{2})$.


    #module Y = Polynomial (P);;
    
    #(* (X + Y) *)
    #let r = Y.make [
    #  ([1, 1], 0); (*(1 X^1) Y^0*)
    #  ([1, 0], 1)  (*(1 X^0) Y^1*)
    #];;

    #(* (X - Y) *)
    #let s = Y.make [
    #  ([1, 1], 0); (*(1 X^1) Y^0*)
    #  ([-1, 0], 1) (*((-1) X^0) Y^1*)
    #];;

    #(* (X^2 - Y^2) *)
    #let t = Y.make [
    #  ([1, 2], 0);   (*(1 X^2) Y^0*)
    #  ([-1, 0], 2)  (* (-1 X^0) Y^2*)
    #];;

    #Y.equal (Y.mul r s) t;;
    - : bool = true

OCaml Weekly News

2017-05-09T12:00:00-00:00

OCaml – Jane Street Tech Blogs

2017-05-11T19:02:45-00:00

From now and then, I found myself having to write some mechanical and repetitive code. The usual solution for this is to write a code generator; for instance in the form of a ppx rewriter in the case of OCaml code. This however comes with a cost: code generators are harder to review than plain code and it is a new syntax to learn for other developers. So when the repetitive pattern is local to a specific library or not widely used, it is often not worth the effort. Especially if the code in question is meant to be reviewed and maintained by several people.

Then there is the possibility of using a macro pre-processor such as cpp or cppo which is the equivalent of cpp but for OCaml. This can help in some cases but this has a cost as well:

macros generally make the code harder to read
errors tends to be harder to understand since they don't point where you'd expect
you can say goodbye to merlin

In fact, when the repetitive pattern is specific to one particular case and of reasonable size, committing and reviewing the generated code is acceptable. That's the problem Cinaps tries to solve.

What is cinaps?

Cinaps is an application that reads input files and recognize special syntactic forms. Such forms are expected to embed some OCaml code printing something to stdout. What they print is compared against what follow these special forms. The rest works exactly the same as expectation tests.

The special form is (*$ <ocaml-code> *) for ml source files, /*$ <ocaml-code> */ for C source files and #|$ <ocaml-code> |# for S-expression files.

For instance:

$ cat file.ml
let x = 1
(*$ print_newline ();
    List.iter (fun s -> Printf.printf "let ( %s ) = Pervasives.( %s )\n" s s)
      ["+"; "-"; "*"; "/"] *)
(*$*)
let y = 2

$ cinaps file.ml
---file.ml
+++file.ml.corrected
File "file.ml", line 5, characters 0-1:
  let x = 1
  (*$ print_newline ();
      List.iter (fun s -> Printf.printf "let ( %s ) = Pervasives.( %s )\n" s s)
        ["+"; "-"; "*"; "/"] *)
+|let ( + ) = Pervasives.( + )
+|let ( - ) = Pervasives.( - )
+|let ( * ) = Pervasives.( * )
+|let ( / ) = Pervasives.( / )
  (*$*)
  let y = 2

$ echo $?
1
$ cp file.ml.corrected file.ml
$ cinaps file.ml
$ echo $?
0

Real example

What follows is a real example where using Cinaps made the code much easier to write and maintain. However, I changed the names for this blog post since this code is not released publicly. Note also that this example shows one way we usually write C bindings at Jane Street. It is not meant as a model of how to write C bindings, and the excellent ctypes library should be the default choice in most cases. However, this code pre-dates ctypes and migrating it would be quite a lot of work.

The example itself is part of a C binding that I wrote a few years ago. While doing so I used Core.Flags in order to represent a few C enumerations on the OCaml side. Core.Flags is a module providing a nice abstraction for representing C flags.

The OCaml code looks like what you'd expect from code using Core.Flags:

module Open_flags = struct
  external get_rdonly   : unit -> Int63.t = "mylib_O_RDONLY"   [@@noalloc]
  external get_wronly   : unit -> Int63.t = "mylib_O_WRONLY"   [@@noalloc]
  external get_rdwr     : unit -> Int63.t = "mylib_O_RDWR"     [@@noalloc]
  external get_nonblock : unit -> Int63.t = "mylib_O_NONBLOCK" [@@noalloc]
  external get_append   : unit -> Int63.t = "mylib_O_APPEND"   [@@noalloc]
  external get_creat    : unit -> Int63.t = "mylib_O_CREAT"    [@@noalloc]
  external get_trunc    : unit -> Int63.t = "mylib_O_TRUNC"    [@@noalloc]
  external get_excl     : unit -> Int63.t = "mylib_O_EXCL"     [@@noalloc]
  external get_noctty   : unit -> Int63.t = "mylib_O_NOCTTY"   [@@noalloc]
  external get_dsync    : unit -> Int63.t = "mylib_O_DSYNC"    [@@noalloc]
  external get_sync     : unit -> Int63.t = "mylib_O_SYNC"     [@@noalloc]
  external get_rsync    : unit -> Int63.t = "mylib_O_RSYNC"    [@@noalloc]
 
  let rdonly   = get_rdonly   ()
  let wronly   = get_wronly   ()
  let rdwr     = get_rdwr     ()
  let nonblock = get_nonblock ()
  let append   = get_append   ()
  let creat    = get_creat    ()
  let trunc    = get_trunc    ()
  let excl     = get_excl     ()
  let noctty   = get_noctty   ()
  let dsync    = get_dsync    ()
  let sync     = get_sync     ()
  let rsync    = get_rsync    ()
 
  include Flags.Make(struct
      let known =
        [ rdonly   , "rdonly"
        ; wronly   , "wronly"
        ; rdwr     , "rdwr"
        ; nonblock , "nonblock"
        ; append   , "append"
        ; creat    , "creat"
        ; trunc    , "trunc"
        ; excl     , "excl"
        ; noctty   , "noctty"
        ; dsync    , "dsync"
        ; sync     , "sync"
        ; rsync    , "rsync"
        ]
      let remove_zero_flags = false
      let allow_intersecting = false
      let should_print_error = true
    end)
end

And there are about 3 modules like this in this file, plus the corresponding stubs in the C file. Writing this code initially was no fun, and adding new flags now that the C library has evolved is still no fun.

The rest of this section explains how to make it more fun with cinaps.

Setting up and using cinaps

First I add a rule in the build system to call cinaps appropriately. I use a few settings specific to our jenga based builds and it is currently not possible to replicate this outside of Jane Street, but assuming you have a Makefile, you can write:

.PHONY: cinaps
cinaps:
    cinaps -i src/*.ml src/*.c

Now whenever you call make cinaps, all the files will be updated in place. You can then do git diff to see what changed.

Then I write a file src/cinaps_helpers. It is plain OCaml source file, however it is not suffixed with .ml so that it is not confused with a regular module of the library. It contains the various bits that are common between the ml/C files in the library:

(* -*- tuareg -*- *)
 
let stub_prefix = "mylib_"
let stub name = stub_prefix ^ name
 
let open_flags =
  [ "O_RDONLY"
  ; "O_WRONLY"
  ; "O_RDWR"
  ; "O_NONBLOCK"
  ; "O_APPEND"
  ; "O_CREAT"
  ; "O_TRUNC"
  ; "O_EXCL"
  ; "O_NOCTTY"
  ; "O_DSYNC"
  ; "O_SYNC"
  ; "O_RSYNC"
  ]
 
let other_flags =
  [ ...
  ]
 
 
let yet_other_flags =
  [ ...
  ]
 
let all_flags = open_flags @ other_flags @ yet_other_flags
 
open StdLabels
open Printf
let pr fmt = printf (fmt ^^ "\n")
 
let flags_module module_name flags ~prefix ~allow_intersection =
  <code to print an Open_flags like module>

Now, in my original .ml file, I can write:

(*$ #use "cinaps_helpers" $*)
 
(*$ flags_module "Open_flags" open_flags ~prefix:"O_" ~allow_intersecting:false *)
module Open_flags = struct
  external get_rdonly   : unit -> Int63.t = "mylib_O_RDONLY"   [@@noalloc]
  external get_wronly   : unit -> Int63.t = "mylib_O_WRONLY"   [@@noalloc]
  external get_rdwr     : unit -> Int63.t = "mylib_O_RDWR"     [@@noalloc]
  external get_nonblock : unit -> Int63.t = "mylib_O_NONBLOCK" [@@noalloc]
  external get_append   : unit -> Int63.t = "mylib_O_APPEND"   [@@noalloc]
  external get_creat    : unit -> Int63.t = "mylib_O_CREAT"    [@@noalloc]
  external get_trunc    : unit -> Int63.t = "mylib_O_TRUNC"    [@@noalloc]
  external get_excl     : unit -> Int63.t = "mylib_O_EXCL"     [@@noalloc]
  external get_noctty   : unit -> Int63.t = "mylib_O_NOCTTY"   [@@noalloc]
  external get_dsync    : unit -> Int63.t = "mylib_O_DSYNC"    [@@noalloc]
  external get_sync     : unit -> Int63.t = "mylib_O_SYNC"     [@@noalloc]
  external get_rsync    : unit -> Int63.t = "mylib_O_RSYNC"    [@@noalloc]
 
  let rdonly   = get_rdonly   ()
  let wronly   = get_wronly   ()
  let rdwr     = get_rdwr     ()
  let nonblock = get_nonblock ()
  let append   = get_append   ()
  let creat    = get_creat    ()
  let trunc    = get_trunc    ()
  let excl     = get_excl     ()
  let noctty   = get_noctty   ()
  let dsync    = get_dsync    ()
  let sync     = get_sync     ()
  let rsync    = get_rsync    ()
 
  include Flags.Make(struct
      let known =
        [ rdonly   , "rdonly"
        ; wronly   , "wronly"
        ; rdwr     , "rdwr"
        ; nonblock , "nonblock"
        ; append   , "append"
        ; creat    , "creat"
        ; trunc    , "trunc"
        ; excl     , "excl"
        ; noctty   , "noctty"
        ; dsync    , "dsync"
        ; sync     , "sync"
        ; rsync    , "rsync"
        ]
      let remove_zero_flags = false
      let allow_intersecting = false
      let should_print_error = true
    end)
end
(*$*)

And cinaps will check that the text between the (*$ ... *) and (*$*) forms is what is printed by flags_module "Open_flags" .... I write something similar in the .c file. Note the initial (*$ ... $*) form, which is not expected to print anything and is only used for its other side effects.

Adding new flags become trivial: add it to the list in src/cinaps_helper and execute make cinaps.

Pushing the system

Now I decide that I don't like the fact that all my constant flags are initialized at runtime and I want them to be static constant on the ml side. A simple way to do this is to write a C program that include the right headers and output a .ml file defining these constants. I use cynaps to write this C file as well:

/*$ #use "cinaps_helpers" $*/
 
#include <stdio.h>
 
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
 
int main()
{
  printf("open Core\n");
  printf("let mk = Int63.of_int_exn\n");
  /*$
    printf "\n";
    let len = longest all_flags in
    List.iter all_flags ~f:(fun f ->
      pr {|  printf("let _%-*s = mk %%d\n", %-*s);|} len f len f );
    printf "  " */
  printf("let _O_RDONLY   = mk %d\n", O_RDONLY  );
  printf("let _O_WRONLY   = mk %d\n", O_WRONLY  );
  printf("let _O_RDWR     = mk %d\n", O_RDWR    );
  printf("let _O_NONBLOCK = mk %d\n", O_NONBLOCK);
  printf("let _O_APPEND   = mk %d\n", O_APPEND  );
  printf("let _O_CREAT    = mk %d\n", O_CREAT   );
  printf("let _O_TRUNC    = mk %d\n", O_TRUNC   );
  printf("let _O_EXCL     = mk %d\n", O_EXCL    );
  printf("let _O_NOCTTY   = mk %d\n", O_NOCTTY  );
  printf("let _O_DSYNC    = mk %d\n", O_DSYNC   );
  printf("let _O_SYNC     = mk %d\n", O_SYNC    );
  printf("let _O_RSYNC    = mk %d\n", O_RSYNC   );
  /*$*/
  return 0;
}

Updating the various flag modules in the the ml code is as simple as editing src/cinaps_helpers and doing make cinaps:

(*$ flags_module "Open_flags" open_flags ~prefix:"O_" ~allow_intersecting:false *)
module Open_flags = struct
  let rdonly   = Consts._O_RDONLY
  let wronly   = Consts._O_WRONLY
  let rdwr     = Consts._O_RDWR
  let nonblock = Consts._O_NONBLOCK
  let append   = Consts._O_APPEND
  let creat    = Consts._O_CREAT
  let trunc    = Consts._O_TRUNC
  let excl     = Consts._O_EXCL
  let noctty   = Consts._O_NOCTTY
  let dsync    = Consts._O_DSYNC
  let sync     = Consts._O_SYNC
  let rsync    = Consts._O_RSYNC
 
  include Flags.Make(struct
      let known =
        [ Consts._O_RDONLY   , "rdonly"
        ; Consts._O_WRONLY   , "wronly"
        ; Consts._O_RDWR     , "rdwr"
        ; Consts._O_NONBLOCK , "nonblock"
        ; Consts._O_APPEND   , "append"
        ; Consts._O_CREAT    , "creat"
        ; Consts._O_TRUNC    , "trunc"
        ; Consts._O_EXCL     , "excl"
        ; Consts._O_NOCTTY   , "noctty"
        ; Consts._O_DSYNC    , "dsync"
        ; Consts._O_SYNC     , "sync"
        ; Consts._O_RSYNC    , "rsync"
        ]
      let remove_zero_flags = false
      let allow_intersecting = false
      let should_print_error = true
    end)
end
(*$*)

Tweak: indenting the generated code

You can either write cinaps code that produce properly indented code, or you can use the styler option:

.PHONY: cinaps
cinaps:
    cinaps -styler ocp-indent -i src/*.ml src/*.c

History behind the name

I initially wrote this tool while I did some work on the ocaml-migrate-parsetree project. ocaml-migrate-parsetree was started by Alain Frisch and continued by Frederic Bour and aims at providing a solid and stable base for authors of ppx rewriters or other tools using the OCaml frontend. I helped a bit during development and did some testing on a large scale while rebasing our ppx infrastructure on top it.

Due to its nature, this project contains a lot of repetitive code that cannot be factorized other than by using some kind of meta-programming. Initially we had a small pre-preprocessor that was interpreting a made-up syntax and was working like cpp does. The syntax was yet another DSL and the generated code was generated on the fly. This made the .ml and .mli files harder to understand since you had to decode this DSL in order to understand what the code was.

Cinaps replaced this tool and the name was chosen to emphasize that it is not a preprocessor. It means "Cinaps Is Not A Preprocessing System".

Status

Cinaps is published on github and is part of the upcoming v0.9 Jane Street release. The version that is published doesn't yet support the C/S-expression syntaxes but once the stable release has gone through, an updated version of Cinaps supporting these syntaxes will be released.

OCamlPro

2017-05-11T13:20:05-00:00

The new opam 2.0 release, currently in beta, introduces several new features. This post gets into some detail on the new opam build command, its purpose, its use, and some implementation aspects.

opam build is run from the source tree of a project, and does not rely on a pre-existing opam installation. As such, it adds a new option besides the existing workflows based on managing shared OCaml installations in the form of switches.

What does it do ?

Typically, this is used in a fresh git clone of some OCaml project. Like when pinning the package, opam will find and leverage package definitions found in the source, in the form of opam files.

if opam hasn’t been initialised (no ~/.opam), this is taken care of.
if no switch is otherwise explicitely selected, a local switch is used, and
created if necessary (i.e. in ./_opam/)
the metadata for the current project is registered, and the package installed
after its dependencies, as opam usually does

This is particularly useful for distributing projects to people not used to opam and the OCaml ecosystem: the setup steps are automatically taken care of, and a single opam build invocation can take care of resolving the dependency chains for your package.

If building the project directly is preferred, adding --deps-only is a good way to get the dependencies ready for the project:

  $ opam build --deps-only
  $ eval $(opam config env)
  $ ./configure; make; etc.

Note that if you just want to handle project-local opam files, opam build can also be used in your existing switches: just specify --no-autoinit, --switch or make sure the OPAMSWITCH variable is set. E.g. opam build --no-autoinit --deps-only is a convenient way to get the dependencies for the local project ready in your current switch.

Additional functions

Installation

The installation of the packages happens as usual to the prefix corresponding to the switch used (<project-root>/_opam/ for a local switch). But it is possible, with --install-prefix, to further install the package to the system:

opam build --install-prefix ~/local

will install the results of the package found in the current directory below ~/local.

The dependencies of the package won’t be installed, so this is intended for programs, assuming they are relocatable, and not for libraries.

Choosing custom repositories

The user can pre-select the repositories to use on the creation of the local switch with:

  opam build --repositories <repos>

where <repos> is a comma-separated list of repositories, specified either as name=URL, or name if already configured on the system.

Multiple packages

Multiple packages are commonly found to share a single repository. In this case, opam build registers and builds all of them, respecting cross-dependencies. The opam files to use can also be explicitely selected on the command-line. In this case, specific opam files must be named <package-name>.opam.

Implementation details

The choice of the compiler, on automatic initialisation, is either explicit, using the --compiler option, or automatic. In the latter case, the default selection is used (see opam init --help, section “CONFIGURATION FILE” for details), but a compiler compatible with the local packages found is searched from that. This allows, for example, to choose a system compiler when available and compatible, avoiding a recompilation of OCaml.

When using --install-prefix, the normal installation is done, then the tracking of package-installed files, introduced in opam 2.0, is used to extract the installed files from the switch and copy them to the prefix.

The packages installed through opam build are not registered in any repository, and this is not an implicit use of opam pin: the rationale is that packages installed this way will also be updated by repeating opam build. This means that when using other commands, e.g. opam upgrade, opam won’t try to keep the packages to their local, source version, and will either revert them to their repository definition, or remove them, if they need recompilation.

Planned extensions

This is still in beta: there are still rough edges, please experiment and give feedback! It is still possible that the command syntax and semantics change significantly before release.

Another use-case that we are striving to improve is sharing of development setups (share sets of pinned packages, depend on specific remotes or git hashes, etc.). We have many ideas to improve on this, but opam build is not, as of today, a direct solution to this. In particular, installing this way still relies on the default opam repository; a way to define specific options for the switch that is implicitely created on opam build is in the works.

NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!

OCaml – Jane Street Tech Blogs

2017-05-11T19:02:45-00:00

I'm happy to announce our next public tech talk, called Seven Implementations of Incremental, on Wednesday, April 5th, presented by yours truly. You can register here.

The talk covers the history of Incremental, a library for building efficient online algorithms. The need to update computations incrementally is pretty common, and we've found Incremental to be useful in creating such computations in a number of different domains, from constructing efficient financial calculations to writing responsive, data-rich web UIs.

The ideas behind Incremental aren't new with us; there is a lot of prior art, most notably the work from Umut Acar's work on self-adjusting computations, on which Incremental is most directly modeled.

But there's a big gap between the academic version of an idea and a production ready instantiation, and this talk is about crossing that gap. It discusses the 7 different implementations we went through and the various mistakes we made along the way towards the current one we use in production.

So join us. I hope you enjoy seeing what we learned about building this kind of system, as well as hearing about the hilarious pratfalls along the way.

On another note, we have finally posted videos from our two previous talks, including Brian Nigito's talk on the architecture of the modern exchange, and Arjun Guha's talk on taming Puppet. And, of course, you can subscribe to our channel while you're there.

OCaml Weekly News

2017-05-09T12:00:00-00:00

Gagallium

2017-03-14T08:00:00-00:00

I have been asked whether an automatically-generated visitor, as produced by the visitors syntax extension for OCaml, can be used to construct an iterator.

It turns out that this can be done in a simple and efficient manner. (Up to a constant factor, the time complexity of this solution is optimal.) As the problem is interesting and its solution is somewhat nonobvious, I am describing them here.

To play with this code in an OCaml toplevel, first install visitors via the command opam install visitors. Then, launch ocaml and type this:

#use "topfind";;
#require "visitors.ppx";;
#require "visitors.runtime";;

Problem Statement

Suppose we have an arbitrary data structure that contains elements of type 'a. Here, it is a binary tree, but it could be anything:

type 'a sometree =
  | Leaf
  | Node of 'a sometree * 'a * 'a sometree

We would like to enumerate the elements of this data structure. More precisely, we would like to construct an iterator, that is, an on-demand producer of elements. Here is a simple definition of a stateful iterator:

type 'a iterator =
  unit -> 'a option

The question is, can we construct an iterator for the type 'a sometree, based on an automatically-generated visitor, so that the construction is entirely independent of the type 'a sometree?

Cascades

For starters, let us define cascades, which are a more pleasant kind of iterators. A cascade is a persistent (stateless) iterator. It can be thought of as a delayed list, that is, a list whose elements are computed only on demand.

Cascades could (should) be part of a separate library. As the time of writing (March, 2017), there is in fact a proposal to add them to OCaml's standard library.

type 'a cascade =
  unit -> 'a head

and 'a head =
  | Nil
  | Cons of 'a * 'a cascade

A delayed computation is represented as a function of type unit -> _.

The cascade constructors are as follows:

let nil : 'a cascade =
  fun () -> Nil

let cons (x : 'a) (xs : 'a cascade) : 'a cascade =
  fun () -> Cons (x, xs)

Forcing a cascade reveals its head:

let force xs =
  xs()

A cascade can be easily converted to a stateful iterator:

let cascade_to_iterator (xs : 'a cascade) : 'a iterator =
  let s = ref xs in
  fun () ->
    match force !s with
    | Nil ->
        s := nil;
        None
    | Cons (x, xs) ->
        s := xs;
        Some x

In the above code, writing nil into s may seem superfluous, but is in fact necessary to guarantee that the computation that led to a Nil outcome is not repeated in the future.

Because cascades are close cousins of lists, they are easy to work with. Constructing a cascade for a tree-like data structure is straightforward, whereas directly constructing a stateful iterator would be more involved.

Back to the problem

Now, can we use some kind of visitor to turn a tree of type 'a sometree into a cascade of type 'a cascade?

At first sight, this does not seem very easy, because of two issues:

a visitor usually traverses a tree in an eager manner, whereas we need the traversal to make progress only as cascade elements are demanded;
a visitor performs a bottom-up computation, without a left-to-right bias (assuming mutable state is not used), whereas a cascade enumerates elements in a left-to-right manner. (Or in a right-to-left manner. As will be apparent below, both directions are possible.)

The trick is to use another intermediate step. Instead of turning a tree directly into a cascade, we first transform it into a generic tree-like structure: a delayed tree. The first issue above is solved because, by introducing delays into the new tree, we allow its construction to be carried out on demand. The second issue is solved because this tree-to-tree transformation can be carried out in a purely bottom-up manner by a reduce visitor. Then, finally, it is straightforward to transform a delayed tree into a cascade.

From Delayed Trees to Cascades and Iterators

A delayed tree contains ordinary nodes of arity 0, 1, and 2. Furthermore, it contains DTDelay nodes, of arity 1, whose child is delayed, that is, computed only on demand.

type 'a delayed_tree =
  | DTZero
  | DTOne of 'a
  | DTTwo of 'a delayed_tree * 'a delayed_tree
  | DTDelay of (unit -> 'a delayed_tree)

A delayed tree is converted to a cascade as follows. As is often the case, when building a cascade, one must take a continuation k as an argument, so as to avoid naive and costly cascade concatenation operations. Thus, the specification of the function call delayed_tree_to_cascade dt k is to construct a cascade whose elements are the elements of the delayed tree dt (listed from left to right), followed with the elements of the cascade k.

let rec delayed_tree_to_cascade (dt : 'a delayed_tree) (k : 'a cascade)
: 'a cascade =
  fun () -> delayed_tree_to_head dt k

and delayed_tree_to_head (dt : 'a delayed_tree) (k : 'a cascade) : 'a head =
  match dt with
  | DTZero ->
      force k
  | DTOne x ->
      Cons (x, k)
  | DTTwo (dt1, dt2) ->
      delayed_tree_to_head dt1 (delayed_tree_to_cascade dt2 k)
  | DTDelay dt ->
      delayed_tree_to_head (force dt) k

let delayed_tree_to_cascade (dt : 'a delayed_tree) : 'a cascade =
  delayed_tree_to_cascade dt nil

let delayed_tree_to_iterator (dt : 'a delayed_tree) : 'a iterator =
  cascade_to_iterator (delayed_tree_to_cascade dt)

In the above code, we have chosen to perform a left-to-right traversal of the delayed tree, but could just as well have chosen a right-to-left traversal.

It is possible to make the delayed tree data structure implicit in the code; this is explained further on.

Constructing Delayed Trees

The type 'a delay is a synonym for 'a. We will use it as a decoration, in a type definition, to indicate that a call to the method visit_delay is desired.

type 'a delay = 'a

We now set up four constructor functions or constructor methods, which construct delayed trees, and which we will use in a reduce visitor.

Delayed trees form a monoid, in the sense that we concatenate them using DTTwo, and the neutral element is DTZero. We package these two data constructors in the methods zero and plus, which are automatically called in an automatically-generated reduce visitor.

The visitor method visit_delay delays the visit of a subtree by constructing and returning a DTDelay node, which carries a delayed recursive call to a visitor.

The visitor function yield will be invoked at elements of type 'a. It constructs a one-element delayed tree.

class ['self] delayed_tree_monoid = object (_ : 'self)

  method zero =
    DTZero

  method plus s1 s2 =
    match s1, s2 with
    | DTZero, s
    | s, DTZero ->
        s
    | _, _ ->
        DTTwo (s1, s2)

  method visit_delay: 'env 'a .
    ('env -> 'a -> 'b delayed_tree) ->
    'env -> 'a delay -> 'b delayed_tree
  = fun visit_'a env x ->
      DTDelay (fun () -> visit_'a env x)

end

let yield _env x =
  DTOne x

In the method plus, above, treating DTZero specially is not mandatory. It is an optimization, which helps allocate fewer nodes.

Generating a Visitor

It is now time to generate a reduce visitor for the type 'a sometree. This is the only part of the code which is specific of sometree. Everything else is generic.

We must insert delays into the structure of the type 'a sometree so as to indicate where visit_delay should be called and (therefore) where DTDelay nodes should be allocated. To do this, we write a copy of the definition of the type 'a sometree, with extra uses of delay in it. The new type is actually considered equal to 'a sometree by OCaml. Its role is purely to carry a @@deriving visitors annotation.

In the data constructor Node, the left-hand delay is in fact superfluous. With or without it, our iterators will eagerly descend along the leftmost branch of a tree, anyway.

type 'a mytree = 'a sometree =
  | Leaf
  | Node of 'a mytree delay * 'a * 'a mytree delay

and 'a mytree_delay =
  'a mytree delay

[@@deriving visitors { variety = "reduce"; polymorphic = true;
                       concrete = true; ancestors = ["delayed_tree_monoid"] }]

This approach is pleasant insofar as one controls exactly where delays are inserted. However, it requires copying the type definition, which may be unpleasant. Another approach is described further on.

Using the Generated Visitor

For demonstration purposes, let us make our visitor verbose. In production, one should remove verbose_reduce and use reduce instead.

class ['self] verbose_reduce = object (_ : 'self)
  inherit [_] reduce as super
  method! visit_Leaf visit_'a env =
    Printf.printf "Visiting a leaf.\n%!";
    super#visit_Leaf visit_'a env
  method! visit_Node visit_'a env t1 x t2 =
    Printf.printf "Visiting a node.\n%!";
    super#visit_Node visit_'a env t1 x t2
end

The problem is solved! There remains to write a couple lines of glue code:

let sometree_to_delayed_tree (t : 'a sometree) =
  new verbose_reduce # visit_mytree_delay yield () t

let sometree_to_iterator (t : 'a sometree) : 'a iterator =
  delayed_tree_to_iterator (sometree_to_delayed_tree t)

We use visit_mytree_delay above, even though visit_mytree would work just as well, so as to ensure that we get a delayed tree whose root is a DTDelay node. Thus, absolutely no work is performed when the iterator is created; iteration begins only when the first element is demanded.

Demo

A little demo helps see what is going on.

let t : int sometree =
  Node (Node (Leaf, 1, Leaf), 2, Node (Leaf, 3, Leaf))

let i : int iterator =
  sometree_to_iterator t

Here is a transcript of an OCaml toplevel session:

  # i();;
  Visiting a node.
  Visiting a node.
  Visiting a leaf.
  - : int option = Some 1
  # i();;
  Visiting a leaf.
  - : int option = Some 2
  # i();;
  Visiting a node.
  Visiting a leaf.
  - : int option = Some 3
  # i();;
  Visiting a leaf.
  - : int option = None
  # i();;
  - : int option = None

Variant: Avoiding Duplication of the Type Definition

Earlier, we have generated a visitor for the existing type 'a sometree in a posteriori style. We have manually created an isomorphic copy of the type 'a sometree, which we have named 'a mytree, and have annotated this copy with [@@deriving visitors { ... }]. Furthermore, we have taken this opportunity to insert delay type constructors into the type, so as to influence the generated visitor.

This style is relatively pleasant because it is declarative and lets us control exactly where delays are inserted. However, it requires duplicating the definition of the type 'a sometree, which may be unpleasant (if the definition is large) or impossible (if the definition is hidden behind an abstraction barrier).

Another approach is to generate a visitor in a priori style. When the type 'a sometree is first defined, a reduce visitor can be immediately generated for it, as follows:

type 'a sometree =
  | Leaf
  | Node of 'a sometree * 'a * 'a sometree

[@@deriving visitors { variety = "reduce"; polymorphic = true;
                       name = "sometree_reduce" }]

At this point, we pretend that we do not know yet what this visitor will be used for, so we do not annotate the type definition with delay, and do not use delayed_tree_monoid as a base class. We get a visitor class, named sometree_reduce. This class has two virtual methods, zero and plus.

Then, we create a subclass, named reduce, which we tailor to our needs. We mix the generated class sometree_reduce with the class delayed_tree_monoid, and insert a delay at every tree node by overriding the method visit_sometree:

class ['self] reduce = object (self : 'self)
  inherit [_] sometree_reduce as super
  inherit [_] delayed_tree_monoid
  method! visit_sometree visit_'a env t =
    self#visit_delay (super#visit_sometree visit_'a) env t
end

The rest of the code is unchanged (except the method visit_mytree_delay no longer exists; one calls visit_sometree instead).

Variant: Getting Rid of Explicit Delayed Trees

I like to present delayed trees as an explicit data structure, because this helps understand what is going on. However, if desired, it is possible to hide them by refunctionalization (the opposite of defunctionalization).

Above, the function delayed_tree_to_cascade was written with the help of an auxiliary function, delayed_tree_to_head. We could also have written it directly, as follows:

let rec delayed_tree_to_cascade (dt : 'a delayed_tree) (k : 'a cascade)
: 'a cascade =
  match dt with
  | DTZero ->
      k
  | DTOne x ->
      cons x k
  | DTTwo (dt1, dt2) ->
      delayed_tree_to_cascade dt1 (delayed_tree_to_cascade dt2 k)
  | DTDelay dt ->
      fun () -> delayed_tree_to_cascade (force dt) k ()

In this form, the only function that is ever applied to a delayed tree is delayed_tree_to_cascade. Therefore, wherever we construct a delayed tree t, we could instead directly build a closure whose behavior is equivalent to delayed_tree_to_cascade t. This is refunctionalization.

The data structure of delayed trees seems to disappears. The type 'a delayed_tree can be defined as a synonym for a 'a cascade -> 'a cascade. I usually refer to this type as 'a producer: it is the type of a function that concatenates elements in front of the cascade that it receives as an argument.

type 'a producer =
  'a cascade -> 'a cascade

type 'a delayed_tree =
  'a producer

The four data constructors are defined as follows:

let _DTZero k =
  k

let _DTOne x k =
  cons x k

let _DTTwo dt1 dt2 k =
  dt1 (dt2 k)

let _DTDelay dt k =
  fun () -> force dt k ()

The reader can check that the types of these data constructors are the same as in the previous version of the code. For instance, _DTDelay has type (unit -> 'a delayed_tree) -> 'a delayed_tree.

The root function delayed_tree_to_cascade (without a continuation argument) is defined as follows:

let delayed_tree_to_cascade (dt : 'a delayed_tree) : 'a cascade =
  dt nil

The delayed tree monoid uses the new versions of the four data constructors:

class ['self] delayed_tree_monoid = object (_ : 'self)

  method zero =
    _DTZero

  method plus =
    _DTTwo

  method visit_delay: 'env 'a .
    ('env -> 'a -> 'b delayed_tree) ->
    'env -> 'a delay -> 'b delayed_tree
  = fun visit_'a env x ->
      _DTDelay (fun () -> visit_'a env x)

end

let yield _env x =
  _DTOne x

The four functions _DTZero, _DTOne, _DTTwo, and _DTDelay could be inlined, if desired, so as to make the above code seem even more concise.

The rest of the code is unchanged.

Acknowledgements

KC Sivaramakrishnan asked whether a visitor can be used to construct an iterator. Gabriel Scherer pointed out that the delayed tree data structure can be hidden by refunctionalization.

Shayne Fletcher

2017-05-11T20:18:34-00:00

Universal type

A universal type is a type into which all other types can be embedded. A module implementing such a type here will satisfy the following signature.


module type UNIVERSAL = sig
  type t
  val embed : unit → (α → t) * (t → α option)
end;;

The type t is the universal type and each call to embed returns a pair of functions : an injection function for embedding a value into the universal type and, a projection function for extracting the value from its embedding in the universal type. The following code demonstrates intended usage.


module type TEST = sig
  val run : unit → unit
end;;

module type UNIVERSAL_TEST = functor (U : UNIVERSAL) → TEST;;

module Basic_usage : UNIVERSAL_TEST = 
  functor (U : UNIVERSAL) → struct
    let run () =
      let ((of_int : int → U.t)
         , (to_int : U.t → int option)) = U.embed () in
      let ((of_string : string → U.t)
         , (to_string : U.t → string option)) = U.embed () in

      let r : U.t ref = ref (of_int 13) in

      begin
        assert (to_int !r = Some 13);
        assert (to_string !r = None);

        r := of_string "foo";

        assert (to_string !r = Some "foo");
        assert (to_int !r = None);
      end
  end;;

One possible implementation is via the use of exceptions together with local modules. The core idea exploits the fact that the primitive type exn is an open extensible sum. Here's the complete implementation. We'll break it down later.


module Universal_exn : UNIVERSAL = struct

  type t = exn

  module type ANY = sig
    type c
    exception E of c
  end

  type α any = (module ANY with type c = α)

  let mk : unit → α any =
    fun (type s) () →
      (module struct
        type c = s
        exception E of c
      end : ANY with type c = s)

  let inj (type s) (p : s any) (x : s) : t =
    let module Any = (val p : ANY with type c = s) in
    Any.E x

  let proj (type s) (p : s any) (y : t) : s option =
    let module Any = (val p : ANY with type c = s) in
    match y with
    | Any.E x → Some x
    | _ as e →  None

  let embed () = let p = mk () in inj p, proj p

end;;

Before delving into an explanation of the program, one can verify it satisfies the basic test.


# module Test_basic = Mk_universal_test(Universal_exn);;
# Test_basic.run ();;
- : unit = ()

The definition of the universal type t is an alias to the predefined type exn.


type t = exn

A module type ANY is introduced. Modules that implement this signature define an abstract type c and introduce an exn constructor E of c.


module type ANY = sig
  type c
  exception E of c
end

An alias for the type of a module value satisfying this signature comes next. Using aliases of this kind are helpful in reducing "syntactic verbosity" in code accepting and returning module values.


type α any = (module ANY with type c = α)

Next follow a set of functions that are private to the implementation of the module. The first of these is mk.


let mk : unit → α any =
  fun (type s) () →
    (module struct
      type c = s
      exception E of c
    end : ANY with type c = s)

This function mk takes the unit argument and each invocation computes a new module instance which is packed as a first class value and returned. The locally abstract type s connects to the α in the return type.

The next function to be defined is inj which computes a universal type value from its argument.


let inj (type s) (p : s any) (x : s) : t =
  let module Any = (val p : ANY with type c = s) in
  Any.E x

As was the case for mk, a locally abstract type is used in the definition of inj and observe how that type ensures a coherence between the module parameter p and the type of the parameter to be embedded x.

The projection function proj comes next and also uses a locally abstract type ensuring coherence among its parameters.


let proj (type s) (p : s any) (y : t) : s option =
  let module Any = (val p : ANY with type c = s) in
  match y with
  | Any.E x → Some x
  | _ as e →  None

The body of proj unpacks the module value parameter into a module named Any and then attempts to match y against the constructor defined by Any. Recall, at the end of the day, y is of type exn. The match contains two cases : the first matching the constructor Any.E x with x having type s, the second anything else (that is, proj is total).

Finally we come to the public function embed.


let embed () = let p = mk () in inj p, proj p

embed calls mk to produce a new embedding p and returns the pair of partial applications (inj p, proj p).

Our examination of the implementation is concluded. There are however various implications of the implementation we are now in a position to understand that are not immediately obvious from the specification. As a first example, consider the inferred type of a call to embed.


# module U = Universal_exn;;
# let inj, proj = U.embed ();;
val inj : '_a → U.t = 
val proj : U.t → '_a option =

Property 1 : embed produces weakly polymorphic functions.

Consider the following scenario:


# let of_int, to_int = U.embed ();;
# let of_string, to_string = U.embed ();;

At this point all of of_int, to_int, of_string and to_string are weakly polymorphic.


# let r : U.t ref = ref (of_int 13);;

The call to of_int fixes it's type to int → U.t and that of to_int to U.t → int option. The types of of_string and to_string remain unfixed.


# to_string !r = Some 13;;
- : bool = false

The programmer has mistakenly used a projection function from a different embedding.


# r := of_string "foo";;
Error: This expression has type string but an expression was expected of
type int

The erroneous usage of to_string incidentally fixed the type of of_string to int → U.t! One can imagine errors of this kind being difficult to diagnose.

Property 2 : Injection/projection functions of different embeddings may not be mixed.


# let ((of_int : int → U.t), (to_int : U.t → int option)) = U.embed ();;
# let ((of_int' : int → U.t), (to_int' : U.t → int option)) = U.embed ();;

Two embeddings have been created, the programmer has fixed the types of the injection/projection functions a priori (presumably as a defense against the problems of the preceding example).


# let r : U.t ref = ref (of_int 13);;
# to_int !r = Some 13;;
- : bool = true

The first embedding is used to inject an integer. That integer can be extracted using the projection function of that embedding.


#  to_int' !r = Some 13;;
- : bool = false

We cannot extract the integer from the universal type value using the projection function from the other embedding despite all of the types involved being the same. One can imagine that also as being quite the source of bugs and confusion to the unknowing programmer.

The constraint of property 2 is beyond the ability of the type system to enforce. About the best one can do is to enhance the specification of the type with documentation.


module type UNIV = sig
  type t

  val embed : unit → (α → t) * (t → α option)
 
  (* A pair [(inj, proj)] returned by [embed] works together in that
     [proj u] will return [Some v] if and only if [u] was created by
     [inj v]. If [u] was created by a different function then [proj u] 
     will return [None]. *)

end;;

Shayne Fletcher

2017-05-11T20:18:34-00:00

Polymorphic variants : subtyping and variance

Polymorphic variants : Subtyping and variance

Here are some expressions in the top-level involving polymorphic variant types.


    # let x = [ `On; `Off ];;
    val x : [> `Off | `On ] list = [ `On; `Off ]

The notation [> `Off | `On ] represents a type that at least contains the constructors `Off and `On. Of course, there are an unlimited number of such types so [> `Off | `On ] is a set in fact.


    # let n = `Number 1;;
    val n : [> `Number of int ] = `Number 1

The value n is of a type that at least contains the constructor `Number of int.


    # let f = function | `On → 1 | `Off → 0 | `Number n → n;;
    val f : [< `Number of int | `Off | `On ] → int = <fun>

The function f accomodates exactly three cases corresponding to the constructors `Off, `On and `Number of int. This informs us that [< `Number of int | `Off | `On ] is of a type that at most has the constructors `Off, `On and `Number of int.

The expression ($expr$ :> $typexpr$) coerces the expression $expr$ to type $typexpr$. The expression ($expr$ : $typ_{1}$ :> $typ_{2}$ ) coerces the exprssion $expr$ from $typ_{1}$ to $typ_{2}$. It is only possible to coerce an expression $expr$ from type $typ_{1}$ to type $typ_{2}$, if the type of $expr$ is an instance of $typ_{1}$ and $typ_{1}$ is a subtype of $typ_{2}$.


    # let f x = (x :> [ `A | `B ]);;
    val f : [< `A | `B ] → [ `A | `B ] = <fun>


    # let f x = (x :> [ `A | `B ] * [ `C | `D ]);;
    val f : [< `A | `B ] * [< `C | `D ] → [ `A | `B ] * [ `C | `D ] = <fun>

We see x needs to be coercible to [ `A | `B ] * [ `C | `D ]. This coercion can only proceed if x designates a pair with first component a subtype of [ `A | `B ] and second component a subtype of [ `C | `D ]. So we see, [< `A | `B ] * [< `C | `D ] is the set of subtypes of [ `A | `B ] * [ `C | `D ].


    # let f x = (x  :> [ `A ] → [ `C | `D ]);;
    val f : ([> `A ] → [< `C | `D ]) → [ `A ] → [ `C | `D ] = <fun>

We see x needs to be coercible to [ `A ] → [ `C | `D ]. This coercion can only proceed if x designates an arrow where the argument is of a type that at least contains the constructor `A. That is [ `A ], [ `A | ... ] where the "..." represent more constructors. From this we see that [> `A] is the set of supertypes of [ `A ]. The return value of x (by logic we've already established) is a subtype of [ `C | `D ]. So, [> `A ] → [< `C | `D ] is the set of subtypes of [ `A ] → [ `C | `D ].

The transformation represented by f above, has coerced an arrow x to a new arrow. The type of the argument of x is [> `A ] and the type of the argument of the resulting arrow is [ `A ]. That is, for the argument, the transformation between types has taken a supertype to a subtype. The argument type is said to be in a "contravariant" position. On the other hand, the result type of x is [< `C | `D ] and the arrow that is produced from it, f x, has result type [ `C | `D ]. That is, the coercion for the result type has taken a subtype to a supertype : the result type in x is said to be in "covariant" position.

In the following, the type α t is abstract.


    # type α t;;
    # let f (x : [ `A ] t) = (x :> [ `A | `B ] t);;
    Error: Type [ `A ] t is not a subtype of [ `A | `B ] t

Indeed, [ `A ] is a subtype of [ `A | `B ] but that, a priori, does not say anything about the subtyping relationship between [ `A ] t and [ `A | `B ] t. For this coercion to be legal, the parameter α must be given a covariant annotation:


    # type +α t;;
    # let f (x : [ `A ] t) = (x :> [ `A | `B ] t);;
    val f : [ `A ] t → [ `A | `B ] t = <fun>

The declaration type +α t declares that the abstract type is covariant in its parameter : if the type $\tau$ is a subtype of $\sigma$, then $\tau\;t$ is a subtype of $\sigma\;t$. Here, $\tau = $[ `A ], $\sigma = $[ `A | `B ], $\tau$ is a subtype of $\sigma$ and [ `A ] t is a subtype of [ `A | `B] t.

Here is a similar example, but this time, in the other direction.


    # type α t;;
    # let f (x : [ `A | `B ] t) = (x :> [ `A ] t);;
    Error: This expression cannot be coerced to type [ `A ] t

The type variable can be annotated as contravariant however, and the coercion function typechecks.


    # type -α t;;
    # let f (x : [`A | `B] t) = (x :> [ `A ] t);;
    val f : [ `A | `B ] t → [ `A ] t = <fun>

The declaration type -α t declares that the abstract type t is contravariant in its parameter : if $\tau$ is a subtype of $\sigma$ then $\sigma\;t$ is a subtype of $\tau\;t$. In this example, $\tau = $[ `A ] and $\sigma = $[`A | `B], $\tau$ is a subtype of $\sigma$ and [ `A | `B ] t is a subtype of [ `A ] t.

In the following, type α t is not abstract and variance can be inferred.


    # type α t = {x : α} ;;
    # let f x = (x : [`A] t :> [`A | `B] t);;
    val f : [ `A ] t → [ `A | `B ] t = <fun>

Introducing a constraint however inhibits variance inference.


    # type α t = {x : α} constraint α = [< `A | `B ];;
    # let f x = (x : [ `A ] t :> [ `A | `B ] t);;
    Error: Type [ `A ] t is not a subtype of [ `A | `B ] t

This situation can be overcome by introducing a covariance annotation.


    # type +α t = {x : α} constraint α = [< `A | `B ];;
    # let f x = (x : [ `A ] t :> [ `A | `B ] t);;
    val f : [ `A ] t → [ `A | `B ] t = <fun>

In the following example, α does not participate in the definition of type α t.


    # type α t = {x : int};;
    # let f x = (x : [ `A ] t :> [ `B ] t);;
    val f : [ `A ] t → [ `B ] t = <fun>

In this case, any conversion between δ t and ε t is legal : the types δ and ε are not required to have a subtyping relation between them.

OCaml Weekly News

2017-05-09T12:00:00-00:00

OCaml feed from KC Sivaramakrishnan

2017-05-07T11:34:10-00:00

One of the key indicators of maturity of a language ecosystem is the ease of building, managing and publishing software packages in that language. OCaml platform has made steady progress in the last few years to this end. While OPAM simplified package (and compiler) management, the developing and publishing packages remained a constant pain point. This situation has remarkably improved recently with the Topkg and Carcass. This post provides a short overview of my workflow for building and publishing an OCaml package using Topkg and Carcass.

Topkg is packager for distributing OCaml software. It provides an API for describing rules for package builds and installs. Topkg-care provides the command line tool topkg with support for creating and linting the distribution, publishing the distribution and its documentation on WWW, and making the package available through OPAM. Carcass is a library and a command line tool for defining and generating the directory structure for the OCaml package. At the time of writing this post, carcass was unreleased.

Workflow

I recently released a package for mergeable vectors based on operational transformation. The following describes my workflow to build and publish the package.

Setup

Install topkg-care and carcass:

$ opam install topkg-care opam-publish
$ opam pin add -kgit carcass https://github.com/dbuenzli/carcass

Develop

Create the directory structure

  $ carcass body topkg/pkg mergeable_vector

Init

  $ cd mergeable_vector && git init && git add . && git commit -m "First commit."
  $ git remote add origin https://github.com/kayceesrk/mergeable-vector
  $ git push --set-upstream origin master

Develop: The mergeable_vector/src directory has the source files. I use this Makefile at the root of the package.
Test the package locally with OPAM
```
  $ opam pin add mergeable_vector .
```

Publish

Update the CHANGES file for the new release.
Tag the release
```
  $ topkg tag 0.1.0
```
Build the distribution
```
  $ topkg distrib
```
Publish the distribution
```
  $ topkg publish distrib
```
This makes a new release on Github.
Publish the doc
```
  $ topkg publish doc
```
This publishes the documentation on Github.
Make an OPAM package info and submit it to OPAM repository at opam.ocaml.org.
```
  $ topkg opam pkg
  $ topkg opam submit
```
This creates a Github PR to the opam-repository. Once the PR is merged, the package becomes available to the users.

OCaml – Jane Street Tech Blogs

2017-05-11T19:02:45-00:00

Are you thinking about applying to Jane Street for a developer role? Or already have a phone interview scheduled but unsure what to expect? Read on as we walk through an example phone interview with you.

We want to give you some insight into what a typical Jane Street phone interview looks like and give you a chance to prepare. We're going to take a look at a question we call "Memo" which we used to ask regularly (but of course don't ask anymore so no need to memorize anything on this page!). As such this post is meant to be a specific case analysis. If you haven't yet seen it, we recommend reading this blog post for a general overview what we are looking for in candidates.

Getting started

To allow us to work on the question together, we'll use a shared online editor. We'll provide you with the link to use (either before hand or during the interview).

We expect you to write code in a real programming language in the interview, not pseudo-code. You can use any programming language you'd like, but we strongly recommend you use the one you're most familiar with and will let you solve the problem in the best way (it's fine to change your mind as you're exploring the problem!). You should be comfortable with basic data structures and APIs in the language of your choice.

Note, there are no bonus points for using a functional language like OCaml. Please don't use OCaml just because you think it will make us happy. We want to see you at your best, so you should use the language you're most comfortable with. If OCaml isn't your strongest language, then don't use it.

If during the interview you realize you have seen a question before, then please let us know and we can do a different question. As you'll see in this post, knowing a question in advance greatly reduces what we can learn about you!

Part 1 - Basic coding

Have you heard about memoization? Can you carefully describe what it is? If you haven't heard about it, don't worry. We'll bring you up to speed. (A good introduction is on Wikipedia.)

Now let's say there is a function f of type int -> int whose output only depends on the input. f is very expensive to compute. We'd like you to write a memoized version of this function, i.e. another function g of the same type, that returns the same values – g(x) = f(x) for all x – but only does the expensive computation once for each input value.

A typical first solution we're looking for at this stage uses a hash-table to store calculated results. A possible solution in OCaml might be:

let memo f =
  let results = Hashtbl.create 256 in
  (fun input ->
     match Hashtbl.find results input with
     | None ->
        let result = f input in
        Hashtbl.add results ~key:input ~data:result;
        result
     | Some result -> result)

(As I said above, you're not required or expected to write in OCaml but in this blog post I'm going to follow my own advice and use the language I'm most familiar with, which is OCaml. You might also object that this does a test-and-set without a lock, so can't possibly be thread-safe. Nice spot! For the purpose of this question let's ignore re-entry to focus on the core ideas.)

Whichever APIs or data structures you end up using for your solution: you should be prepared to talk about how they work and what the complexity of various operations is.

Part 2 - Reasoning about and improving your code

Can you think of any issues we could run into when using the function from part 1? For example, let's say we run your function in production and notice our application performs significantly worse than before. Quite the opposite from what we hoped memoization would do! Can you see what the problem might be?

The big problem is memory usage. Our application might call f with lots of different inputs and each result will be stored in the hashtable forever – a memory leak! Can you come up with some ideas to improve upon this?

A reasonable approach to control the memory usage is to bound the size of the hash-table and to evict things from it in a FIFO fashion. What trade-offs does FIFO have versus other eviction schemes? How could you modify your memo function to implement FIFO? Let's aim for O(1) complexity when evicting from the cache.

There are a few different ways to do this. A good solution keeps a separate queue: when adding a new result to your hashtable, if the size grows beyond the bound, then dequeue from the queue and remove that element from the hashtable.

Besides being able to write code to do this, we look for how you communicate your thoughts on the problem and ideas to improve it. We don't necessarily expect every candidate to immediately jump to the O(1) solution, but we're interested in the process of talking through this problem and what you can come up with.

Part 3 - Going further

As you probably realize, FIFO can be very inefficient in some use-cases. Let's say we want to do LRU (least recently used) instead. We still want the implementation to be as efficient as possible. How can you implement this?

Once more, there are multiple ways to do this. The easiest solution stores timestamps with each value in the hashtable and linearly scans through the table when evicting. This is O(n). It's possible to improve to O(log n) using a min-heap or even to O(1) using a doubly-linked list.

Side-note: implementing the most efficient solution One way to get to O(1) is to maintain a doubly linked list and make the values in the hash-table point to elements in that list. Then when looking up a cached value in the hash-table, you can slice it out of its current position in the list and put it at the top in O(1). You maintain a separate pointer to the bottom of the doubly linked list to evict in O(1).

Few candidates come up with the doubly-linked list immediately, so don't worry if this is not something you thought of straight away. While we might ask you to implement parts of your solution, this part is intended as a discussion and test your ideas for improving complexity. We'll guide you to the right level of abstraction, so you don't have to worry too much about which details to include.

We also have various other extensions for this question that make it possible to keep going further as far as time allows.

What we look for

Again, for a good overview see this post.

While interviewing, we try to evaluate how well you would fit in our work environment by collaboratively solving a problem. This means the journey through the interview is a lot more important than the snapshot of the solution at the end of it. We are more interested in seeing how you approach a difficult problem than just capturing a boolean flag whether you can come up with the solution.

As a concrete example, we might prefer a candidate that wrote careful and clear code, communicated well and had good insights and ideas along the way, but didn't get as far through the problem compared to another candidate that immediately solved every part but was sloppy and hard to follow.

This makes it impossible to give a rigid set of requirements of what needs to be achieved during the interview. Nevertheless, to give you some rough guidelines:

Every successful candidate should achieve a bug-free solution for part 1 relatively quickly. If some of part 1 sounds unfamiliar to you, it might be better to hold off applying to give yourself more time to prepare.

Most candidates that we pass also complete part 2 fully in the time of the interview. Strong candidates finish part 3 with a complete solution, but not finishing this part doesn't mean that you will be rejected! As said above, it's what happens during the interview that really counts, not the result at the end.

Sharing questions afterwards

We are interested in seeing you approach a difficult problem and walk through the process of deriving a solution with you. If you already know the question in advance (either by finding it online, or by talking to friends who have previously applied), it reduces the efficacy of the interview: We now only get to test if you can write code for a problem you've already understood and solved, which is a small subset of things we are interested in learning about. For example, if you've read through this post, we wouldn't learn much from asking you Memo!

Thus we would like to ask you not to share the interview question with anybody else (in person or online).

What happens next

We usually try to get back to you about the outcome of the interview within one week. During busy periods this might take a bit longer, but not hearing from us after 10 days is unexpected and you should definitely poke us.

If the interview went well, we invite you to come to one of our offices (New York, London or Hong Kong) for a day of onsite interviews. These proceed in much the same way as the phone interview – all questions are technical.

Ready to apply?

If you read this post and feel ready to apply, then simply go here. We have a very straightforward application process – all we need is your resume or CV. We look forward to interviewing you!

OCaml Weekly News

2017-05-09T12:00:00-00:00

Erratique

2017-02-27T19:15:55-00:00

Ten years ago I published a first bit of free software. Over the years, this has now expanded to 28 OCaml packages, without counting random informally published elucubrations. Thanks to OCaml Labs' support, there will be more.

the OCaml Labs wiki - Blog:News [en]

2017-02-23T13:57:42-00:00

We are pleased to announce that the new and improved OCaml Labs website is here!

This wiki will remain active while we transition our content to ocamllabs.io, but it will be retired eventually. The new site will have all the recent news and exciting developments from OCaml Labs together with links to related projects and people, so it will be easier than ever to keep up to date with everything we are doing.

We hope you enjoy the new site!

full stack engineer

2017-03-12T13:27:37-00:00

Less than two years after the initial proposal, we're happy to present conex 0.9.2. Pleas note that this is still work in progress, to be deployed with opam 2.0 and the opam repository.

Conex is a library to verify and attest release integrity and authenticity of a community repository through the use of cryptographic signatures.

Packages are collected in a community repository to provide an index and allowing cross-references. Authors submit their packages to the repository. which is curated by a team of janitors. Information about a package stored in a repository includes: license, author, releases, their dependencies, build instructions, url, tarball checksum. When someone publishes a new package, the janitors integrate it into the repository, if it compiles and passes some validity checks. For example, its name must not be misleading, nor may it be too general.

Janitors keep an eye on the repository and fix emergent failures. A new compiler release, or a release of a package on which other packages depend, might break the compilation of a package. Janitors usually fix these problems by adding a patch to the build script, or introducing a version constraint in the repository.

Conex ensures that every release of each package has been approved by its author or a quorum of janitors. A conex-aware client initially verifies the repository using janitor key fingerprints as anchor. Afterwards, the on-disk repository is trusted, and every update is verified (as a patch) individually. This incremental verification is accomplished by ensuring all resources that the patch modifies result in a valid repository with sufficient approvals. Additionally, monotonicity is preserved by embedding counters in each resource, and enforcing a counter increment after modification. This mechanism avoids rollback attacks, when an attacker presents you an old version of the repository.

A timestamping service (NYI) will periodically approve a global view of the verified repository, together with a timestamp. This is then used by the client to prevent mix-and-match attacks, where an attacker mixes some old packages and some new ones. Also, the client is able to detect freeze attacks, since at least every day there should be a new signature done by the timestamping service.

The trust is rooted in digital signatures by package authors. The server which hosts the repository does not need to be trusted. Neither does the host serving release tarballs.

If a single janitor would be powerful enough to approve a key for any author, compromising one janitor would be sufficient to enroll any new identities, modify dependencies, build scripts, etc. In conex, a quorum of janitors (let's say 3) have to approve such changes. This is different from current workflows, where a single janitor with access to the repository can merge fixes.

Conex adds metadata, in form of resources, to the repository to ensure integrity and authenticity. There are different kinds of resources:

Authors, consisting of a unique identifier, public key(s), accounts.
Teams, sharing the same namespace as authors, containing a set of members.
Authorisation, one for each package, describing which identities are authorised for the package.
Package index, for each package, listing all releases.
Release, for each release, listing checksums of all data files.

Modifications to identities and authorisations need to be approved by a quorum of janitors, package index and release files can be modified either by an authorised id or by a quorum of janitors.

Documentation

API documentation is available online, also a coverage report.

We presented an abstract at OCaml 2016 about an earlier design.

Another article on an earlier design (from 2015) is also available.

Conex is inspired by the update framework, especially on their CCS 2010 paper, and adapted to the opam repository.

The TUF spec has a good overview of attacks and threat model, both of which are shared by conex.

What's missing

See issue 7 for a laundry list
Timestamping service
Key revocation and rollover
Tool to approve a PR (for janitors)
Camelus like opam-repository check bot
Integration into release management systems

Getting started

At the moment, our opam repository does not include any metadata needed for signing. We're in a bootstrap phase: we need you to generate a keypair, claim your packages, and approve your releases.

We cannot verify the main opam repository yet, but opam2 has support for a repository validation command, builtin, which should then call out to conex_verify (there is a --nostrict flag for the impatient). There is also an example repository which uses the opam validation command.

To reduce the manual work, we analysed 7000 PRs of the opam repository within the last 4.5 years (more details here. This resulted in an educated guess who are the people modifying each package, which we use as a basis whom to authorise for which packages. Please check with conex_author status below whether your team membership and authorised packages were inferred correctly.

Each individual author - you - need to generate their private key, submit their public key and starts approving releases (and old ones after careful checking that the build script, patches, and tarball checksum are valid). Each resource can be approved in multiple versions at the same time.

Installation

TODO: remove clone once PR 8494 is merged.

$ git clone -b auth https://github.com/hannesm/opam-repository.git repo
$ opam install conex
$ cd repo

This will install conex, namely command line utilities, conex_author and conex_verify_nocrypto/conex_verify_openssl. All files read and written by conex are in the usual opam file format. This means can always manually modify them (but be careful, modifications need to increment counters, add checksums, and be signed). Conex does not deal with git, you have to manually git add files and open pull requests.

Author enrollment

For the opam repository, we will use GitHub ids as conex ids. Thus, your conex id and your GitHub id should match up.

repo$ conex_author init --repo ~/repo --id hannesm
Created keypair hannesm.  Join teams, claim your packages, sign your approved resources and open a PR :)

This attempts to parse ~/repo/id/hannesm, errors if it is a team or an author with a publickey. Otherwise it generates a keypair, writes the private part as home.hannes.repo.hannesm.private (the absolute path separated by dots, followed by your id, and private - if you move your repository, rename your private key) into ~/.conex/, the checksums of the public part and your accounts into ~/repo/id/hannesm. See conex_author help init for more options (esp. additional verbosity -v can be helpful).

repo$ git status -s
 M id/hannesm

repo$ git diff //abbreviated output
-  ["counter" 0x0]
+  ["counter" 0x1]

-  ["resources" []]
+  [
+    "resources"
+    [
+      [
+        ["typ" "key"]
+        ["name" "hannesm"]
+        ["index" 0x1]
+        ["digest" ["SHA256" "ht9ztjjDwWwD/id6LSVi7nKqVyCHQuQu9ORpr8Zo2aY="]]
+      ]
+      [
+        ["typ" "account"]
+        ["name" "hannesm"]
+        ["index" 0x2]
+        ["digest" ["SHA256" "aCsktJ5M9PI6T+m1NIQtuIFYILFkqoHKwBxwvuzpuzg="]]
+      ]
+
+keys: [
+  [
+    [
+      "RSA"
+      """
+-----BEGIN PUBLIC KEY-----
+MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAyUhArwt4XcxLanARyH9S
...
+9KQdg6QnLsQh/j74QKLOZacCAwEAAQ==
+-----END PUBLIC KEY-----"""
+      0x58A3419F
+    ]
+    [
+      0x58A79A1D
+      "RSA-PSS-SHA256"
+      "HqqicsDx4hG9pFM5E7"
+    ]
+  ]
+]

Status

If you have a single identity and contribute to a single signed opam repository, you don't need to specify --id or --repo from now on.

The status subcommand presents an author-specific view on the repository. It lists the own public keys, team membership, queued resources, and authorised packages.

The opam repository is in a transitionary state, we explicitly pass --quorum 0, which means that every checksum is valid (approved by a quorum of 0 janitors).

repo$ conex_author status --quorum 0 arp
author hannesm #1 (created 0) verified 3 resources, 0 queued
4096 bit RSA key created 1487094175 approved, SHA256: ht9ztjjDwWwD/id6LSVi7nKqVyCHQuQu9ORpr8Zo2aY=
account GitHub hannesm approved
account email hannes@mehnert.org approved
package arp authorisation approved
conex_author: [ERROR] package index arp was not found in repository

This shows your key material and accounts, team membership and packages you are authorised to modify (inferred as described here.

The --noteam argument limits the package list to only these you are personally authorised for. The --id argument presents you with a view of another author, or from a team perspective. The positional argument is a prefix matching on package names (leave empty for all).

Resource approval

Each resource needs to be approved individually. Each author has a local queue for to-be-signed resources, which is extended with authorisation, init, key, release, and team (all have a --dry-run flag). The queue can be dropped using conex_author reset. Below shown is conex_author sign, which let's you interactively approve queued resources and cryptopgraphically signs your approved resources afterwards.

The output of conex_author status listed an authorisation for conf-gsl, which I don't feel responsible for. Let's drop my privileges:

repo$ conex_author authorisation conf-gsl --remove -m hannesm
modified authorisation and added resource to your queue.

I checked my arp release careful (checksums of tarballs are correct, opam files do not execute arbitrary shell code, etc.), and approve this package and its single release:

repo$ conex_author release arp
conex_author.native: [WARNING] package index arp was not found in repository
conex_author.native: [WARNING] release arp.0.1.1 was not found in repository
wrote release and added resources to your queue.

Once finished with joining and leaving teams (using the team subcommand), claiming packages (using the authorisation subcommand), and approve releases (using the release subcommand), you have to cryprographically sign your queued resource modifications:

repo$ conex_author sign
release arp.0.1.1 #1 (created 1487269425)
[descr: SHA256: aCsNvcj3cBKO0GESWG4r3AzoUEnI0pHGSyEDYNPouoE=;
opam: SHA256: nqy6lD1UP+kXj3+oPXLt2VMUIENEuHMVlVaG2V4z3p0=;
url: SHA256: FaUPievda6cEMjNkWdi0kGVK7t6EpWGfQ4q2NTSTcy0=]
approved (yes/No)?
package arp #1 (created 1487269425) [arp.0.1.1]
approved (yes/No)?y
authorisation conf-gsl #1 (created 0) empty
approved (yes/No)?y
wrote hannesm to disk

repo$ conex_author status --quorum 0 arp
author hannesm #1 (created 0) verified 7 resources, 0 queued
4096 bit RSA key created 1487094175 approved, SHA256: ht9ztjjDwWwD/id6LSVi7nKqVyCHQuQu9ORpr8Zo2aY=
account GitHub hannesm approved
account email hannes@mehnert.org approved
package arp authorisation approved package index approved
release arp.0.1.1: approved

If you now modify anything in packages/arp (add subdirectories, modify opam, etc.), this will not be automatically approved (see below for how to do this).

You manually need to git add some created files.

repo$ git status -s
 M id/hannesm
 M packages/conf-gsl/authorisation
?? packages/arp/arp.0.1.1/release
?? packages/arp/package

repo$ git add packages/arp/arp.0.1.1/release packages/arp/package
repo$ git commit -m "hannesm key enrollment and some fixes" id packages

Now push this to your fork, and open a PR on opam-repository!

Editing a package

If you need to modify a released package, you modify the opam file (as before, e.g. introducing a conflict with a dependency), and then approve the modifications. After your local modifications, conex_author status will complain:

repo$ conex_author status arp --quorum 0
package arp authorisation approved package index approved
release arp.0.1.1: checksums for arp.0.1.1 differ, missing on disk: empty, missing in checksums file: empty, checksums differ: [have opam: SHA256: QSGUU9HdPOrwoRs6XJka4cZpd8h+8NN1Auu5IMN8ew4= want opam: SHA256: nqy6lD1UP+kXj3+oPXLt2VMUIENEuHMVlVaG2V4z3p0=]

repo$ conex_author release arp.0.1.1
released and added resources to your resource list.

repo$ conex_author sign
release arp.0.1.1 #1 (created 1487269943)
[descr: SHA256: aCsNvcj3cBKO0GESWG4r3AzoUEnI0pHGSyEDYNPouoE=;
opam: SHA256: QSGUU9HdPOrwoRs6XJka4cZpd8h+8NN1Auu5IMN8ew4=;
url: SHA256: FaUPievda6cEMjNkWdi0kGVK7t6EpWGfQ4q2NTSTcy0=]
approved (yes/No)? y
wrote hannesm to disk

The release subcommand recomputed the checksums, incremented the counter, and added it to your queue. The sign command signed the approved resource.

repo$ git status -s
 M id/hannesm
 M packages/arp/arp.0.1.1/opam
 M packages/arp/arp.0.1.1/package

repo$ git commit -m "fixed broken arp package" id packages

Janitor tools

Janitors need to approve teams, keys, accounts, and authorisations.

To approve resources which are already in the repository on disk, the key subcommand queues approval of keys and accounts of the provided author:

repo$ conex_author key avsm
added keys and accounts to your resource list.

The authorisation subcommand, and team subcommand behave similarly for authorisations and teams.

Bulk operations are supported as well:

conex_author authorisation all

This will approve all authorisations of the repository which are not yet approved by you. Similar for the key and team subcommands, which also accept all.

Don't forget to conex_author sign afterwards (or yes | conex_author sign).

Verification

The two command line utlities, conex_verify_openssl and conex_verify_nocrypto contain the same logic and same command line arguments.

For bootstrapping purposes (nocrypto is an opam package with dependencies), conex_verify_openssl relies on the openssl command line tool (version 1.0.0 and above) for digest computation and verification of the RSA-PSS signature.

The goal is to use the opam2 provided hooks, but before we have signatures we cannot enable them.

See the example repository for initial verification experiments, and opam2 integration.

I'm interested in feedback, please open an issue on the conex repository. This article itself is stored as Markdown in a different repository.

full stack engineer

2017-03-12T13:27:37-00:00

A very important data point for conex, the new opam signing utility, is who is authorised for a given package. We could have written this manually down, or force each author to create a pull request for their packages, but this would be a long process and not easy: the main opam repository has around 1500 unique packages, and 350 contributors. Fortunately, it is a git repository with 5 years of history, and over 6900 pull requests. Each opam file may also contain a maintainers entry, a list of strings (usually a mail address).

The data sources we correlate are the maintainers entry in opam file, and who actually committed in the opam repository. This is inspired by some GitHub discussion.

GitHub id and email address

For simplicity, since conex uses any (unique) identifier for authors, and the opam repository is hosted on GitHub, we use a GitHub id as author identifier. Maintainer information is an email address, thus we need a mapping between them.

We wrote a shell script to find all PR merges, their GitHub id (in a brittle way: using the name of the git remote), and email address of the last commit. It also saves a diff of the PR for later. This results in 6922 PRs (opam repository version 38d908dcbc58d07467fbc00698083fa4cbd94f9d).

The metadata output is processed by github_mail: we ignore PRs from GitHub organisations PR.ignore_github, where commits PR.ignore_pr are picked from a different author (manually), bad mail addresses, and Jeremy's mail address (it is added to too many GitHub ids otherwise). The goal is to have a for an email address a single GitHub id. 329 authors with 416 mail addresses are mapped.

Maintainer in opam

As mentioned, lots of packages contain a maintainers entry. In maintainers we extract the mail addresses of the most recently released opam file. Some hardcoded matches are teams which do not properly maintain the maintainers field (such as mirage and xapi-project ;). We're open for suggestions to extend this massaging to the needs. Additionally, the contact at ocamlpro mail address was used for all packages before the maintainers entry was introduced (based on a discussion with Louis Gesbert). 132 packages with empty maintainers.

Fitness

Combining these two data sources, we hoped to find a strict small set of whom to authorise for which package. Turns out some people use different mail addresses for git commits and opam maintainer entries, which are be easily fixed.

While processing the full diffs of each PR (using the diff parser of conex mentioned above), ignoring the 44% done by janitors (a manually created set by looking at log data, please report if wrong), we categorise the modifications: authorised modification (the GitHub id is authorised for the package), modification by an author to a team-owned package (propose to add this author to the team), modification of a package where no GitHub id is authorised, and unauthorised modification. We also ignore packages which are no longer in the opam repository.

2766 modifications were authorised, 418 were team-owned, 452 were to packages with no maintainer, and 570 unauthorised. This results in 125 unowned packages.

Out of the 452 modifications to packages with no maintainer, 75 are a global one-to-one author to package relation, and are directly authorised.

Inference of team members is an overapproximation (everybody who committed changes to their packages), additionally the janitors are missing. We will have to fill these manually.

alt-ergo -> OCamlPro-Iguernlala UnixJunkie backtracking bobot nobrowser
janestreet -> backtracking hannesm j0sh rgrinberg smondet
mirage -> MagnusS dbuenzli djs55 hannesm hnrgrgr jonludlam mato mor1 pgj pqwy pw374 rdicosmo rgrinberg ruhatch sg2342 talex5 yomimono
ocsigen -> balat benozol dbuenzli hhugo hnrgrgr jpdeplaix mfp pveber scjung slegrand45 smondet vasilisp
xapi-project -> dbuenzli djs55 euanh mcclurmc rdicosmo simonjbeaumont yomimono

Alternative approach: GitHub urls

An alternative approach (attempted earlier) working only for GitHub hosted projects, is to authorise the use of the user part of the GitHub repository URL. Results after filtering GitHub organisations are not yet satisfactory (but only 56 packages with no maintainer, output repo. This approach completely ignores the manually written maintainer field.

Conclusion

Manually maintained metadata is easily out of date, and not very useful. But combining automatically created metadata with manually, and some manual tweaking leads to reasonable data.

The resulting authorised inference is available in this branch.

OCaml – Jane Street Tech Blogs

2017-05-11T19:02:45-00:00

Our first Jane Street Tech Talk went really well! Thanks to everyone who came and made it a fun event.

Now it's time for another. We're planning for the series to feature a combination of voices from inside and outside Jane Street. This one is of the latter variety: on March 6th, Arjun Guha will be presenting On Verification for System Configuration Languages, which is about using static verification techniques for catching bugs in Puppet configs.

I've known Arjun for years, and he's a both a good hacker and a serious academic with a real knack for finding good ways of applying ideas from programming languages to systems problems. Also, he has excellent taste in programming languages...

I hope you'll come! You can sign up here.

OCaml feed from KC Sivaramakrishnan

2017-05-07T11:34:10-00:00

Ezirmin is an easy interface over the Irmin, a library database for building persistent mergeable data structures based on the principles of Git. In this post, I will primarily discuss the Ezirmin library, but also discuss some of the finer technical details of mergeable data types implemented over Irmin.

Contents
Irmin and Ezirmin
Quick tour of Ezirmin
Mergeable persistent data types
Next steps

Irmin and Ezirmin

Irmin is a library for manipulating persistent mergeable data structures (including CRDTs) that follows the same principles of Git. In particular, it has built-in support for snapshots, branches and reverts, and can compile to multiple backends. Being written in pure OCaml, apps written using Irmin, as well as running natively, can run in the browsers or be compiled to Unikernels. A good introduction to the capabilities of Irmin can be found in the Irmin README file.

One of the downsides to being extremely configurable is that the Irmin library is not beginner friendly. In particular, the library tends to be rather functor heavy, and even simple uses require multiple functor instantiations¹. The primary goal of Ezirmin is to provide a defuntorized interface to Irmin, specialized to useful defaults. However, as I’ve continued to build Ezirmin, it has come to include a collection of useful mergeable data types including counters, queues, ropes, logs, etc. I will spend some time describing some of the interesting aspects of these data structures.

Quick tour of Ezirmin

You can install the latest version of Ezirmin by

$ git clone https://github.com/kayceesrk/ezirmin
$ cd ezirmin
$ opam pin add ezirmin .

Stable versions are also available through OPAM:

$ opam install ezirmin

Let’s fire up utop and get started:

$ utop
utop # #require "ezirmin";;
utop # open Lwt.Infix;;

We’ll create a mergeable queue of strings using the Git file system backend rooted at /tmp/ezirminq:

utop # module M = Ezirmin.FS_queue(Tc.String);; (* Mergeable queue of strings *)
utop # open M;;
utop # let m = Lwt_main.run (init ~root:"/tmp/ezirminq" ~bare:true () >>= master);;
val m : branch = <abstr>

m is the master branch of the repository. Ezirmin exposes a key value store, where keys are hierarchical paths and values are whatever data types is stored in the repo. In this case, the data type is a queue. Let’s push some elements into the queue:

utop # push m ["home"; "todo"] "buy milk";;
- : unit = ()
utop # push m ["work"; "todo"] "publish ezirmin";;
- : unit = ()
utop # to_list m ["home"; "todo"];;
- : string list = ["buy milk"]

The updates to the queue is saved in the Git repository at /tmp/ezirminq. In another terminal,

$ utop
utop # #require "ezirmin";;
utop # module M = Ezirmin.FS_queue(Tc.String);; (* Mergeable queue of strings *)
utop # open M;;
utop # open Lwt.Infix;;
utop # let m = Lwt_main.run (init ~root:"/tmp/ezirminq" ~bare:true () >>= master);;
val m : branch = <abstr>
utop # pop m ["home"; "todo"];;
- : string option = Some "buy milk"

For concurrency control, use branches. In the first terminal,

utop # let wip = Lwt_main.run @@ clone_force m "wip";;
utop # push wip ["home"; "todo"] "walk dog";;
- : unit = ()
utop # push wip ["home"; "todo"] "take out trash";;
- : unit = ()

The changes are not visible until the branches are merged.

utop # to_list m ["home"; "todo"];;
- : string list = []
utop # merge wip ~into:m;;
- : unit = ()
utop # to_list m ["home"; "todo"];;
- : string list = ["walk dog"; "take out trash"]

Merge semantics

What should be the semantics of popping the queue at home/todo concurrently at the master branch and wip branch? It is reasonable to ascribe exactly once semantics to pop such that popping the same element on both branches and subsequently merging the queues would lead to a merge conflict. However, a more useful semantics is where we relax this invariant and allow elements to be popped more than once on different branches. In particular, the merge operation on the queue ensures that:

An element popped in one of the branches is not present after the merge.
Merges respect the program order in each of the branches.
Merges converge.

Hence, our merge queues are CRDTs.

Working with history

Irmin is fully compatible with Git. Hence, we can explore the history of the operations using the git command line. In another terminal:

$ cd /tmp/ezirminq
$ git lg
* e75da48 - (4 minutes ago) push - Irmin xxxx.cam.ac.uk.[73126] (HEAD -> master, wip)
* 40ed32d - (4 minutes ago) push - Irmin xxxx.cam.ac.uk.[73126]
* 6a56fb0 - (5 minutes ago) pop - Irmin xxxx.cam.ac.uk.[73221]
* 6a2cc9a - (6 minutes ago) push - Irmin xxxx.cam.ac.uk.[73126]
* 55f7fc8 - (6 minutes ago) push - Irmin xxxx.cam.ac.uk.[73126]

The Git log shows that there have been 4 pushes and 1 pop in this repository. In addition to the data structures being mergeable, they are also persistent. In particular, every object stored in Irmin has complete provenance. You can also manipulate history using the Git command line.

$ git reset HEAD~2 --hard
$ git lg
* e75da48 - (8 minutes ago) push - Irmin xxxx.cam.ac.uk.[73126] (wip)
* 40ed32d - (9 minutes ago) push - Irmin xxxx.cam.ac.uk.[73126]
* 6a56fb0 - (9 minutes ago) pop - Irmin xxxx.cam.ac.uk.[73221] (HEAD -> master)
* 6a2cc9a - (10 minutes ago) push - Irmin xxxx.cam.ac.uk.[73126]
* 55f7fc8 - (10 minutes ago) push - Irmin xxxx.cam.ac.uk.[73126]

Back in the first terminal:

utop # to_list m ["home"; "todo"];;
- : string list = []

Since we rolled back the master to before the pushes were merged, we see an empty list. Ezirmin also provides APIs for working with history programmatically.

utop # let run = Lwt_main.run;;
utop # let repo = run @@ init ();;
utop # let path = ["Books"; "Ovine Supply Logistics"];;
utop # let push_msg = push m ~path;;

utop # begin
  push_msg "Baa" >>= fun () ->
  push_msg "Baa" >>= fun () ->
  push_msg "Black" >>= fun () ->
  push_msg "Camel"
end;;

utop # to_list m path;;
- : string list = ["Baa"; "Baa"; "Black"; "Camel"]

Clearly this is wrong. Let’s fix this by reverting to earlier version:

utop # let m_1::_ = run @@ predecessors repo m;; (** HEAD~1 version *)
utop # to_list m_1 path;;
- : string list = ["Baa"; "Baa"; "Black"]
utop # update_branch m ~set:m_1;;
utop # to_list m path;;
- : string list = ["Baa"; "Baa"; "Black"]

Now that we’ve undone the error, we can do the right thing.

utop # push_msg "Sheep";;
utop # run @@ to_list m path;;
- : string list = ["Baa"; "Baa"; "Black"; "Sheep"]

Reacting to changes

Ezirmin supports watching a particular key for updates and invoking a callback function when there is one.

utop # let cb _ = Lwt.return (print_endline "callback: update to home/todo");;
utop # watch m ["home"; "todo"] cb

The code above installs a listener cb on the queue at home/todo, which is run every time the queue is updated. This includes local push and pop operations as well as updates due to merges.

utop # push m ["home"; "todo"] "hang pictures";;
callback: update to home/todo
- : unit = ()

Interaction with remotes

Unlike distributed data stores, where the updates are disseminated transparently between the replicas, Ezirmin provides you the necessary building blocks for building your own dissemination protocol. As with Git, Ezirmin exposes the functionality to push² and pull changes from remotes.

#show_module Sync;;
module Sync : sig
  type remote
  val remote_uri : string -> remote
  val pull : remote -> branch -> [ `Merge | `Update ] -> [ `Conflict of string | `Error | `No_head | `Ok ] Lwt.t
  val push : remote -> branch -> [ `Error | `Ok ] Lwt.t
end

This design provides the flexibility to describe your own network layout, with anti-entropy mechanisms built-in to the synchronization protocol. For example, one might deploy the replicas in a hub-and-spoke model where each replica accepts client writes locally and periodically publishes changes to the master and also fetches any latest updates. The data structures provided by Ezirmin are always mergeable and converge. Hence, the updates are never rejected. It is important to note that even though we have a centralized master, this deployment is still highly available. Even if the master is unavailable, the other nodes can still accept client requests. The replicas may also be connected in a peer-to-peer fashion without a centralized master for a more resilient deployment.

Mergeable persistent data types

Ezirmin is equipped with a growing collection of mergeable data types. The mergeable datatypes occupy a unique position in the space of CRDTs. Given that we have the history, the design of mergeable datatypes is much simpler. Additionally, this also leads to richer structures typically not found in CRDTs. It is worth studying them in detail.

Irmin Architecture

Irmin provides a high-level key-value interface built over two lower level heaps: a block store and a tag store. A block store is an append-only content-addressable store that stores serialized values of application contents, prefix-tree nodes, history meta-data, etc. Instead of using physical memory address of blocks, the blocks are identified by the hash of their contents. As a result block store enjoys very nice properties. Being content-addressed, we get sharing for free: two blocks with the same content will have the have the same hash. This not only applies for individual blocks, but also for linked-structures. For example,

The linked list above is uniquely identified by hash h0 since h0 was computed from the content a and the hash of the tail of the list h1. No other list has hash h0. Changing c to C in this list would result in a different hash for the head of the list³. Moreover, since the block store is append-only, all previous versions of a application-level data structure is also available, and thus providing persistence. This also makes for a nice concurrency story for multiple processes/threads operating on the block store. The absence of mutations on block store mean that no additional concurrency control mechanisms are necessary.

The only mutable part of the Irmin architecture is the tag store, that maps global names to blocks in the block store. The notion of branches are built on top of the tag store. Cloning a branch creates a new tag that points to the same block as the cloned branch.

User-defined merges

The real power of Irmin is due to the user-defined merges. Irmin expects the developer to provide a 3-way merge function with the following signature:

type t
(** User-defined contents. *)

val merge : old:t -> t -> t -> [`Ok of t | `Conflict of string]
(** 3-way merge. *)

Given the common ancestor old and the two versions, merge function can either return a successful merge or mark a conflict. It is up to the developer to ensure that merges are commutative (merge old a b = merge old b a) and that the merge captures the intent of the two branches. If the merge function never conflicts, we have CRDTs.

Mergeable Counters

The simplest mergeable data type is a counter with an increment and decrement operations. Given that we have a 3-way merge function, the merge is intuitive:

Given the two new values for the counter t1 and t2, and their lowest common ancestor value old, the new value of the counter is the sum of the old value and the two deltas: old + (t1 - old) + (t2 - old) = t1 + t2 - old.

Theory of merges

While this definition is intuitive, the proof of why this strategy (i.e., computing deltas and applying to the common ancestor) is correct is quite subtle. It happens to be the case that the patches (deltas) in this case, integers under addition, form an abelian group. Judah Jacobson formalizes patches for Darcs as inverse semigroups and proves convergence. Every abelian group is also an inverse semigroup. Hence, the above strategy is correct. Merges can also be equivalently viewed as a pushout in category theory, leading to the same result. I will have to save the discussion of the category theoretic reasoning of Irmin merges for another time. But Liam O’Connor has written a concise post on the theory of patches which is worth a read.

Recursive merges

Since Ezirmin allows arbitrary branching and merging, the lowest common ancestor need not be unique. One way to end up with multiple lowest common ancestors is criss-cross merges. For example, consider the history graph below:

The counter at some key in the master was initially 0. The branch wip was cloned at this point. The counter is incremented by 1 at master and 2 at wip. At this point, both branches are merged into the other branch. The common ancestor here is the initial state of counter 0. This results in counter value of 3 in both branches. Suppose there are further increments, 2 at master and 4 at wip, resulting in counter values 5 and 7 respectively in master and wip.

If the wip branch is now merged in master, there are two lowest common ancestors: the commit with value 1 at master and 2 in wip. Since the 3-way merge algorithm only work for a single common ancestor, the we adopt a recursive merge strategy, where the lowest common ancestors are first merged resulting in a internal commit with value 3 (represented by a dotted circle). This commit is now used as the common ancestor for merging, which results in 9 as the new state of the counter. This matches the increments done in both branches. The recursive merge strategy is also the default merge strategy for Git.

Mergeable logs

Another useful data type is mergeable logs, where each log message is a string. The merge operation accumulates the logs in reverse chronological order. To this end, each log entry is a pair of timestamp and message, and the log itself is a list of entries. They are constructed using mirage-tc:

The merge function extracts the newer entries from either branches, sorts them and appends to the front of the old list.

While this implementation is simple, it does not scale well. In particular, each commit stores the entire log as a single serialized blob. This does not take advantage of the fact that every commit can share the tail of the log with its predecessor. Moreover, every append to the log needs to deserialize the entire log, append the new entry and serialize the log again. Hence, append is an O(n) operation, where n is the size of the log. Merges are also worst case O(n). This is undesirable.

Efficient mergeable logs

We can implement a efficient logs by taking advantage of the fact that every commit shares the tail of the log with its predecessor.

type log_entry = {
  time    : Time.t;
  message : V.t;        (** V.t is type of message. *)
  prev    : K.t option  (** K.t is the type of address in the block store. *)
}

Merges simply add a new node which points to the logs of merged branches, resulting in a DAG that captures the causal history. The following sequence of operations:

utop # #require "ezirmin";;
utop # open Lwt.Infix;;
utop # module M = Ezirmin.Memory_log(Tc.String);;
utop # open M;;
utop # let m = Lwt_main.run (init () >>= master);;
utop # Lwt_main.run (
  append m [] "m0" >>= fun _ ->
  append m [] "m1" >>= fun _ ->
  clone_force m "wip" >>= fun w ->
  append w [] "w0" >>= fun _ ->
  append m [] "m2" >>= fun _ ->
  merge w ~into:m >>= fun _ ->
  append w [] "w1" >>= fun _ ->
  append w [] "w2" >>= fun _ ->
  append m [] "m3" >>= fun _ ->
  append m [] "m4"
);;

results in the heap below.

Read traverses the log in reverse chronological order.

utop # read_all m [];;
- : string list = ["m4"; "m3"; "m2"; "w0"; "m1"; "m0"]

This implementation has O(1) appends and O(1) merges, resulting in much better performance. The graph below compares the blob log implementation and this linked implementation with file system backend by performing repeated appends to the log and measuring the latency for append.

Each point represents the average latency for the previous 100 appends. The results show that the append latency for linked implementation remains relatively constant while the blob implementation slows down considerably with increasing number of appends. Additionally, the linked implementation also supports efficient paginated reads.

Mergeable ropes

A rope data structure is used for efficiently storing and manipulating very long strings. Ezirmin provides mergeable ropes where for arbitrary contents, but also specialized for strings. Ropes automatically rebalance to maintain the invariant that the height of the tree is proportional to the length of the contents. The crux of the merge strategy is that given a common ancestor and the two trees to be merged, the merge algorithm works out the smallest subtrees where the modification occurred. If the modifications are on distinct subtrees, then the merge is trivial.

If the modification is on the same subtree, then the algorithm delegates to merge the contents. This problem has been well studied under the name of operational transformation (OT). OT can be categorically explained in terms of pushouts. Mergeable strings with insert, delete and replace operations are isomorphic to counters with increment and decrement. We apply a similar strategy to merge string.

First we compute the diff between the common ancestor and the new tree using Wagner-Fischer algorithm. Then we transform one patch with respect to the other using standard OT definition such that we can first apply one of the original patch to the common ancestor and then apply the transformed patch of the other branch to get the result tree.

For example,

utop # #require "ezirmin";;
utop # open Lwt.Infix;;
utop # open Ezirmin.Memory_rope_string;;
utop # let m = Lwt_main.run (init () >>= master);;
utop # let t = Lwt_main.run (
  make "abc" >>= fun t ->
  write m [] t >>= fun _ ->
  Lwt.return t
);;
utop # let w = Lwt_main.run (clone_force m "w");;
utop # let _ = Lwt_main.run (
  set t 1 'x' >>= fun t' (* "axc" *) ->
  write m [] t' >>= fun _ ->

  insert t 1 "y" >>= fun t' (* "aybc" *)->
  write w [] t' >>= fun _ ->

  merge w ~into:m >>= fun _ (* "ayxc" *) ->
  merge m ~into:w
);;
utop # Lwt_main.run (
  read m [] >>= function
  | None -> failwith "impossible"
  | Some r -> flush r >|= fun s ->
  Printf.printf "m is \"%s\"\n" s
);;
- : unit = ()
m is "ayxc"
utop # Lwt_main.run (
  read w [] >>= function
  | None -> failwith "impossible"
  | Some r -> flush r >|= fun s ->
  Printf.printf "w is \"%s\"\n" s
)
- : unit = ()
w is "ayxc"

The combination of mergeable ropes with OT gets the best of both worlds. Compared to a purely OT based implementation, diffs are only computed if updates conflict at the leaves. The representation using ropes is also efficient in terms of storage where multiple versions of the tree shares blocks. A purely rope based implementation either has the option of storing individual characters (atoms) at the leaves (and resolve conflicts based on some deterministic mechanism such as timestamps or other deterministic strategies) or manifest the conflict at the leaves to the user to get it resolved. A simple strategy might be to present both of the conflicting strings, and ask the user to resolve it. Hence, mergeable ropes + OT is strictly better than either of the approaches.

Next steps

Ezirmin is open to comments and contributions. Next steps would be:

Implement more mergeable data types
Implement generic mergeable datatypes using depyt.
Explore the data types which admit conflicts. For example, a bank account with non-negative balance does not form a CRDT with a withdraw operation. However, operations such as deposit and accrue_interest can be coordination-free.

Footnotes

Things are indeed improving with a cleaner API in the 1.0 release. ↩
Push is currently broken. But given that Irmin is compatible with git, one can use git-push to publish changes. ↩
The same principle underlies the irrefutability of blockchain. No block can be changed without reflecting the change in every subsequent block. ↩

OCaml – log.examachine.net

2017-02-14T13:14:17-00:00

Parallel all-pairs similarity search algorithms in OCaml Sources Git repository ---> https://github.com/examachine/parallpairs The repository contains the 1.0 sources, a release will be made soon. Citation If you use this code, please cite the following paper. It is currently under review

Ocsigen Blog

2017-02-10T19:45:44-00:00

The Ocsigen team is very happy to announce the first release of two major new projects:

Ocsigen Start: a Web/mobile application skeleton written with Js_of_ocaml and Eliom. You can use this skeleton as a basis for your own app, or to learn Web/mobile app development in OCaml.
Ocsigen Toolkit: a set of responsive widgets for your mobile and Web applications in OCaml.

Both projects are available via OPAM and released under the LGPL.

Before installing, you can try out a demo of Ocsigen Start. The demo is available

on the Web;
via the Google Play store;
as an Android APK file; and
as an iOS application to be installed via XCode.

Ocsigen Start provides the basic features for user management (registration, activation links, password recovery, etc.), and some useful libraries. Remove the parts you don’t need for your own app.

Ocsigen Start also contains many examples of code: remote procedure calls, push notifications from the server, reactive pages, database interaction, session data, internationalization, and some widgets from Ocsigen Toolkit.

Ocsigen Toolkit contains common widgets for mobile and Web apps, written natively in OCaml and specifically designed for Eliom’s multi-tier and multi-platform programming style. You can also use them in client-only Js_of_ocaml programs. Many new widgets will be added in future versions.

Ocsigen Start and Ocsigen Toolkit are developed by the Ocsigen team at Be Sport, at the Paris Diderot University, and at the IRILL.

OCaml – log.examachine.net

2017-02-14T13:14:17-00:00

examachine has 22 repositories available. Follow their code on GitHub. Source: examachine (Eray Özkural) I just uploaded a bunch of free software projects to github. Some of them were on google code, which was terminated by google. You may

Gagallium

2017-03-14T08:00:00-00:00

I am no longer an official member of Gallium, but of course Gallium will always remain dear to my heart. And it is with a mix of sadness and anticipation that I've started writing articles on another blog, namely, the F* blog. If you wish to hear the latest news about F*, and chat about language issues, or just get a sense of what's happening, you can go out there and start lurking in the comments section.

Looking forward to some bikeshedding^W constructive discussions!

Ocsigen Blog

2017-02-10T19:45:44-00:00

The Eliom framework is the part of the ocsigen project that aims to provide high level libraries for developing client/server web applications. It contains a language extension of OCaml that allows implementing both the client and the server parts of your application as a single program. It also contains several libraries and utilities to facilitate web programming.

The various Ocsigen libraries have received a lot of care lately. Notably, we have reworked the service API, we have added support for mobile applications and, we have developed ocsigen-start.

Today, I will not talk about the ocsigen libraries. I will talk solely about the language extension.

The current language extension

The Eliom language extension extends OCaml with various annotations that allows specifying where things are to be defined and executed.

let%server s = 1 + 2 (* I'm executed on the server *)
let%server s2 = (* I'm declared on the server *)
  [%client 1 + 2 (* But I will be executed on the client! *) ]
let%client c =
  ~%s + ~%s2 + 1
  (* I access values on the server and execute things on the client! *)

The semantics is that the server part is executed first, then the web page is sent to the client, then the client part is executed. See the documentation for detail on the current extension.

The language extension is currently implemented using a PPX extension and a custom (and a bit sophisticated) compilation scheme. Note here that I used the word “language” extension on purpose: this is not a simple syntax extension, the Eliom language has its own type system, semantics and compilation scheme, which are extensions of the OCaml ones.

The current implementation of our language, based on PPX, started to show its limits in terms of flexibility, convenience and with respect to the safety guarantees it can provide. This is why I started, as part of my PhD thesis, to redesign and improve it.

Formalizing the Eliom language

Our first goal was to formalize the Eliom language as an extension of the OCaml language. Formalizing the language allowed us to better understand its type system and semantics, which led to various improvements and bug fixes. The formalization was published in APLAS 2016. In this paper, we present a (rather simple) type system based on two distinct type universes and the notion of converters, that allows passing values from the server to the client. We also show that the intuitive semantics of Eliom, that server code is executed immediately and client code is executed later in the exact same order it was encountered, does correspond to the compilation scheme used to slice Eliom programs into a server program and a client program.

In the the current implementation, when passing a server value of type Foo.t to the client. It also has type Foo.t, but the type is now the one available on the client. The actual object can also be transformed while passing the client/server boundary using wrappers. Unfortunately, this API is very difficult to use, not flexible and quite unsafe. Instead, we propose to use converters. Converters can be though as a pair of function: a server serialization function ty_server -> string and a client deserialization function string -> ty_client (the actual implementation will be a bit different to make (de)serializer composable). The correctness of a converter depends of course on the good behavior of these two functions, but the language guarantees that they will be used together properly and each sides will properly respect the types of the converter.

By using converters, we can provide a convenient programming model and make Eliom much easier to extend. We demonstrated this with multiple examples in another paper published in IFL 2016. Unfortunately, a proper implementation of converters is only possible with some form of ad-hoc polymorphism, which involve using modular implicits.

Implementing the Eliom language

In order to actually implement all these new things, I started to work on an extension of the OCaml compiler capable of handling the Eliom language constructs. Working directly in the compiler has several advantages:

We can implement the actual type system of Eliom directly.
Easier to extend with new features.
Much better error messages.
A simpler and faster compilation scheme.

The current work-in-progress compiler is available in the repository ocsigen/ocaml-eliom. A minimal runtime, along with various associated tools are available in ocsigen/eliomlang. A (perpetually broken) playground containing an extremely bare-bone website using eliomlang without the complete framework is available in ocsigen/eliomlang-playground.

Finally, the work on using this new compiler to compile the Eliom framework can be followed via this pull-request.

Going further

A more in-depth presentation of the Eliom language can be found here. The APLAS paper is quite formal and is mostly aimed at people that want to really understand the minute details of the language. The IFL paper, on the other hand, should be accessible to most OCaml programmers (even those who don’t know Eliom) and demonstrates how to use the new Eliom constructs to build nice, tierless and typesafe libraries for client/server web programming.

The future

The work on the Eliom language is far from done. A current area of work is to extend the OCaml module language to be aware of the Eliom annotations. A particularly delicate (but promising!) area is the ability to use Eliom annotations inside functors. A second area of work is that of stabilizing, debugging and documenting the patched compiler. Finally, a difficulty raised by this new compiler is that existing build systems, and in particular ocamlbuild, do not handle the Eliom compilation scheme very well. Some details on this can be found here.

I wish this progress report has awaken your appetite for well-typed and modular tierless programming in OCaml. I hope I will be able to share more news in a few months.

Happy Eliom programming!

My Little Garden of Code

2017-02-03T23:54:21-00:00

I finished up a 16-month LLVM contracting gig at the end of 2016. Got the flu a couple of weeks ago and have been pretty much out of commission until today when I finally had enough mental clarity and energy to get this blog going again.

Since I last posted in 2014 it seems that Octopress has been updated. I had to go back to my old desktop machine and find where all the blog-related files were and transfer them to my current desktop machine. Of course I tried looking at the Octopress documentation to figure out how it works since I’d completely forgotten in the last ~2.5 years. And of course, the docs no longer cover the old version of Octopress. So I went with the newer version and tried moving things over. For the most part it works. The twitter button that used to be over there on the right doesn’t seem to want to show up anymore and I have no idea why (it’s right there in the _config.yml). And pygments now seems to want a space between the language name and the ‘]’ - so whereas ‘[lang:ocaml]’ used to work for the code highlighting incantation, now it seems to want ‘[lang:ocaml ]’ … weird.

So now that the blog mostly seems to be working again, I’m going to be looking into BNNs (Binarized Neural Networks). Weights in BNNs are represented as binary values and instead of matrix multiplications (as is common in regular neural nets) the main operation on binary weights is XNOR which is a lot faster and more amenable for implementation in hardware like FPGAs. But more on BNNs later, I hope to start blogging about my BNN investigations which is why I wanted to get the blog going again.

Enter the void *

2017-02-01T00:00:00-00:00

Hello! It’s 2016, it’s November, and apparently it rhymes with #NaBoMaMo 2016, the National Bot Making Month. I made a bot!.

Full disclosure: it’s actually 2017, but I started writing this in 2016 so it’s OK. Also I’m not actually from the US, but I’ll relax the definition a bit and let’s pretend it means International Bot Making Year. Close enough!

Bots are all the rage - Twitter bots, IRC bots, Telegram bots… I decided to make a Slack bot to get more familiar with that API.

I wanted this to be a small project - write and forget, basically. I started by defining some specs and lock those down:

that bot works on Slack
it uses the “will it rain in the next hour” API from Météo France.
the bot understands 3 commands:
tell you whether it will rain or not.
show you a graph of rain level over the next hour.
tell you when to go out to avoid the rain.

The next step was choosing the tech stack. For hosting itself I was sold on using Heroku from previous projects (or another PaaS host, for what it’s worth)

As for the programming language itself, I hesitated between three choices:

focus on the all-included experience: something that has libraries, tooling, but somehow boring;
focus on the shipping experience: stuff that I use daily, but looking to get something online quickly;
focus on learning something new.

The first one means something like Python or Ruby. I am familiar with the languages and am pretty sure that there are libraries that can take care of the Slack API without me having to ever worry about HTTP endpoints. That means also first-class deployment and hosting.

The second one is about OCaml: it’s a programming language I use daily at work, but the real goal would be to focus on shipping: create a project, write tests, write implementation, deploy, repeat for new features, forget.

The third one means a totally new programming language. I heard a lot of good things about Elixir for backend applications and figured that it would be a good intro project. Learning a new language is always an interesting experience, because it makes you a better programmer in all languages, and having clear specs would make this manageable.

The Python/Ruby solution seemed a bit boring. I probably would not learn a lot, only, maybe add a couple libraries to my toolbelt at most.

Elixir sounds great, but learning a new language and a new project at the same time is too hard and too time consuming. I would rather write in a new language something I previously wrote in another language. Though for something small and focused like this, that could have worked.

I first created the project structure: github repo, ocaml project (topkg, opam, etc). I like to use TDD for this kind of projects, so I added a small alcotest suite. I also created the 12factor separation: a Procfile, a small bin/ shell that reads the application configuration from the environment and starts a bot from lib/.

I asked myself what to test: the cohttp library is nice, because servers and clients are built using normal functions that take a request and returns a response. That makes it possible to test almost everything at the ocaml level without having to go to the HTTP level. This is especially important since there is no way to mock values and functions in ocaml. Everything has to be real objects.

However, even if it was possible to test everything, I decided to just focus on the domain logic without testing the HTTP part: for example, I would pass data structures directly to my bot object rather than building a cohttp request.

A part that is important for me even for a small project like that, is to have some sort of CI: have travis run my test suite, and make a binary ready to be deployed to Heroku. That way, it is impossible to forget how to make changes, test and deploy, since this is all in a script.

The other part that needed work is the actual Slack integration. The “slash” command API is pretty simple: it is possible to configure a Slack team such that typing /rain will hit a particular URL. Some options are passed as POST data and whatever is returned is displayed in Slack.

I set up the Slack integration, wrote a function to distinguish between /rain and /rain list (using the POST data), and by the end of the second iteraton I had my second feature implemented, working, and deployed.

All in all, that was pretty great. The code or the bot itself are not particularly fantastic, but I learned some important lessons:

When you do not want to spend a lot of time on a task, invest in planning and keep the list of features short. That is pretty obvious in the context of paid work, but this is applies well to hobby programming too.
Know what to test and what not to. Tests are useful to ensure that changes can be made without breaking everything, but testing that your HTTP library can parse POST data is a waste of time.
In languages where it is not possible to mock or monkey patch functions, dependency injection is still possible. One may even argue that it leads to a better solution, since it removes the coupling between the different components.

You can find the source of this bot on Github. See you next year, #NaBoMaMo! And thanks to Tully Hansen for organizing this.

full stack engineer

2017-03-12T13:27:37-00:00

Back in 2014, when we implemented TLS in OCaml, at some point I was bored with TLS. I usually need at least two projects (but not more than 5) at the same time to procrastinate the one I should do with the other one - it is always more fun to do what you're not supposed to do. I started to implement another security protocol (Off-the-record, resulted in ocaml-otr) on my own, applying what I learned while co-developing TLS with David. I was eager to actually deploy our TLS stack: using it with a web server (see this post) is fun, but only using one half of the state machine (server side) and usually short-lived connections (discovers lots of issues with connection establishment) - not the client side and no long living connection (which may discover other kinds of issues, such as leaking memory).

To use the stack, I needed to find an application I use on a daily basis (thus I'm eager to get it up and running if it fails to work). Mail client or web client are just a bit too big for a spare time project (maybe not ;). Another communication protocol I use daily is jabber, or XMPP. Back then I used mcabber inside a terminal, which is a curses based client written in C.

I started to develop jackline (first commit is 13th November 2014), a terminal based XMPP client in OCaml. This is a report of a work-in-progress (unreleased, but publicly available!) software project. I'm not happy with the code base, but neverthelss consider it to be a successful project: dozens of friends are using it (no exact numbers), I got contributions from other people (more than 25 commits from more than 8 individuals), I use it on a daily basis for lots of personal communication.

What is XMPP?

The eXtensible Messaging and Presence Protocol (previously known as Jabber) describes (these days as RFC 6120) a communication protocol based on XML fragments, which enables near real-time exchange of structured (and extensible) data between two network entities.

The landscape of instant messaging used to contain ICQ, AOL instant messenger, and MSN messenger. In 1999, people defined a completely open protocol standard, then named Jabber, since 2011 official RFCs. It is a federated (similar to eMail) near-real time extensible messaging system (including presence information) used for instant messaging. Extensions include end-to-end encryption, multi-user chat, audio transport, ... Unicode support is builtin, everything is UTF8 encoded.

There are various open jabber servers where people can register accounts, as well as closed ones. Google Talk used to federate (until 2014) into XMPP, Facebook chat used to be based on XMPP. Those big companies wanted something "more usable" (where they're more in control, reliable message delivery via caching in the server and mandatory delivery receipts, multiple devices all getting the same messages), and thus moved away from the open standard.

XMPP Security

Authentication is done via a TLS channel (where your client should authenticate the server), and SASL that the server authenticates your client. I investigated in 2008 (in German) which clients and servers use which authentication methods (I hope the state of certificate verification improved in the last decade).

End-to-end encryption is achievable using OpenPGP (rarely used in my group of friends) via XMPP, or Off-the-record, which was pioneered over XMPP, and is still in wide use - it gave rise to forward secrecy: if your long-term (stored on disk) asymmetric keys get seized or stolen, they are not sufficient to decrypt recorded sessions (you can't derive the session key from the asymmetric keys) -- but the encrypted channel is still authenticated (once you verified the public key via a different channel or a shared secret, using the Socialist millionaires problem).

OTR does not support offline messages (the session keys may already be destroyed by the time the communication partner reconnects and receives the stored messages), and thus recently omemo was developed. Other messaging protocols (Signal, Threema) are not really open, support no federation, but have good support for group encryption and offline messaging. (There is a nice overview over secure messaging and threats.)

There is (AFAIK) no encrypted group messaging via XMPP; also the XMPP server contains lots of sensible data: your address book (buddy list), together with offline messages, nicknames you gave to your buddies, subscription information, and information every time you connect (research of privacy preserving presence protocols has been done, but is not widely used AFAIK, e.g. DP5).

XMPP client landscape

See wikipedia for an extensive comparison (which does not mention jackline :P).

A more opinionated analysis is that you were free to choose between C - where all code has to do manual memory management and bounds checking - with ncurses (or GTK) and OpenSSL (or GnuTLS) using libpurple (or some other barely maintained library which tries to unify all instant messaging protocols), or Python - where you barely know upfront what it will do at runtime - with GTK and some OpenSSL, or even JavaScript - where external scripts can dynamically modify the prototype of everything at runtime (and thus modify code arbitrarily, violating invariants) - calling out to C libraries (NSS, maybe libpurple, who knows?).

Due to complex APIs of transport layer security, certificate verification is still not always done correctly (that's just one example, you'll find more) - even if, it may not allow custom trust anchors or certificate fingerprint based verification - which are crucial for a federated operations without a centralised trust authority.

Large old code basis usually gather dust and getting bitrot - and if you add patch by patch from random people on the Internet, you've to deal with the most common bug: insufficient checking of input (or output data, if you encrypt only the plain body, but not the marked up one). In some programming languages this easily leads to execution of remote code, other programming languages steal the work from programmers by deploying automated memory management (finally machines take our work away! :)) - also named garbage collection, often used together with automated bounds checking -- this doesn't mean that you're safe - there are still logical flaws, and integer overflows (and funny things which happen at resource starvation), etc.

Goals and non-goals

My upfront motivation was to write and use an XMPP client tailored to my needs. I personally don't use many graphical applications (coding in emacs, mail via thunderbird, firefox, mplayer, mupdf), but stick mostly to terminal applications. I additionally don't use any terminal multiplexer (saw too many active screen sessions on remote servers where people left root shells open).

The goal was from the beginning to write a "minimalistic graphical user interface for a secure (fail hard) and trustworthy XMPP client". By fail hard I mean exactly that: if it can't authenticate the server, don't send the password. If there is no end-to-end encrypted session, don't send the message.

As a user of (unreleased) software, there is a single property which I like to preserve: continue to support all data written to persistent storage. Even during large refactorings, ensure that data on the user's disk will also be correctly parsed. There is nothing worse than having to manually configure an application after update. The solution is straightforward: put a version in every file you write, and keep readers for all versions ever written around. My favourite marshalling format (human readable, structured) are still S-expressions - luckily there is a sexplib in OCaml for handling these. Additionally, once the initial configuration file has been created (e.g. interactively with the application), the application does no further writes to the config file. Users can make arbitrary modifications to the file, and restart the application (and they can make changes while the application is running).

I also appreciate another property of software: don't ever transmit any data or open a network connection unless initiated by the user (this means no autoconnect on startup, or user is typing indications). Don't be obviously fingerprintable. A more mainstream demand is surely that software should not phone home - that's why I don't know how many people are using jackline, reports based on friends opinions are hundreds of users, I personally know at least several dozens.

As written earlier, I often take a look at the trusted computing base of a computer system. Jackline's trusted computing base consists of the client software itself, its OCaml dependencies (including OTR, TLS, tty library, ...), then the OCaml runtime system, which uses some parts of libc, and a whole UNIX kernel underneath -- one goal is to have jackline running as a unikernel (then you connect via SSH or telnet and TLS).

There are only a few features I need in an XMPP client: single account, strict validation, delivery receipts, notification callback, being able to deal with friends logged in multiple times with wrongly set priorities - and end-to-end encryption. I don't need inline HTML, avatar images, my currently running music, leaking timezone information, etc. I explicitly don't want to import any private key material from other clients and libraries, because I want to ensure that the key was generated by a good random number generator (read David's blog article on randomness and entropy).

The security story is crucial: always do strict certificate validation, fail hard, make it noticable by the user if they're doing insecure communication. Only few people are into reading out loud their OTR public key fingerprint, and SMP is not trivial -- thus jackline records the known public keys together with a set of resources used, a session count, and blurred timestamps (accuracy: day) when the publickey was initially used and when it was used the last time.

I'm pragmatic - if there is some server (or client) deployed out there which violates (my interpretation of) the specification, I'm happy to implement workarounds. Initially I worked roughly one day a week on jackline.

To not release the software for some years was something I learned from the slime project (watch Luke's presentation from 2013) - if there's someone complaining about an issue, fix it within 10 minutes and ask them to update. This only works if each user compiles the git version anyways.

User interface

Stated goal is minimalistic. No heavy use of colours. Visibility on both black and white background (btw, as a Unix process there is no way to find out your background colour (or is there?)). The focus is also security - and that's where I used colours from the beginning: red is unencrypted (non end-to-end, there's always the transport layer encryption) communication, green is encrypted communication. Verification status of the public key uses the same colours: red for not verified, green for verified. Instead of colouring each message individually, I use the encryption status of the active contact (highlighted in the contact list, where messages you type now will be sent to) to colour the entire frame. This results in a remarkable visual indication and (at least I) think twice before presssing return in a red terminal. Messages were initially white/black, but got a bit fancier over time: incoming messages are bold, multi user messages mentioning your nick are underlined.

The graphical design is mainly inspired by mcabber, as mentioned earlier. There are four components: the contact list in the upper left, chat window upper right, log window on the bottom (surrounded by two status bars), and a readline input. The sizes are configurable (via commands and key shortcuts). A different view is having the chat window fullscreen (or only the received messages) - useful for copy and pasting fragments. Navigation is done in the contact list. There is a single active contact (colours are inverted in the contact list, and the contact is mentioned in the status bar), whose chat messages are displayed.

There is not much support for customisation - some people demanded to have a 7bit ASCII version (I output some unicode characters for layout). Recently I added support to customise the colours. I tried to ensure it looks fine on both black and white background.

Code

Initially I targeted GTK with OCaml, but that excursion only lasted two weeks, when I switched to a lambda-term terminal interface.

UI

The lambda-term interface survived for a good year (until 7th Feb 2016), when I started to use notty - developed by David - using a decent unicode library.

Notty back then was under heavy development, I spend several hours rebasing jackline to updates in notty. What I got out of it is proper unicode support: the symbol 茶 gets two characters width (see screenshot at top of page), and the layouting keeps track how many characters are already written on the terminal.

I recommend to look into notty if you want to do terminal graphics in OCaml!

Application logic and state

Stepping back, an XMPP client reacts to two input sources: the user input (including terminal resize), and network input (or failure). The output is a screen (80x25 characters) image. Each input event can trigger output events on the display and the network.

I used to use multiple threads and locking between shared data for these kinds of applications: there can go something wrong when network and user input happens at the same time, or what if the output is interrupted by more input (which happens e.g. during copy and paste).

Initially I used lots of shared data and had hope, but this was clearly not a good solution. Nowadays I use mailboxes, and separate tasks which wait for receiving a message: one task which writes persistent data (session counts, verified fingerprints) periodically to ask, another which writes on change to disk, an error handler (init_system) which resets the state upon a connection failure, another task which waits for user input (read_terminal), one waiting for network input (Connect, including reconnecting timers), one to call out the notification hooks (Notify), etc. The main task is simple: wait for input, process input (producing a new state), render the state, and recursively call itself (loop).

Only recently I solved the copy and paste issue by delaying all redraws by 40ms, and canceling if another redraw is scheduled.

The whole state contains some user interface parameters (/buddywith, /logheight, ..), as well as the contact map, which contain users, which have sessions, each containing chat messages.

The code base is just below 6000 lines of code (way too big ;), and nowadays supports multi-user chat, sane multi-resource interaction (press enter to show all available resources of a contact and message each individually in case you need to), configurable colours, tab completions for nicknames and commands, per-user input history, emacs keybindings. It even works with the XMPP gateway provided by slack (some startup doing a centralised groupchat with picture embedding and animated cats).

Road ahead

Common feature requests are: omemo support, IRC support, support for multiple accounts (tbh, these are all things I'd like to have as well).

But there's some mess to clean up:

The XMPP library makes heavy use of functors (to abstract over the concrete IO, etc.), and embeds IO deep inside it. I do prefer (see e.g. our TLS paper, or my ARP post) these days to have a pure interface for the protocol implementation, providing explicit input (state, event, data), and output (state, action, potentially data to send on network, potentially data to process by the application). The sasl implementation is partial and deeply embedded. The XML parser is as well deeply embedded (and has some issues). The library needs to be torn apart (something I procrastinate since more than a year). Once it is pure, the application can have full control over when to call IO (and esp use the same protocol implementation as well for registering a new account - currently not supported).
On the frontend side (the cli subfolder), there is too much knowledge of XMPP. It should be more general, and be reusable (some bits and pieces are notty utilities, such as wrapping a string to fit into a text box of specific width, see split_unicode).
The command processing engine itself is 1300 lines (including ad-hoc string parsing) (Cli_commands), best to replaced by a more decent command abstraction.
A big record of functions (user_data) is passed (during /connect in handle_connect) from the UI to the XMPP task to inject messages and errors.
The global variable xmpp_session should be part of the earlier mentioned cli_state, also contacts should be a map, not a Hashtbl (took me some time to learn).
Having jackline self-hosted as a MirageOS unikernel. I've implemented a a telnet server, there is a notty branch be used with the telnet server. But there is (right now) no good story for persistent mutable storage.
Jackline predates some very elegant libraries, such as logs and astring, even result - since 4.03 part of Pervasives - is not used. Clearly, other libraries (such as TLS) do not yet use result.
After looking in more depths at the logs library, and at user interfaces - I envision the graphical parts to be (mostly!?) a viewer of logs, and a command shell (using a control interface, maybe 9p): Multiple layers (of a protocol), slightly related (by tags - such as the OTR session), and have the layers be visible to users (see also tlstools), a slightly different interface of similarly structured data. In jackline I'd like to e.g. see all messages of a single OTR session (see issue), or hide the presence messages in a multi-user chat, investigate the high-level message, its XML encoded stanza, TLS encrypted frames, the TCP flow, all down to the ethernet frames send over the wire - also viewable as sequence diagram and other suitable (terminal) presentations (TCP window size maybe in a size over time diagram).
Once the API between the sources (contacts, hosts) and the UI (what to display, where and how to trigger notifications, where and how to handle global changes (such as reconnect)) is clear and implemented, commands need to be reinvented (some, such as navigation commands and emacs keybindings, are generic to the user interface, others are specific to XMPP and/or OTR): a new transport (IRC) or end-to-end crypto protocol (omemo) - should be easy to integrate (with similar minimal UI features and colours).

Conclusion

Jackline started as a procrastination project, and still is one. I only develop on jackline if I enjoy it. I'm not scared to try new approaches in jackline, and either reverting them or rewriting some chunks of code again. It is a project where I publish early and push often. I've met several people (whom I don't think I know personally) in the multi-user chatroom jackline@conference.jabber.ccc.de, and fixed bugs, discussed features.

When introducing customisable colours, the proximity to a log viewer became again clear to me - configurable colours are for severities such as Success, Warning, Info, Error, Presence - maybe I really should get started on implementing a log viewer.

I would like to have more community contributions to jackline, but the lack of documentation (there aren't even a lot of interface files), mixed with a non-mainstream programming language, and a convoluted code base, makes me want some code cleanups first, or maybe starting from scratch.

I'm interested in feedback, either via twitter or on the jackline repository on GitHub.

OCaml Planet

Shayne Fletcher

OCamlPro

Specifying dependencies in opam 1.2

Extensions in opam 2.0

Conditional dependencies

Dependency flags

Computed versions

Conclusion

OCaml Weekly News

GitHub Jobs matching ocaml

Shayne Fletcher

Preprocessor extensions for code generation

Implementing the "id_of" ppx

The basic strategy

Implementing the mappings

Building and testing

OCamlPro

Status of opam build

Handling a local project

Specifying a destdir

Initialising

More related options

OCaml Weekly News

OCaml – Jane Street Tech Blogs

m3ga blog

OCamlPro

A reminder about switches

Using within projects

Interface

Implementation

Current status

Limitations and future extensions

OCaml – Jane Street Tech Blogs

OCaml Weekly News

GitHub Jobs matching ocaml

OCaml Labs compiler hacking

Drup's thingies

OCamlPro

Solving it

Wrapping it up

OCaml Weekly News

Frama-C RSS News

OCaml Weekly News

Drup's thingies

Shayne Fletcher

Describing locations

Error reporting with locations

Exception handling involving errors with locations

Usage

OCaml Weekly News

Mathematics and Computation

Shayne Fletcher

Arithmetics and rings

Polynomials

Testing and example usage

OCaml Weekly News

OCaml – Jane Street Tech Blogs

What is cinaps?

Real example

Setting up and using cinaps

Pushing the system

Tweak: indenting the generated code

History behind the name

Status

OCamlPro

What does it do ?

Additional functions

Installation

Choosing custom repositories

Multiple packages

Implementation details

Planned extensions

OCaml – Jane Street Tech Blogs

OCaml Weekly News

Gagallium

Problem Statement

Cascades

Back to the problem

From Delayed Trees to Cascades and Iterators

Implementing the "`id_of`" ppx

Status of `opam build`