RReverser's

Bringing Sharp to WebAssembly and WebContainers

2023-08-03T00:00:00Z

Github project: serde-ndim

2023-04-02T14:12:44Z

Github project: ascom-alpaca-rs

2023-03-18T21:47:25Z

Github project: eos-remote-web

2023-02-27T12:38:18Z

Drawing to canvas in Emscripten

2022-02-07T00:00:00Z

Porting USB applications to the web. Part 2: gPhoto2

2022-02-01T00:00:00Z

WebAssembly feature detection

2022-01-27T00:00:00Z

Porting USB applications to the web. Part 1: libusb

2022-01-20T00:00:00Z

Embedding JavaScript snippets in C++ with Emscripten

2022-01-18T00:00:00Z

Bundling non-JavaScript resources

2021-09-08T00:00:00Z

Using WebAssembly threads from C, C++ and Rust

2021-07-12T00:00:00Z

Using asynchronous web APIs from WebAssembly

2021-04-26T00:00:00Z

Github project: lutz

2021-04-10T22:37:29Z

Github project: wasm-bindgen-rayon

2021-03-08T17:39:29Z

Github project: sep-sys-rs

2021-02-21T23:59:05Z

WebAssembly shell with a real filesystem access in a browser

2020-10-19T00:14:30Z

I've built and open-sourced a new thing - WebAssembly shell with a real filesystem access in a browser.

It's powered by WebAssembly, WASI, Asyncify and File System Access API, and, I believe, shows just how powerful Web can be today when its different components are connected together in a single app.

You can access the live version here: https://wasi.rreverser.com/

Check out the code here: https://github.com/GoogleChromeLabs/wasi-fs-access

Or watch a video demonstrating some of the powerful features:

https://youtu.be/qRmO-8b4WmE

How

It provides WASI bindings implementation that proxies any filesystem requests to a real, host filesystem. This allows apps built in languages like C, C++, Rust and others to be compiled to WebAssembly and work as usual within a browser sandbox, accessing and manipulating files in a "real world".

Since WASI APIs are synchronous by nature, but Web APIs are traditionally asynchronous to avoid blocking the main thread, Asyncify is used to bridge the two types of APIs together. Asyncify is a feature created as part of Emscripten and later extended to work with arbitrary WebAssembly files with the help of a custom JavaScript wrapper.

A Rust port of coreutils with some patches was chosen for the demo purposes, but it should be possible to extract and reuse same bindings for any applications compiled for the WebAssembly + WASI target.

Note that some commands in the demo might not work due to either limitations of the WASI itself (such as an absent concept of "current working directory" and, correspondingly, relative paths and such), limitations of the File System Access API (such as an absent support for symlinks), or simply due to hardcoded assumptions about the target system in the used coreutils codebase itself. Most of those limitations can be easily worked around or will be naturally fixed as both APIs develop over time.

Want to learn more?

Check out my presentation from the WebAssembly Live! here: https://www.slideshare.net/RReverser/asyncifying-webassembly-for-the-modern-web

I'll also write a more detailed blog post later, but, meanwhile, feel free to check out the code and ping me on Twitter with any questions.

wasmbin: a self-generating WebAssembly parser & serializer

2020-10-17T19:01:08Z

If you prefer to skip right to the code, go here: https://github.com/RReverser/wasmbin

Backstory

Few months ago, I've been playing with various WebAssembly analysis and transformations in Rust and had a chance to try out several parsing / transformation / serialization toolchains.

As I was exploring this space, I've realised that what I really want is an API that would be high-level enough to allow me to work with the WebAssembly module AST as a natural Rust structure, yet low-level enough to preserve the binary contents of the module for any parts I don't modify in the process.

Here are some examples of toolchains I've tried and their pros/cons for my particular goals:

wasmparser:
- (+) Low-level access to the WebAssembly binary.
- (+) Allows to lazily skip parts of the binary that you're not interested in parsing more deeply.
- (+) Allows to parse individual items and not only module as a whole.
- (+) Provides event-driven as well as structural API for reading.
- (-) Allows parsing only from a byte slice, making the structural result hard to store / relocate without things like owning-ref.
- (-) No support for serialization, making unsuitable for any transforms on its own.
- (-) Some places could use newtypes a bit more actively (e.g. I'd love to see separate types for function ids / tables ids / etc., rather than all being generic numbers).
- (-) The produced data structures lack standard basic trait implementations (e.g. PartialEq / Eq, Hash, etc.).
walrus:
- (+) Support for both parsing (based on wasmparser) and serialization.
- (+) Very high-level IR, making most modifications easy without worrying about details. For example, one of the problems tricky to solve with lower-level APIs is when you add an import and suddenly all function IDs are shifted and all call instructions are pointing to wrong places - walrus takes care of this for you.
- (+) All IDs are strongly typed, making them harder to mix up.
- (+) Allows parsing from arbitrary sources, the result doesn't have a lifetime linked to the input and can be moved anywhere.
- (+) Very well tested, has built-in debugging helpers like GraphViz output.
- (-) Roundtrip is a non-goal and unsupported, entire module is rebuilt from IR upon serialization. On pre-optimised binaries, this also means that the result can be larger than the original, making it hard to use for writing custom optimisations.
- (-) Due to the IR being very high-level, no standard trait implementations like Eq, Hash, Clone and so on, making it hard to store parsed items in other data structures.
- (-) Working with function bodies can be sometimes awkward as they are represented in SSA-like form. You can't get access to the entire function body as a flat instruction list - instead, you're only given an entry instruction sequence, and a special visitor API that can walk from it to other connected instruction sequences. Similarly, on the other side of the API, there is a builder which you can use to build up instruction sequences and connect them together. This works well for many cases, but, for example, when I wanted to just make a copy of a function, I found it rather hard to make two APIs to correctly sync up with each other and gave up.
- (-) No lazy parsing or a way to parse separate parts, it's all or nothing.
parity-wasm
- (+) Very close to a simple AST I wanted, with clear correspondence to the module structure.
- (+) Support for both parsing and serialization.
- (+) (follows from two points above) Preserves most of the module during round-trip.
- (+) Provides most of the expected traits, others are easy to contribute.
- (+) Allows parsing / serializing separate parts.
- (+) Convenience helpers for common tasks (e.g. setting function as main, inserting section in a correct place etc.).
- (+) Parsing from any source.
- (-) No lazy parsing, entire module is parsed at the same time.
- (-) Although gives access to "raw" representation of all fields, for some reason hides them behind function-based accessors, making them hard to use in pattern matching.
- (-) Like in wasmparser, IDs are simple numbers rather than specialised newtypes.

Once again, I want to reiterate that since are my personal pet peeves that applied to the weird and fun use-cases I've been playing with, and by no means are indicative of how good one or another library is - all of them are successfully used by a lot of other people, and are likely to work for you too!

Let's build one more

Now, one thing you should know about me (in case you haven't noticed from my blog yet) is that I'm a parsing nerd. So what these "downsides" did provide for me, was an excuse to try and build my own low-level parsing/serialization library for WebAssembly.

Moreover, as any parsing nerd, I love Rust macros. I've blogged about writing them before, I've used them in various projects, and generally consider them a powerful tool that can significantly increase readability and maintainability of code when used correctly.

So what I wanted to do for a while is to build a parser & serializer that would fully self-generate themselves using a macros. This way, I as a maintainer would only have to correctly describe all the structures and not bother with handwriting any implementations except for the core types. Think serde, but combining custom Deserialize & Deserializer for one specific format (offtop: there are reasons why "just" using Serde wouldn't work here, but it's a bit out of scope of this post).

This brings us to custom derives. To recap, most of the Rust code out there relies on a powerful feature called derive - a special kind of macros that can be put on any user-defined type and used to auto-generate an accompanying implementation block.

For example, when you write something like:

#[derive(Debug, Hash, PartialEq, Eq)]
pub struct MyStruct {
    pub field: u32,
    pub another_field: bool,
    pub yet_another_field: Option<AnotherStruct>,
}

those Hash and PartialEq within the derive attribute get expanded into a code that recursively walks the structure fields and does the "right thing" to fold the results together:

impl ::core::hash::Hash for MyStruct {
    fn hash<__H: ::core::hash::Hasher>(&self, state: &mut __H) -> () {
        match *self {
            MyStruct {
            field: ref __self_0_0,
            another_field: ref __self_0_1,
            yet_another_field: ref __self_0_2 } => {
                ::core::hash::Hash::hash(&(*__self_0_0), state);
                ::core::hash::Hash::hash(&(*__self_0_1), state);
                ::core::hash::Hash::hash(&(*__self_0_2), state)
            }
        }
    }
}
...
impl ::core::cmp::PartialEq for MyStruct {
    fn eq(&self, other: &MyStruct) -> bool {
        match *other {
            MyStruct {
            field: ref __self_1_0,
            another_field: ref __self_1_1,
            yet_another_field: ref __self_1_2 } =>
            match *self {
                MyStruct {
                field: ref __self_0_0,
                another_field: ref __self_0_1,
                yet_another_field: ref __self_0_2 } =>
                (*__self_0_0) == (*__self_1_0) &&
                    (*__self_0_1) == (*__self_1_1) &&
                    (*__self_0_2) == (*__self_1_2),
            },
        }
    }
    ...
}

What if I could define all the necessary structures corresponding to the WebAssembly spec and put a custom derive in the end that would take care of generating the rest - parsing, serializing and visiting code?

#[derive(..., Wasmbin)]

And this is how Wasmbin was born. First of all, I've defined a couple of core traits:

pub trait Encode {
    fn encode(&self, w: &mut impl std::io::Write) -> std::io::Result<()>;
}

pub trait Decode: Sized {
    fn decode(r: &mut impl std::io::Read) -> Result<Self, DecodeError>;
}

Encode is for serializing the given object into an arbitrary write stream, and Decode is for reading it from an arbitrary read stream.

Then (not without a help of few other macros for templating) I've defined those traits on all necessary core types, according to the rules of the WebAssembly spec.

For example, the implementation for u8:

impl Encode for u8 {
    fn encode(&self, w: &mut impl std::io::Write) -> std::io::Result<()> {
        std::slice::from_ref(self).encode(w)
    }
}

impl Decode for u8 {
    fn decode(r: &mut impl std::io::Read) -> Result<Self, DecodeError> {
        let mut dest = 0;
        r.read_exact(std::slice::from_mut(&mut dest))?;
        Ok(dest)
    }
}

or integers:

macro_rules! def_integer {
    ($ty:ident, $leb128_method:ident) => {
        impl Encode for $ty {
            fn encode(&self, w: &mut impl std::io::Write) -> std::io::Result<()> {
                leb128::write::$leb128_method(w, (*self).into()).map(|_| ())
            }
        }

        impl Decode for $ty {
            fn decode(r: &mut impl std::io::Read) -> Result<Self, DecodeError> {
                const LIMIT: u64 = (std::mem::size_of::<$ty>() * 8 / 7) as u64 + 1;

                let mut r = std::io::Read::take(r, LIMIT);
                let as_64 = leb128::read::$leb128_method(&mut r)?;
                let res = Self::try_from(as_64)
                    .map_err(|_| DecodeError::Leb128(leb128::read::Error::Overflow))?;

                Ok(res)
            }
        }

        impl Visit for $ty {}
    };
}

def_integer!(u32, unsigned);
def_integer!(i32, signed);
def_integer!(u64, unsigned);
def_integer!(i64, signed);

Then I needed to do the same for container types. E.g. slices / vectors of "countable" items:

impl<T: WasmbinCountable + Encode> Encode for [T] {
    fn encode(&self, w: &mut impl std::io::Write) -> std::io::Result<()> {
        self.len().encode(w)?;
        for item in self {
            item.encode(w)?;
        }
        Ok(())
    }
}

impl<T: WasmbinCountable + Decode> Decode for Vec<T> {
    fn decode(r: &mut impl std::io::Read) -> Result<Self, DecodeError> {
        let count = usize::decode(r)?;
        std::iter::repeat_with(|| T::decode(r))
            .take(count)
            .collect()
    }
}

Once I was done with all the core & helper types, I could write a proc-macro to handle the custom derive. From here on, defining the Wasm structures was a breeze.

Let's take an Import node for the example:

#[derive(Wasmbin, ...)]
pub struct Import {
    pub path: ImportPath,
    pub desc: ImportDesc,
}

Under the hood, this auto-generates corresponding implementations by recursively walking over fields, just like other traits do:

impl Encode for Import {
    fn encode(&self, w: &mut impl std::io::Write) -> std::io::Result<()> {
        match *self {
            Import {
                path: ref __binding_0,
                desc: ref __binding_1,
            } => {
                Encode::encode(__binding_0, w)?;
                Encode::encode(__binding_1, w)?;
            }
        }
        Ok(())
    }
}

impl Decode for Import {
    fn decode(r: &mut impl std::io::Read) -> Result<Self, DecodeError> {
        Ok(Import {
            path: Decode::decode(r)?,
            desc: Decode::decode(r)?,
        })
    }
}

For enums, I've leveraged the recently added discriminant support for arbitrary enums. It's currently natively supported only on nightly Rust, but if I drop another custom proc-macro in front (#[wasmbin_discriminants]), then I can "polyfill" the support by reading the discriminants and stripping them away for the stable compiler:

#[wasmbin_discriminants]
#[derive(Wasmbin, ...)]
#[repr(u8)]
pub enum ImportDesc {
    Func(TypeId) = 0x00,
    Table(TableType) = 0x01,
    Mem(MemType) = 0x02,
    Global(GlobalType) = 0x03,
}

This way, I'm telling my derive code that the discriminant should be parsed as a u8 and then matched to parse the corresponding variant (and vice versa during serialization). The example above generates code like following:

impl Encode for ImportDesc {
    fn encode(&self, w: &mut impl std::io::Write) -> std::io::Result<()> {
        match *self {
            ImportDesc::Func(ref __binding_0) => <u8 as Encode>::encode(&0x00, w)?,
            ImportDesc::Table(ref __binding_0) => <u8 as Encode>::encode(&0x01, w)?,
            ImportDesc::Mem(ref __binding_0) => <u8 as Encode>::encode(&0x02, w)?,
            ImportDesc::Global(ref __binding_0) => <u8 as Encode>::encode(&0x03, w)?,
            _ => {}
        };
        match *self {
            ImportDesc::Func(ref __binding_0) => Encode::encode(__binding_0, w)?,
            ImportDesc::Table(ref __binding_0) => Encode::encode(__binding_0, w)?,
            ImportDesc::Mem(ref __binding_0) => Encode::encode(__binding_0, w)?,
            ImportDesc::Global(ref __binding_0) => Encode::encode(__binding_0, w)?,
        }
        Ok(())
    }
}

impl DecodeWithDiscriminant for ImportDesc {
    type Discriminant = u8;

    fn maybe_decode_with_discriminant(
        discriminant: u8,
        r: &mut impl std::io::Read,
    ) -> Result<Option<Self>, DecodeError> {
        Ok(Some(match discriminant {
            0x00 => ImportDesc::Func(Decode::decode(r)?),
            0x01 => ImportDesc::Table(Decode::decode(r)?),
            0x02 => ImportDesc::Mem(Decode::decode(r)?),
            0x03 => ImportDesc::Global(Decode::decode(r)?),
            _ => return Ok(None),
        }))
    }
}

Now, you might notice that I've introduced another intermediate decode trait for enums. The reason here is that I've made my API work with arbitrary streams. It means that, once a "discriminant" (the byte which determines which variant should be parsed) is taken from the stream, it can't be put back.

This could be a problem for nested enums, where we don't know in advance which variant is going to be matched. For example, before the introduction of multi-value types, my BlockType definition looked like this:

#[derive(Wasmbin, ...)]
enum BlockType {
  Empty = 0x80,
  Value(ValueType),
}

Here, the autogenerated code needs to read a discriminant and match it against 0x80. If it matches, then the variant is Empty, but if it doesn't, it needs to reuse the same discriminant to parse the nested ValueType instead:

impl Encode for BlockType {
    fn encode(&self, w: &mut impl std::io::Write) -> std::io::Result<()> {
        match *self {
            BlockType::Empty => <u8 as Encode>::encode(&0x80, w)?,
            _ => {}
        };
        match *self {
            BlockType::Empty => {},
            BlockType::Value(ref __binding_0) => Encode::encode(__binding_0, w)?,
        }
        Ok(())
    }
}

impl DecodeWithDiscriminant for ImportDesc {
    type Discriminant = u8;

    fn maybe_decode_with_discriminant(
        discriminant: u8,
        r: &mut impl std::io::Read,
    ) -> Result<Option<Self>, DecodeError> {
        Ok(Some(match discriminant {
            0x80 => BlockType::Empty,
            _ => {
                if let Some(res) = DecodeWithDiscriminant::maybe_decode_with_discriminant(discriminant, r)? {
                    Self::Value(res)
                } else {
                    return Ok(None);
                }
            }
        }))
    }
}

If ValueType would implement only Decode directly, there would be no way to pass the same discriminant again (another alternative could be to allow only peekable streams, but it would be a user-facing inconvenience and not worth it for just passing one byte around).

Similarly, I can put derive(Wasmbin) on most other structs and enums that Wasm module consists of. For example, this is how the definition of the ValueType enum looks like:

#[derive(Wasmbin, ...)]
#[repr(u8)]
pub enum ValueType {
  I32 = 0x7F,
  I64 = 0x7E,
  F32 = 0x7D,
  F64 = 0x7C,
}

Or, probably the most interesting one, Instruction enum (showing only part of the list because the full one is huge and would require even more, potentially erroneous, code if not for auto-generation):

#[wasmbin_discriminants]
#[derive(Wasmbin, ...)]
#[repr(u8)]
pub enum Instruction {
  Unreachable = 0x00,
  Nop = 0x01,
  BlockStart(BlockType) = OP_CODE_BLOCK_START,
  LoopStart(BlockType) = OP_CODE_LOOP_START,
  IfStart(BlockType) = OP_CODE_IF_START,
  IfElse = 0x05,
  End = OP_CODE_END,
  Br(LabelId) = 0x0C,
  BrIf(LabelId) = 0x0D,
  BrTable { branches: Vec<LabelId>, otherwise: LabelId } = 0x0E,
  Return = 0x0F,
  Call(FuncId) = 0x10,
  CallIndirect { ty: TypeId, table: TableId } = 0x11,
  /* ... truncated as it's reallllly long */
}

Of course, there are few exceptions that can't be mapped as easily and need to be handled manually. For example, I've mentioned that the BlockType definition above was before the introduction of multi-value types to WebAssembly.

In order to accomodate for multi-value types, the spec definition was modified to the following:

Block types are encoded in special compressed form, by either the byte 0x40 indicating the empty type, as a single value type, or as a type index encoded as a positive signed integer.
blocktype ::= 0x40          => ε
            | t:valtype     => t
            | x:s33         => x (if x >= 0)

That is, now only a difference in the bit pattern determines whether something is a ValueType, or a beginning of a LEB128-encoded 33-bit signed integer that should be treated as a type index for the multi-value type... Yeah, there is no good way to automate this and it's easier to define encoding/decoding manually.

impl Decode for BlockType {
  fn decode(r: &mut impl std::io::Read) -> Result<Self, DecodeError> {
    let discriminant = u8::decode(r)?;
    if discriminant == OP_CODE_EMPTY_BLOCK {
      return Ok(BlockType::Empty);
    }
    if let Some(ty) = ValueType::maybe_decode_with_discriminant(discriminant, r)? {
      return Ok(BlockType::Value(ty));
    }
    let index = /* ... */;
    Ok(BlockType::MultiValue(TypeId { index }))
  }
}

Still, aside from those edge-cases, the autogenerated implementations are covering the majority of the spec and have been a huge boost to the maintainability and ease of future additions.

For example, this is all it took to implement the tail-call proposal:

+#[derive(Wasmbin, Debug, Arbitrary, PartialEq, Eq, Hash, Clone, Visit)]
+pub struct CallIndirect {
+    ty: TypeId,
+    table: TableId,
+}
+
 #[wasmbin_discriminants]
 #[derive(Wasmbin, Debug, Arbitrary, PartialEq, Eq, Hash, Clone, Visit)]
 #[repr(u8)]
 pub enum Instruction {
     ...
     Call(FuncId) = 0x10,
-    CallIndirect {
-        ty: TypeId,
-        table: TableId,
-    } = 0x11,
+    CallIndirect(CallIndirect) = 0x11,
+    #[cfg(feature = "tail-call")]
+    ReturnCall(FuncId) = 0x12,
+    #[cfg(feature = "tail-call")]
+    ReturnCallIndirect(CallIndirect) = 0x13,
     Drop = 0x1A,
     ...

High-level API

Generally, you can access and modify fields directly on arbitrary type, as well as ::decode(...) it from a stream or .encode(...) to one.

Aside from that, the top-level Module type has a couple of extra convenience helpers, with few of them - in particular, retrieving and inserting sections in the correct order - inspired by other libraries mentioned above:

Sections are represented as a vector of tagged enums rather than properties to guarantee order preservation.

In the WebAssembly spec, each section is preceded with a byte length to make it easier for parsers to read them lazily, skip them, or even read in parallel. In the Wasmbin type system, this is implemented via a lazy Blob wrapper which you can see in the bodies of each Section variant above.

It means that, when you decode a Module, then only raw byte vectors are read for each section. Later, you can choose which sections you actually want to decode and when. For example, if you were to find an import for env.memory you could write code like the following:

if let Some(imports) = wasmbin.find_std_section::<payload::Import>() {
  for import in imports.try_contents()?.iter() {
    match import.desc {
      ImportDesc::Mem(_) => {
        if import.path.module != "env" || import.path.name != "memory" { /* ... */ }
      }
      _ => {}
    }
  }
}

In this case, Blob::try_contents will lazily parse the contents on the first access and store them for future reads.

As an extra optimisation, if you don't modify the section, then during reserialization it will use the original raw bytes, but if some section is modified by accessing Blob::try_contents_mut instead, then it will recursively serialize the contents, too.

You can find the full API with several more traits and methods here: https://docs.rs/wasmbin/latest/wasmbin/module/struct.Module.html

Trade-offs

This post wouldn't be complete and fair without talking about trade-offs and downsides of this approach and the current implementation. Just to name a few major ones I keep in mind:

The type structures are intentionally mirroring the WebAssembly spec rather than abstracting it away. On one hand, this is what allows to auto-generate parsing, serialization and visiting code for most structures, but, on another, it also means that any change to the spec structure has a higher risk of being a breaking API change in the crate. One example of this can be seen above in the tail-call proposal example - CallIndirect structure got extracted as it's now reused by several variants, so any users will have to make minor adjustment to their code to adapt on upgrade.
The encoding / decoding code is currently designed to work with arbitrary std::io::Read / std::io::Write streams. On one hand, it makes it easier to integrate with any I/O sources, but, on another, this choice makes it slower and more memory-consuming for the common scenario of working with byte vectors.
While most of the changes require less code to work, the expansive usage of macros (derive, proc-macros as well as regular ones) is likely to make the contribution curve steeper than in alternatives for people who are less accustomed to this language feature.

Feedback

Let me know if you find wasmbin useful or have any questions about the article, please do let me know either on the repo or Twitter!

Debugging memory leaks in WebAssembly using Emscripten

2020-08-13T00:00:00Z

Bringing OxiPNG to Squoosh

2020-04-15T12:29:58Z

It took only 7.5 months 😅 (j/k, there were reasons) since opening the PR, but Squoosh.app now utilises OxiPNG instead of OptiPNG for PNG compression!

First attempt

OxiPNG is a Rust alternative to OptiPNG - a popular PNG compressor.

The main benefit of OxiPNG over OptiPNG is it utilises multi-threading on platforms that support it, and we wanted to provide a path to leverage it on WebAssembly.

Unfortunately, during the initial attempt, we found out that OxiPNG always results in a worse compression 😞

Squoosh logo: 33.9k vs 34.3k (1.2% bigger)

https://upload.wikimedia.org/wikipedia/commons/thumb/1/1a/Su25-kompo-vers2.svg/2880px-Su25-kompo-vers2.svg.png 472k vs 573k (21% bigger)

https://upload.wikimedia.org/wikipedia/commons/thumb/2/2f/Google_2015_logo.svg/1000px-Google_2015_logo.svg.png No difference.

The above image, but reduced to 53 colours: 10.3k vs 11k (6.8% bigger).

https://upload.wikimedia.org/wikipedia/commons/thumb/3/3b/Android_new_logo_2019.svg/1000px-Android_new_logo_2019.svg.png also reduced to 53 colours. 7.18k vs 7.66k (6.7% bigger).

https://upload.wikimedia.org/wikipedia/en/thumb/a/a4/Flag_of_the_United_States.svg/1024px-Flag_of_the_United_States.svg.png also reduced to 53 colours. 12.2k vs 12.7k (4% bigger).

https://upload.wikimedia.org/wikipedia/en/thumb/a/ae/Flag_of_the_United_Kingdom.svg/1024px-Flag_of_the_United_Kingdom.svg.png reduced to 53 colours. 3.48k vs 4.1k (17.8% bigger).

On its own, this could be written off as just an unfortunate difference between libraries, and, as such, a dead end...

What has been surprising though, is that these regressions didn't reproduce on OxiPNG outside of our integration 🤨

I wonder if we're holding it wrong. Using ImageOptim, in "extreme" mode:

Squoosh logo:
Our optipng: 33.9k
Our oxipng: 34.3k
ImageOptim oxi: 33.9k

https://upload.wikimedia.org/wikipedia/commons/thumb/1/1a/Su25-kompo-vers2.svg/2880px-Su25-kompo-vers2.svg.png
Our opti: 472k
Our oxi: 573k
ImageOptim oxi: 574k

https://upload.wikimedia.org/wikipedia/commons/thumb/2/2f/Google_2015_logo.svg/1000px-Google_2015_logo.svg.png reduced to 53 colours
Our opti: 10.3k
Our oxi: 11k
ImageOptim oxi: 10.3k

https://upload.wikimedia.org/wikipedia/en/thumb/a/a4/Flag_of_the_United_States.svg/1024px-Flag_of_the_United_States.svg.png also reduced to 53 colours
Our opti: 12.2k
Our oxi: 12.7k
ImageOptim oxi: 12.2k.

https://upload.wikimedia.org/wikipedia/en/thumb/a/ae/Flag_of_the_United_Kingdom.svg/1024px-Flag_of_the_United_Kingdom.svg.png reduced to 53 colours.
Our opti: 3.48k
Our oxi: 4.1k
ImageOptim oxi: 3.5k

ImageOptim's version/integration of oxi is better than our's. Although there's the odd output size regression, the only major regression in that large airplane image.

There is no way same code compiled to Wasm could result in a worse compression.

Indeed, upon further inspection, it turned out that OxiPNG utilises two different DEFLATE libraries to compress PNG image data, depending on the target platform support.

One is a Rust wrapper for Cloudflare's fork of the common zlib library that provides fantastic compression speed improvements, but works only on x64 and ARM64 targets.

Another is miniz_oxide - a Rust port of a miniz library, which is used on all other platforms by OxiPNG - including our WebAssembly target. miniz is designed to be a fast & tiny drop-in replacement for zlib, but doesn't provide as good compression, which explains the results above.

Once we found this out, there were a few possible paths to explore.

cloudflare-zlib (C fork)

First, cloudflare-zlib could add support for Wasm. Unfortunately, SIMD in WebAssembly is not yet stable, and while it can be experimented with, it wouldn't solve our immediate problem for most users yet.

cloudflare-zlib could also add fallbacks to regular implementation of functions for unsupported targets, but that doesn't align with goals of the fork and would likely result in maintenance complications, so wasn't brought up.

cloudflare-zlib (Rust wrapper)

Second, the Rust wrapper could fall back to a regular zlib on unsupported targets. We talked a bit about this with the author of the wrapper in DMs, but this could be tricky to do in a general case, because the wrapper wants to detect SIMD support at runtime and we can't link to both libraries statically due to symbol conflicts.

It might still work if we decide to use regular zlib at least on platforms that are definitely not supported by cloudflare-zlib, but meanwhile I decided to look into other approaches.

flate2-rs

Third was to make OxiPNG use another wrapper that already abstracts over these libraries statically.

Luckily, a popular flate2-rs library has already supported miniz-oxide as well as zlib, and someone has recently contributed cloudflare-zlib support too, which made it a perfect match!.. Well, almost.

One problem is that OxiPNG, like OptiPNG, works by iterating over various combinations of PNG filters as well as low-level zlib options and essentially brute-forcing its way to the one that works best for the given image.

flate2 exposes these on low-level zlib bindings, but not yet on a high-level API. It wouldn't be hard to propagate these options, but it still requires some extra work.

zlib (raw Rust bindings)

Another problem is that, by default, zlib depends on some libc functions to take care of allocation, deallocation and copying memory around.

Unfortunately, Clang ships with a completely bare-bones wasm32-unknown-unknown target, which doesn't provide any functions or headers, even platform-independent ones. It expects you to bring your own sysroot.

Some projects that provide their own sysroot are Emscripten and WASI SDK, but it feels like an overkill to bring an entire new toolchain to build a library that doesn't even need any platform-specific APIs.

Moreover, Rust provides its own allocator for the wasm32-unknown-unknown target and we would want to reuse it rather than bring an extra libc one (I wonder if two allocators even work on the same Wasm memory).

As it turned out, zlib has already taken care of this back in 2011 - not for WebAssembly, I presume 😀 - by introducing a separate compilation "solo" mode that avoids any library dependencies at the cost of losing some high-level utilities as well as helpers that could operate on files, and requires embedders to always specify custom alloc / free functions via options.

Luckily, flate2 already checks all these boxes - it doesn't use any of these helpers anyway and already passes custom allocation functions to reuse Rust allocator - so it can be made to work with this solo mode quite easily.

As the first step, I've made an upstream PR to official "raw" Rust + zlib bindings to add such support.

I've also updated flate2 locally to use these updated bindings, and verified that it finally compiles and works great on wasm32-unknown-unknown target, and even provides some nice size savings for other platforms!

Unfortunately, the raw bindings don't seem to be actively maintained these days, so the PR is still waiting for a review, and I've decided to explore yet another approach meanwhile.

libdeflate

I was looking through alternative wrappers for zlib as well as pure-Rust libraries in case I missed an even better solution.

There were mostly decoder implementations, some basic encoders, the already known bindings to zlib / cloudflare-zlib / miniz, the mentioned miniz port, bindings to the best-in-class-but-unbearably-slow zopfli encoder... and then there was one crate that caught my eye: libdeflater.

From the description:

Rust bindings to libdeflate. A high-performance library for working with gzip/zlib/deflate data.

Warning: libdeflate is for specialized use-cases. You should use something like flate2 if you want a general-purpose deflate library.

libdeflate is optimal in applications that have all input data up and have a mechanism for chunking large input datasets (e.g. genomic bam files, some object stores, specialized backends, game netcode packets). It has a much simpler API than zlib but can't stream data.

Usually I'm ~~sceptical~~ cautious about "high-performance" claims in READMEs, unless they're backed by reproducible benchmarks as well as explanation on which corners has been cut to achieve such performance, but this description checked all the boxes and made the library hard to pass by.

It seemed particularly intriguing that libdeflate chose to focus on optimising compression (both speed- and ratio-wise) just for a fixed-size data, rather than attempting to replicate the full streaming zlib API.

This matches our use-case (fixed-size image data) perfectly, so I've decided to go ahead and try and integrate it with OxiPNG, and compare with cloudflare-zlib implementation on a native x64. Benchmarking it against the corpus of test files in the OxiPNG repo gave mixed, but promising results.

Check out the spreadsheet for raw numbers, or just the highlights below for the improvement/regression distribution diagrams:

Time difference

IDAT size difference

Total size difference

If we ignore some outliers (mainly tiny files that contribute too much when converted to relative differences), it's clear that in most cases OxiPNG + libdeflate provides both better compression ratio and does it even faster than OxiPNG + SIMD-optimised cloudflare-zlib. Seems promising!

If you look at the code in the PR, you might say that it's not a fair comparison, because libdeflate doesn't provide as many fine-tuning knobs as zlib, so OxiPNG ends up iterating over far fewer iterations with libdeflate than it would otherwise.

You would be correct in saying so, and for other cases (cloudflare-)zlib could still be the right choice, but for our use-case it's the end result that matters, and not the speed of a single iteration. We don't really care about the lost fine-tuning knobs either - instead, we just use the maximum compression level (12 in libdeflate) and as long as it does provide better ratio/speed balance, I'd say we take it.

This integration is now merged upstream in OxiPNG, and it was time to test it on the Squoosh PR again.

One limitation of the WebAssembly target is that we can't yet benefit from the SIMD optimisations, so we lose some of the native libdeflate speed improvements. On the other hand, we couldn't benefit from SIMD in cloudflare-zlib either, so I'd say it's a fair game in the end.

Moreover, unlike other mentioned solutions, libdeflate already provides fallbacks for unsupported platforms, so it's one less thing to worry about that made the integration quite straightforward.

Let's take one more look at the updated numbers for files mentioned in the beginning of the post, but now with OptiPNG vs OxiPNG + Libdeflate integration in Squoosh:

File	optipng	oxipng	optipng time (ms)	oxipng time (ms)
Device screen demo	1.49MB	1.42MB	76909	6991
Google logo (53 colours)	10.3KB	9.37KB	1975	832
Android logo (53 colours)	7.22KB	6.75KB	1191	412
US flag (53 colours)	12.3KB	10.9KB	2841	963
UK flag (53 colours)	3.48KB	3.72KB	2438	1028

Just like in the general test suite, there are some regressions, but usually we get a better ratio, and always a much higher throughput.

What now?

Now that all the PRs are merged upstream, I have more potential optimisations to play with.

One, already mentioned in the beginning of the post, is utilising WebAssembly threads on supported platforms. Even though we are already much faster (up to 11x for the files above) than where we started at, we could do even better by leveraging OxiPNG multithreading support, now that the basic integration is complete and this path is unlocked.

Another potentially interesting idea is to add a mode that would use both libdeflate and zlib in parallel in OxiPNG.

This would eliminate any regressions by always choosing the best encoder possible, but at the cost of eliminating any speed wins as well (since now we have to try even more different encoder+options combinations than we started with).

Also, it still requires getting at least one of the high-level wrappers to compile with both cloudflare-zlib and regular zlib depending on the target platform, to make it work on WebAssembly. All in all, it's not yet clear if this path is worth pursuing, or it's better to find and report specific poorly compressable patterns to libdeflate and hope that they can be improved upstream.

Finally, we could play with upcoming WebAssembly + SIMD support in both libdeflate and cloudflare-zlib to get even faster single-thread compression. There are some potential big upcoming changes to the bytecode, so we wouldn't want to ship code using SIMD to production yet, but nothing stops us from already playing with it in forks / branches by using corresponding intrinsics.

But these are all separate stories for later 🙂 Stay tuned and stay safe!

Github project: cow-utils-rs

2020-02-16T17:02:59Z

A History of HTML Parsing at Cloudflare: Part 1

2019-11-28T00:00:00Z

Github project: asyncify

2019-10-02T10:38:42Z

Github project: lazyhtml

2019-06-20T16:25:08Z

Faster script loading with BinaryAST?

2019-05-17T00:00:00Z

Github project: serde-wasm-bindgen

2019-05-02T15:09:28Z

Github project: astexplorer-syn

2019-03-19T00:11:31Z

Building fast interpreters in Rust

2019-03-04T00:00:00Z

Improving request debugging in Cloudflare Workers

2018-12-28T00:00:00Z

#[derive(Debug)] on serde steroids

2018-07-31T15:00:45Z

In this post I'd like to introduce a serdebug helper which is a drop-in replacement for #[derive(Debug)] with some of the advanced features that serde can provide.

Yes, it's meant to be read as "ser"+"debug" (serialise to Debug) or "serde"+"debug" (Debug with serde) and not the way you thought ;).

Why would I want it?

Often enough, in Rust I've ran into situations where I wanted to make Debug representations more readable by hiding certain fields (like PhantomData or similar markers that don't contain any useful information) or providing custom representation for some of them (which is especially hard in case of third-party types).

Unfortunately, what this usually means is a choice between giving up on auto-derived representation completely and implementing Debug for your types manually, which might be quite verbose for large structures or enums, or wrapping types into intermediate newtype, which is a popular pattern in Rust lands, but not very convenient one (IMO), as it pollutes every usage side and/or requires reimplementing all traits you care about on top of the wrapper (often manually, unless you are lucky and it's easy to #[derive(...)] with existing crates).

When I looked for solutions on crates.io, surely, I found that some of these issues were already tackled by others - for example, you can use debug_stub_derive to provide custom formatting function for certain fields or derivative to also unwrap newtypes or rename fields (the latter also has a number of customisations for other standard traits like Hash, Clone, PartialEq, etc., so definitely check it out if you ever wanted to customise them too).

What bothered me however, is that they still felt not flexible enough, and at the same time I already use serde for most of my types which provides these and many more customisations out of the box, as it has to deal with all sorts of output formats, and usually I'd just want my debug representation to use the same names and fields as I'm already serialising, expect in a more human-readable format that Debug provides.

So then I thought, what is Debug if not just another serialisation format? Moreover, looking at the advanced API, it's easy to see how DebugMap is similar to SerializeMap or DebugTuple to SerializeTuple and so on.

So yes, I've decided to wrap all these std::fmt structures into newtypes that implement serde::ser traits, and then to implement main Serializer interface that would call out to them simulating exact same output that #[derive(Debug)] would normally produce for you with its own auto-generated code, unless you enabled some of the serde-specific attributes to control serialisation behaviour.

How do I use it?

On any types where you already use #[derive(Debug)], you can replace with #[derive(SerDebug)] and add serde-provided #[derive(Serialize)] if you don't already have it.

As mentioned above, by default the output is aimed to exactly match the built-in derive, and that's not very interesting to observe, so instead, let's add a bunch of serde attributes to customise it (usually you won't need that many, this is done purely for demonstration purposes):

#[derive(Serialize, SerDebug)]
pub enum MyEnum {
    // renaming items works as expected
    #[serde(rename = "AAAAAAA!!!")]
    A,

    B(u32),

    C { flag: bool },
}

#[derive(Serialize, SerDebug)]
// so does bulk rename on containers
#[serde(rename_all = "PascalCase")]
pub struct MyStruct {
    number: u32,

    my_enum: Vec<MyEnum>,

    // we might want to hide some items from the output
    #[serde(skip_serializing)]
    hidden: bool,

    // or override serialisation for otherwise verbose wrappers or
    // third-party types that don't implement `Debug` and/or `Serialize`
    #[serde(serialize_with = "custom_serialize")]
    custom_type: CustomType,
}

fn custom_serialize<S: serde::Serializer>(value: &CustomType, ser: S) -> Result<S::Ok, S::Error> {
    use serde::Serialize;

    value.0.serialize(ser)
}

Now, if we try to define a sample MyStruct value

let s = MyStruct {
    number: 42,
    my_enum: vec![MyEnum::A, MyEnum::B(10), MyEnum::C { flag: true }],
    hidden: true,
    custom_type: CustomType(20),
};

and print its debug representation as we normally would

println!("{:#?}", s);

you will see that the output still resembles Debug but has our customisations applied:

MyStruct {
    Number: 42,
    MyEnum: [
        AAAAAAA!!!,
        B(
            10
        ),
        C {
            flag: true
        }
    ],
    CustomType: 20
}

Why you might not want it

serde, especially serde_derive, might be a heavy dependency to add to your project as it sometimes adds non-trivial overhead to compilation times.

On average-sized and larger projects, chances are that you already have it somewhere in your tree and use for other (de)serialisation targets. In that case the difference should be negligible as now you will need to compile just one more custom derive (which you will need anyway if you decided to customise your output with one of the helper crates). However, if you do not use serde yet, you might be better off with derivative helper - I haven't personally compared compilation times, but I do think that it will be faster than all of serde + serde_derive + serdebug combined.

This is it for now, but if you find any bugs or have improvement suggestions which are specific to serdebug and not serde in general, please do report them at github issue tracker.

Github project: xmatch

2018-04-30T19:30:10Z

Github project: ws-proxy

2018-04-13T14:34:45Z

Writing complex macros in Rust: Reverse Polish Notation

2018-01-15T16:25:00Z

Among other interesting features, Rust has a powerful macro system. Unfortunately, even after reading The Book and various tutorials, when it came to trying to implement a macro which involved processing complex lists of different elements, I still struggled to understand how it should be done, and it took some time till I got to that "ding" moment and started misusing macros for everything :) (ok, not everything as in the i-am-using-macros-because-i-dont-want-to-use-functions-and-specify-types-and-lifetimes everything like I've seen some people do, but anywhere it's actually useful)

So, here is my take on describing the principles behind writing such macros. It assumes you have read the Macros section from The Book and are familiar with basic macros definitions and token types.

I'll take a Reverse Polish Notation as an example for this tutorial. It's interesting because it's simple enough, you might be already familiar with it from school, and yet to implement it statically at compile time, you already need to use a recursive macros approach.

Reverse Polish Notation (also called postfix notation) uses a stack for all its operations, so that any operand is pushed onto the stack, and any [binary] operator takes two operands from the stack, evaluates the result and puts it back. So an expression like following:

2 3 + 4 *

translates into:

Put 2 onto the stack.
Put 3 onto the stack.
Take two last values from the stack (3 and 2), apply operator + and put the result (5) back onto the stack.
Put 4 onto the stack.
Take two last values from the stack (4 and 5), apply operator * (4 * 5) and put the result (20) back onto the stack.
End of expression, the single value on the stack is the result (20).

In a more common infix notation, used in math and most modern programming languages, the expression would look like (2 + 3) * 4.

So let's write a macro that would evaluate RPN at compile-time by converting it into an infix notation that Rust understands.

macro_rules! rpn {
  // TODO
}

println!("{}", rpn!(2 3 + 4 *)); // 20

Let's start with pushing numbers onto the stack.

Macros currently don't allow matching literals, and expr won't work for us because it can accidentally match sequence like 2 + 3 ... instead of taking just a single number, so we'll resort to tt - a generic token matcher that matches only one token tree (whether it's a primitive token like literal/identifier/lifetime/etc. or a ()/[]/{}-parenthesized expression containing more tokens):

macro_rules! rpn {
  ($num:tt) => {
    // TODO
  };
}

Now, we'll need a variable for the stack.

Macros can't use real variables, because we want this stack to exist only at compile time. So, instead, the trick is to have a separate token sequence that can be passed around, and so used as kind of an accumulator.

In our case, let's represent it as a comma-separated sequence of expr (since we will be using it not only for simple numbers but also for intermediate infix expressions) and wrap it into brackets to separate from the rest of the input:

macro_rules! rpn {
  ([ $($stack:expr),* ] $num:tt) => {
    // TODO
  };
}

Now, a token sequence is not really a variable - you can't modify it in-place and do something afterwards. Instead, you can create a new copy of this token sequence with necessary modifications, and recursively call same macro again.

If you are coming from functional language background or worked with any library providing immutable data before, both of these approaches - mutating data by creating a modified copy and processing lists with a recursion - are likely already familiar to you:

macro_rules! rpn {
  ([ $($stack:expr),* ] $num:tt) => {
    rpn!([ $num $(, $stack)* ])
  };
}

Now, obviously, the case with just a single number is rather unlikely and not very interesting to us, so we'll need to match anything else after that number as a sequence of zero or more tt tokens, which can be passed to next invocation of our macro for further matching and processing:

macro_rules! rpn {
  ([ $($stack:expr),* ] $num:tt $($rest:tt)*) => {
      rpn!([ $num $(, $stack)* ] $($rest)*)
  };
}

At this point we're still missing operator support. How do we match operators?

If our RPN would be a sequence of tokens that we would want to process in an exactly same way, we could simply use a list like $($token:tt)*. Unfortunately, that wouldn't give us an ability to go through list and either push an operand or apply an operator depending on each token.

The Book says that "macro system does not deal with parse ambiguity at all", and that's true for a single macros branch - we can't match a sequence of numbers followed by an operator like $($num:tt)* + because + is also a valid token and could be matched by the tt group, but this is where recursive macros helps again.

If you have different branches in your macro definition, Rust will try them one by one, so we can put our operator branches before the numeric one and, this way, avoid any conflict:

macro_rules! rpn {
  ([ $($stack:expr),* ] + $($rest:tt)*) => {
    // TODO
  };

  ([ $($stack:expr),* ] - $($rest:tt)*) => {
    // TODO
  };

  ([ $($stack:expr),* ] * $($rest:tt)*) => {
    // TODO
  };

  ([ $($stack:expr),* ] / $($rest:tt)*) => {
    // TODO
  };

  ([ $($stack:expr),* ] $num:tt $($rest:tt)*) => {
    rpn!([ $num $(, $stack)* ] $($rest)*)
  };
}

As I said earlier, operators are applied to the last two numbers on the stack, so we'll need to match them separately, "evaluate" the result (construct a regular infix expression) and put it back:

macro_rules! rpn {
  ([ $b:expr, $a:expr $(, $stack:expr)* ] + $($rest:tt)*) => {
    rpn!([ $a + $b $(, $stack)* ] $($rest)*)
  };

  ([ $b:expr, $a:expr $(, $stack:expr)* ] - $($rest:tt)*) => {
    rpn!([ $a - $b $(, $stack)* ] $($rest)*)
  };

  ([ $b:expr, $a:expr $(, $stack:expr)* ] * $($rest:tt)*) => {
    rpn!([ $a * $b $(,$stack)* ] $($rest)*)
  };

  ([ $b:expr, $a:expr $(, $stack:expr)* ] / $($rest:tt)*) => {
    rpn!([ $a / $b $(,$stack)* ] $($rest)*)
  };

  ([ $($stack:expr),* ] $num:tt $($rest:tt)*) => {
    rpn!([ $num $(, $stack)* ] $($rest)*)
  };
}

I'm not really fan of such obvious repetitions, but, just like with literals, there is no special token type to match operators.

What we can do, however, is add a helper that would be responsible for the evaluation, and delegate any explicit operator branch to it.

In macros, you can't really use an external helper, but the only thing you can be sure about is that your macros is already in scope, so the usual trick is to have a branch in the same macro "marked" with some unique token sequence, and call it recursively like we did in regular branches.

Let's use @op as such marker, and accept any operator via tt inside it (tt would be unambiguous in such context because we'll be passing only operators to this helper).

And the stack does not need to be expanded in each separate branch anymore - since we wrapped it into [] brackets earlier, it can be matched as any another token tree (tt), and then passed into our helper:

macro_rules! rpn {
  (@op [ $b:expr, $a:expr $(, $stack:expr)* ] $op:tt $($rest:tt)*) => {
    rpn!([ $a $op $b $(, $stack)* ] $($rest)*)
  };

  ($stack:tt + $($rest:tt)*) => {
    rpn!(@op $stack + $($rest)*)
  };

  ($stack:tt - $($rest:tt)*) => {
    rpn!(@op $stack - $($rest)*)
  };

  ($stack:tt * $($rest:tt)*) => {
    rpn!(@op $stack * $($rest)*)
  };

  ($stack:tt / $($rest:tt)*) => {
    rpn!(@op $stack / $($rest)*)
  };

  ([ $($stack:expr),* ] $num:tt $($rest:tt)*) => {
    rpn!([ $num $(, $stack)* ] $($rest)*)
  };
}

Now any tokens are processed by corresponding branches, and we need to just handle final case when stack contains a single item, and no more tokens are left:

macro_rules! rpn {
  // ...

  ([ $result:expr ]) => {
    $result
  };
}

At this point, if you invoke this macro with an empty stack and RPN expression, it will already produce a correct result:

Playground

println!("{}", rpn!([] 2 3 + 4 *)); // 20

However, our stack is an implementation detail and we really wouldn't want every consumer to pass an empty stack in, so let's add another catch-all branch in the end that would serve as an entry point and add [] automatically:

Playground

macro_rules! rpn {
  // ...

  ($($tokens:tt)*) => {
    rpn!([] $($tokens)*)
  };
}

println!("{}", rpn!(2 3 + 4 *)); // 20

Our macro even works for more complex expressions, like the one from Wikipedia page about RPN!

println!("{}", rpn!(15 7 1 1 + - / 3 * 2 1 1 + + -)); // 5

Error handling

Now everything seems to work smoothly for correct RPN expressions, but for a macros to be production-ready we need to be sure that it can handle invalid input as well, with a reasonable error message.

First, let's try to insert another number in the middle and see what happens:

println!("{}", rpn!(2 3 7 + 4 *));

Output:

error[E0277]: the trait bound `[{integer}; 2]: std::fmt::Display` is not satisfied
  --> src/main.rs:36:20
   |
36 |     println!("{}", rpn!(2 3 7 + 4 *));
   |                    ^^^^^^^^^^^^^^^^^ `[{integer}; 2]` cannot be formatted with the default formatter; try using `:?` instead if you are using a format string
   |
   = help: the trait `std::fmt::Display` is not implemented for `[{integer}; 2]`
   = note: required by `std::fmt::Display::fmt`

Okay, that definitely doesn't look helpful as it doesn't provide any information relevant to the actual mistake in the expression.

In order to figure out what happened, we will need to debug our macros. For that, we'll use a trace_macros feature (and, like for any other optional compiler feature, you'll need a nightly version of Rust). We don't want to trace println! call, so we'll separate our RPN calculation to a variable:

Playground

#![feature(trace_macros)]

macro_rules! rpn { /* ... */ }

fn main() {
  trace_macros!(true);
  let e = rpn!(2 3 7 + 4 *);
  trace_macros!(false);
  println!("{}", e);
}

In the output we'll now see how our macro is being recursively evaluated step by step:

note: trace_macro
  --> src/main.rs:39:13
   |
39 |     let e = rpn!(2 3 7 + 4 *);
   |             ^^^^^^^^^^^^^^^^^
   |
   = note: expanding `rpn! { 2 3 7 + 4 * }`
   = note: to `rpn ! ( [  ] 2 3 7 + 4 * )`
   = note: expanding `rpn! { [  ] 2 3 7 + 4 * }`
   = note: to `rpn ! ( [ 2 ] 3 7 + 4 * )`
   = note: expanding `rpn! { [ 2 ] 3 7 + 4 * }`
   = note: to `rpn ! ( [ 3 , 2 ] 7 + 4 * )`
   = note: expanding `rpn! { [ 3 , 2 ] 7 + 4 * }`
   = note: to `rpn ! ( [ 7 , 3 , 2 ] + 4 * )`
   = note: expanding `rpn! { [ 7 , 3 , 2 ] + 4 * }`
   = note: to `rpn ! ( @ op [ 7 , 3 , 2 ] + 4 * )`
   = note: expanding `rpn! { @ op [ 7 , 3 , 2 ] + 4 * }`
   = note: to `rpn ! ( [ 3 + 7 , 2 ] 4 * )`
   = note: expanding `rpn! { [ 3 + 7 , 2 ] 4 * }`
   = note: to `rpn ! ( [ 4 , 3 + 7 , 2 ] * )`
   = note: expanding `rpn! { [ 4 , 3 + 7 , 2 ] * }`
   = note: to `rpn ! ( @ op [ 4 , 3 + 7 , 2 ] * )`
   = note: expanding `rpn! { @ op [ 4 , 3 + 7 , 2 ] * }`
   = note: to `rpn ! ( [ 3 + 7 * 4 , 2 ] )`
   = note: expanding `rpn! { [ 3 + 7 * 4 , 2 ] }`
   = note: to `rpn ! ( [  ] [ 3 + 7 * 4 , 2 ] )`
   = note: expanding `rpn! { [  ] [ 3 + 7 * 4 , 2 ] }`
   = note: to `rpn ! ( [ [ 3 + 7 * 4 , 2 ] ] )`
   = note: expanding `rpn! { [ [ 3 + 7 * 4 , 2 ] ] }`
   = note: to `[(3 + 7) * 4, 2]`

If we carefully look through the trace, we'll notice that the problem originates in these steps:

   = note: expanding `rpn! { [ 3 + 7 * 4 , 2 ] }`
   = note: to `rpn ! ( [  ] [ 3 + 7 * 4 , 2 ] )`

Since [ 3 + 7 * 4 , 2 ] was not matched by ([$result:expr]) => ... branch as a final expression, it was caught by our final catch-all ($($tokens:tt)*) => ... branch instead, prepended with an empty stack [] and then the original [ 3 + 7 * 4 , 2 ] was matched by generic $num:tt and pushed onto the stack as a single final value.

In order to prevent this from happening, let's insert another branch between these last two that would match any stack.

It would be hit only when we ran out of tokens, but stack didn't have exactly one final value, so we can treat it as a compile error and produce a more helpful error message using a built-in compile_error! macro.

Note that we can't use format! in this context since it uses runtime APIs to format a string, and instead we'll have to limit ourselves to built-in concat! and stringify! macros to format a message:

Playground

macro_rules! rpn {
  // ...

  ([ $result:expr ]) => {
    $result
  };

  ([ $($stack:expr),* ]) => {
    compile_error!(concat!(
      "Could not find final value for the expression, perhaps you missed an operator? Final stack: ",
      stringify!([ $($stack),* ])
    ))
  };

  ($($tokens:tt)*) => {
    rpn!([] $($tokens)*)
  };
}

The error message is now more meaningful and contains at least some details about current state of evaluation:

error: Could not find final value for the expression, perhaps you missed an operator? Final stack: [ (3 + 7) * 4 , 2 ]
  --> src/main.rs:31:9
   |
31 |         compile_error!(concat!("Could not find final value for the expression, perhaps you missed an operator? Final stack: ", stringify!([$($stack),*])))
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
40 |     println!("{}", rpn!(2 3 7 + 4 *));
   |                    ----------------- in this macro invocation

But what if, instead, we miss some number?

Playground

println!("{}", rpn!(2 3 + *));

Unfortunately, this one is still not too helpful:

error: expected expression, found `@`
  --> src/main.rs:15:14
   |
15 |         rpn!(@op $stack * $($rest)*)
   |              ^
...
40 |     println!("{}", rpn!(2 3 + *));
   |                    ------------- in this macro invocation

If you try to use trace_macros, even it won't expand the stack here for some reason, but, luckily, it's relatively clear what's going on - @op has very specific conditions as to what should be matched (it expects at least two values on the stack), and, when it can't, @ gets matched by the same way-too-greedy $num:tt and pushed onto the stack.

To avoid this, again, we'll add another branch to match anything starting with @op that wasn't matched already, and produce a compile error:

Playground

macro_rules! rpn {
  (@op [ $b:expr, $a:expr $(, $stack:expr)* ] $op:tt $($rest:tt)*) => {
    rpn!([ $a $op $b $(, $stack)* ] $($rest)*)
  };

  (@op $stack:tt $op:tt $($rest:tt)*) => {
    compile_error!(concat!(
      "Could not apply operator `",
      stringify!($op),
      "` to the current stack: ",
      stringify!($stack)
    ))
  };

  // ...
}

Let's try again:

error: Could not apply operator `*` to the current stack: [ 2 + 3 ]
  --> src/main.rs:9:9
   |
9  |         compile_error!(concat!("Could not apply operator ", stringify!($op), " to current stack: ", stringify!($stack)))
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
46 |     println!("{}", rpn!(2 3 + *));
   |                    ------------- in this macro invocation

Much better! Now our macro can evaluate any RPN expression at compile-time, and gracefully handles most common mistakes, so let's call it a day and say it's production-ready :)

There are many more small improvements we could add, but I'd like to leave them outside this demonstration tutorial.

Feel free to let me know if this has been useful and/or what topics you'd like to see better covered on Twitter!

Conditional enum variants in Rust

2017-12-26T23:30:00Z

Recently, I had to implement a set of types (AST nodes) in Rust that would be generic over boolean flags. More specifically, they had to be statically dependant on compile-time boolean parameters, where instances of the same type with false and true values of such parameter would be incompatible, as these flags would not only determine behaviour of impl but also affect e.g. which branches of enum are allowed. Pseudo code sample for demonstration purposes:

enum MyEnum<AllowB: bool> {
  A,
  B if AllowB, // this won't be allowed on MyEnum<false>
}

impl<AllowB: bool> MyEnum<AllowB> {
  fn is_a(&self) -> bool {
    match *self {
      MyEnum::A => true,
      _ => false,
    }
  }
}

impl MyEnum<true> {
  // this won't be allowed on MyEnum<false>, giving a helpful compile-time error
  fn is_b(&self) -> bool {
    match *self {
      MyEnum::B => true,
      _ => false,
    }
  }
}

The first thing that comes to mind is the RFC #2000 for const generics which would allow parametrising types over any compile-time constant values. Unfortunately, while the RFC has been accepted, it's not implemented even in nightly yet, and some implementation details are still in flux. Apart from that, it's unlikely to help with the enum variants requirement, even though it does ensure the other two.

What can we do instead? Well, why don't we try and implement booleans using the Rust type system? Then we will be able to pass them around, check and manipulate purely at compile-time.

First of all, let's define trait for our Bool which would determine what we can do with such boolean flags.

For the beginning, let's say we want to have just conversion to actual runtime boolean value. This will be done through an associated constant as our boolean variants are statically determined at compile-time.

pub trait Bool {
  const VALUE: bool;
}

Now, let's implement actual variants of our Bool as zero-size types implementing this trait:

pub struct True;

impl Bool for True {
  const VALUE: bool = true;
}

pub struct False;

impl Bool for False {
  const VALUE: bool = false;
}

Great! Now we can use these flags to implement our enum. Note that if someFlag is still not a valid Rust syntax, so instead we just store the flag inside of the variant payload:

pub enum MyEnum<AllowB: Bool> {
  A,
  B(AllowB), // TODO: disallow on MyEnum<False>
}

impl<AllowB: Bool> MyEnum<AllowB> {
  pub fn is_a(&self) -> bool {
    match *self {
      MyEnum::A => true,
      _ => false,
    }
  }
}

impl MyEnum<True> {
  // this won't be allowed on MyEnum<False>, giving a helpful compile-time error
  pub fn is_b(&self) -> bool {
    match *self {
      MyEnum::B(_) => true,
      _ => false,
    }
  }
}

If we try it, we can see that impl blocks work as expected:

[playground]

let e: MyEnum<False> = MyEnum::A;
println!("{}", e.is_a()); // prints "true"
println!("{}", e.is_b()); // error[E0599]: no method named `is_b` found for type `MyEnum<False>` in the current scope

Great! One issue remains though: while these flags are visible in the generated docs and should help the developer using our API understand that they should never use the B variant when AllowB is false-ish, this is not a strong enough guarantee which might still lead to accidental mistakes:

[playground]

let e: MyEnum<False> = MyEnum::B(False); // compiles while it shouldn't

Apart from that, the compiler doesn't know either that B variant simply can't exist on MyEnum<False> and so can't provide better optimisations. If we look at the generated code for, say, MyEnum<False>::is_a, we can see that it still tries to check the tag at the beginning of our enum at runtime even though it should be able to statically determine the result as true since only variant A can exist on such type!

[disassembly]

example::test_is_a:
  xor dil, 1
  mov eax, edi
  ret

To solve these two issues, we can use a trick with so-called uninhabited type:

pub enum Void {}

The trick is, while this type definition is perfectly valid in Rust, it has no variants, which means that you can't actually construct values of this type (unlike in C, Rust enums have much stronger guarantees and are not just glorified integer constants).

Moreover, such type "infects" all its usages - since you can't construct values of this type, any type containing Void becomes uninhabited too, and, in turn, any place where you're dealing with values of any such type, becomes unreachable and can be optimised out as dead code.

This is often used as a placeholder for generic params which we know can't be ever instantiated. For example, it's used by standard library as a ParseError for FromStr implementation on String - indeed, since string can be always converted to string, there is no way an error could be produced by this operation, which means we can provide a placeholder for Err part of Result which can't be actually constructed, and compiler will know that this operation is infallible.

Let's see how we can use this property for our original task. Assuming that our desired flag is always true-ish (and if not, we can implement unary not operator), we can make False an uninhabited type, which in turn will make any variants containing it uninhabited too. This might sound complicated, but the change is as simple as:

-pub struct False;
+pub enum False {}

Now, if we try this with our last example, we will immediately get a compile error as False is an enum now:

[playground]

error[E0423]: expected value, found enum `False`
  --> src/main.rs:42:36
   |
42 |   let _e: MyEnum<True> = MyEnum::B(False);
   |                                    ^^^^^ not a value

And, since it has no variants, there is nothing you can put on the right-hand side of this assignment to make it work. Neither you can assign MyEnum::B(True) to MyEnum<False> variable:

[playground]

42 |   let _e: MyEnum<False> = MyEnum::B(True);
   |                                     ^^^^ expected enum `False`, found struct `True`
   |
   = note: expected type `False`
              found type `True`

However, if you set the AllowB to True, you can start accepting MyEnum::B as expected:

[playground]

let _e: MyEnum<True> = MyEnum::B(True); // compiles

Apart from type safety, the advanced optimisations are now possible too. In particular, MyEnum<False>::is_a now just returns constant true as we wanted:

[disassembly]

example::test_is_a:
  mov al, 1
  ret

So we can see everything works as expected, even if with the little annoyance of having to store the True flag on the variant.

If we wish, we could now implement all the logical operators on top of these values (after all, Rust's type system is Turing-complete), but I wouldn't go that far in this post.

Hope this has been useful. Let me know on Twitter if you have any thoughts about the approach.

Github project: derex

2017-09-26T10:56:15Z

Github project: emnapi

2017-06-04T14:18:06Z

Parsing JavaScript using SpiderMonkey from Rust

2017-06-03T14:21:00Z

View on GitHub

SpiderMonkey AST is largely compatible with ESTree, and can be made fully compatible by overriding AST builders via options.

Additionally, SpiderMonkey, unlike all the other JavaScript engines, exposes its parser via a public API - Reflect.parse - available even from their JavaScript console.

This repo is a prototype to get ESTree AST in Rust by literally embedding SpiderMonkey and invoking JavaScript that calls Reflect.parse and passes result back in JSON format, which is easily consumable by Rust side or can be passed further.

As turned out, (surprise!) speed of such approach is not very appealing, but it might work for prototyping JavaScript tooling in Rust until better parsers are available.

Reducing test cases for JavaScript tooling

2017-05-10T08:15:17Z

At some point, I've performed testing of one Go-based streaming JavaScript minifier for compatibility with real-world code.

I've decided to run it against CDNjs repo (https://cdnjs.com/libraries) as a source of all the possible popular libraries, which makes it a decent test suite for JavaScript/CSS tooling.

After narrowing the list to just JavaScript libraries, then only to latest version of each, and then to those that don't end with .min.js to avoid obviously pre-minified ones (and speed up testing), I've got 17,547 JavaScript files with a total weight of 771 MB.

Then, I've run them all against latest minify binary and parsed with a full-featured (non-streaming) ES5 parser to compare ASTs before and after transformation to detect semantic mismatches or syntactically broken files.

You can see detailed results and findings on the original issue, but I'd like to stop on one interesting case.

Minification of moxie-html4.js from webshim library and further AST (format-agnostic) comparison revealed the following changes:

@@ -2736,12 +2736,12 @@
                     4: 'Flash',
                     9: 'Fine weather',
                     10: 'Cloudy weather',
                     11: 'Shade',
-                    12: 'Daylight fluorescent (D 5700 - 7100K)',
-                    13: 'Day white fluorescent (N 4600 -5400K)',
-                    14: 'Cool white fluorescent (W 3900 - 4500K)',
-                    15: 'White fluorescent (WW 3200 - 3700K)',
+                    12: 'Daylight fluorescent(D 5700-7100K)',
+                    13: 'Day white fluorescent(N 4600-5400K)',
+                    14: 'Cool white fluorescent(W 3900-4500K)',
+                    15: 'White fluorescent(WW 3200-3700K)',
                     17: 'Standard light A',
                     18: 'Standard light B',
                     19: 'Standard light C',
                     20: 'D55',
...

Clearly, something went wrong that caused significant spaces in strings to disappear.

But how do we figure out what exactly caused that, given that the rest of the file was correct? We can't just manually remove lines here and there in 74 KB minified file expecting that we will be lucky and reduce it quickly (believe me, I tried that first :) ).

Luckily, there is an automatic solution for that: creduce

Wait, you might say, but it's README says it's a reducer for C programs. Yeah, it happens to be primarily designed for C, but, by its nature, it can also work on JavaScript, Rust and any other C-like languages, even if they have custom syntax.

Basically, this tool tries to remove blocks ({ ... }), expressions (identifiers, numbers, parenthesized expressions ( ... ), etc.), comments, pieces of strings etc. as long as provided script still fails (erm, passes), resulting in a minimal reproducible test case.

First of all, we should store the original file for the base sample:

$ curl https://cdnjs.cloudflare.com/ajax/libs/webshim/1.16.0/minified/shims/moxie/js/moxie-html4.js -o html4.js

Now, creduce expects an "interestingness test" - a script that returns 0 (succeeds) by default on our example, and fails otherwise. That means, we should fail when resulting code either doesn't parse or it's structurally equal to the original.

Let's install, for example, Esprima to parse scripts and use it for a simple one-off script that will perform the actual comparison:

#!/usr/bin/node

const { readFileSync } = require("fs");
const { parse } = require("esprima");
const { notDeepEqual } = require("assert");

let [ast1, ast2] = process.argv
  .slice(2) // skip node executable and script filename
  .map((file) => readFileSync(file, "utf-8")) // read given files as strings
  .map((code) => parse(code)); // parse into ASTs

notDeepEqual(ast1, ast2); // ensure they're still not equal

Assume we saved it as compare.js.

Now, we need to call it right after the minifier, so let's write a wrapper shell script (compare.sh) for that (we could do that from Node as well, but this might be a bit easier):

#!/usr/bin/env bash
~/minify html4.js -o html4.min.js && ~/compare.js html4.js html4.min.js

Make sure you're using absolute paths for custom executables. creduce will be generating reduced test cases with same file name as original in temporary folders (in order to parallelize reduction) and so you can't expect your script to be caled from the same working directory as you're launching creduce in.

Time to ensure that it returns exit code 0 by invoking ./compare.sh manually, and ready to go:

$ ./compare.sh
$ echo $?
0
$ creduce compare.sh html4.js

Now, after quite some time of waiting and log lines, creduce ends up with this JavaScript in html4.js:

( function (  )
{
    try
    {
        if ( 0 ) /, /
        }
    catch ( e )
    {
    }
}
)
( "moxie/runtime/html5/image/ExifParsersRGBUncalibratedUnknownAverageCenterWeightedAverageSpotMultiSpotPatternPartialOtherDaylightFliorescentTungstenFlashFine
                                                                                                                                                   longitude" )

This is much better than original, and, while there are few more small things we could still reduce manually, it already reveals the problem - indeed, it's the most common problem in JavaScript parsing where context affects tokenizations and / after ) can be parsed either as a division or a regular expression depending on tokens before the matching ).

Depending on that, it's either safe to remove space after / in if ( 0 ) /, / example or not - and in the given failure the tested minifier removes it despite this construct being, in fact, a regular expression.

As you can see, creduce is quite a powerful tool for manual reductions before trying to fix or report issues to language tooling - where it's C, Rust, JavaScript or, who knows, CSS.

Emscripten + Rust: how to interop with real JavaScript

2017-05-06T17:43:27Z

My slides from RustFest 2017:

https://www.slideshare.net/RReverser/rust-javascript

UPD: Video is up!

https://youtu.be/zxIbTfsOJZE

Github project: override-hosts

2017-04-21T15:03:31Z

Github project: esmod

2017-03-14T21:29:56Z

Github project: serde-xml-rs

2017-02-16T16:54:58Z

How we brought HTTPS Everywhere to the cloud

2016-09-24T00:00:00Z

Tabs vs spaces: the ultimate argument

2016-04-19T08:11:48Z

Did you know that using specific code style can affect your JavaScript performance? Gave a 1.5x performance boost in my case ;)

That beautiful moment when you change indentation from spaces to tabs and your function suddenly becomes short enough to get inlined by V8. — Ingvar Stepanyan (@RReverser) April 18, 2016

Github project: buffer-utf8-length

2015-10-26T14:51:02Z

GOTOs in JavaScript are now possible

2015-10-20T14:08:50Z

And I'm back with my crazy experiments.

Currently I'm actively playing with Ragel - state machine compiler that generates highly optimized code for streaming lexers, scanners etc. in a set of languages (C, Java, Ruby, C#, ...). Unfortunately, it didn't have support for JavaScript, and I recently implemented it (likely to be included in Ragel 7.0, but it's anyway worth dedicated article).

Now, Ragel can generate code with various levels of optimizations. The least optimized one uses arrays of indices to search through using binary search and jump between, and the more optimized ones use goto instruction for quick jumps directly between states.

For languages such as C or Assembler this works great and generates proper jumps/gotos. However higher-level languages (Java, C#, Ruby, ..., and of course JavaScript) don't have any instruction like that due to "dirt" associated with this operator that discourages it's use for handwritten code - simply because it becomes unmaintainable. But this doesn't matter when our code is anyway generated from other language and result doesn't have to be comprehensible.

JavaScript has a long history of compilation and transpiling from other langs (and across versions of itself), and yet it doesn't have operator that could be used when generating direct jumps. So what's the solution? Of course, to implement own extension to language with the most EVIL feature ever!

And here we go: I forked regenerator - transpiler for resumable generator and async functions. It already had all the needed state analysis and expansion which are easy to reuse, and all that you need is to add new instructions and tell it where they might break code into pieces, to which places you can jump and so on.

For goto statements theirselves I decided to simplify task and not to introduce new syntax, but reuse existing but rare one - labels - so that you can write:

function f() {
  goto: x;
  console.log("do not enter");
  z: console.log(3);
  return 4;
  y: console.log(2);
  goto: z;
  x: console.log(1);
  goto: y;
}

console.log("Let's do some evil!");
console.log("Result:", f());

And you get in console:

Let's do some evil!
1
2
3
Result: 4

You can play with my fork here: js-goto-transpiler.

DISCLAIMER: I'm not responsible if you decide to use this feature in production and will be fired for using it :)

ES6 modules are dead, long live C preprocessor!

2015-08-13T16:40:35Z

Did you know you could use C preprocessor even for JavaScript files? (yes, it doesn't care about syntax that much)

Say, we want to implement C-like "modules" (inclusions) in JavaScript.

Create first file 1.js (filenames don't matter) like this:

#ifndef HELLO_1
#define HELLO_1

console.log("hello");

#endif

(#ifndef/#define is standard C practice for avoiding double-inclusion)

Then, create 2.js:

#ifndef HELLO_2
#define HELLO_2

#include "1.js"
console.log("world");

#endif

And now you can bundle them into one file with:

cpp -P -Wundef -nostdinc -C 2.js

(here we ask cpp just to preprocess our file with basic warning and preserving comments; also we ask not to search standard C library for includes)

And as a result you would get:

console.log("hello");
console.log("world");

Note that you can even define and use full-featured macroses inside of your JavaScript - just #define them and ready to go!

P.S. Yes this is a crazy hack (as most of other stuff in my blog), and there are more appropriate tools for transforming JavaScript, but why not to try if we can? :)

UPD [17.08.2015]: Turns out it's not as crazy experiment as I initially thought. Ryan Grove on Twitter just mentioned that they actually used this technique at Yahoo Search in 2007:

@RReverser Believe it or not, we used this technique in our production JS workflow at Y! Search ~2007. Worked well. http://t.co/HGSFfU1W1u > — Ryan Grove (@yaypie) August 16, 2015

ECMAScript Syntax Graph

2015-08-09T22:12:05Z

Ever wondered how ECMAScript syntax spec would look like if represented as a graph? Wonder no more - I just built one for fun!

Here it is in SVG:

If your browser doesn't support SVG, you can view PNG version instead (but be careful on metered connections - it's 3.4 MB).

Note: for simplicity literals and exclusion relations ("but not" / "but not one of") were removed from the graph - it's already too complex IMO :)

ECMAScript decorators on functions

2015-08-08T00:49:14Z

This one particular question is being often raised around the usage of the new ECMAScript decorators proposal:

why can't I use it for regular function declarations?

Well, the answer is simple: hoisting :(

Since JS hoists up function declarations, those expressions that you use inside of decorators, depending on proposed "solution", might get hoisted up too, and thus break execution flow, or prevent function from hoisting, and thus break backwards compatibility. See Sebastian McKenzie's answer here for a complete example and explanation. (This is not an issue for classes since those are specified as non-hoistable)

Now, let's switch to a more interesting question: how can we workaround this?

Workaround #1 (just call it)

Well, first of all, decorators are just function calls. That is, whenever you do

@decorator
class A {}

that is equivalent of

class A {}
A = decorator(A) || A;

So, apart from || part, imaginable decorator for funtions

@decorator
function f() {}

could be written in old good JavaScript as simply as

var f = decorator(function f() {});

(note that this also prevents ambiguity caused by hoisting since in this case variable will be hoisted, but expression won't)

The case where it makes a difference is when you have a bunch of decorators that you want to compose onto the original definition (class, method, function). No-one really wants to write things like

var f = ...(((((...reportErrors(customReporter, measureTime(debounce(function f() {}, 1000)))...)))))...; // whatever

Workaround #2 (runtime - boring but works)

Just use some fancy runtime API that would allow you to pass a bunch of decorators that would be wrapped around your function. For example, Lodash can definitely do this, and I'm pretty sure there is a lot of others around. Again, this works in old good pre-ES6 JavaScript.

Workaround #3 (hacky but with true decorators)

Since decorators can be defined on object methods, we can use a simple trick: define decorator on the method and then extract the function back:

function debounce(time) {
  return function (target, key, desc) {
    var fn = desc.value,
      lastStart = 0;
    desc.value = function () {
      var now = Date.now();
      if (now - lastStart >= time) {
        lastStart = now;
        return fn.apply(this, arguments);
      }
    };
  };
}

function measureTime(target, key, desc) {
  var fn = desc.value,
    { name } = fn;
  desc.value = function () {
    console.time(name);
    var result = fn.apply(this, arguments);
    console.timeEnd(name);
    return result;
  };
}

var myFn = {
  @debounce(200)
  @measureTime
  func(n) {
    console.log(`myFn(${n})`);
  },
}.func;

myFn(1);
myFn(2);
myFn(3);

setTimeout(() => myFn(4), 500);

See it in action in Babel REPL

Workaround #4 (syntactic and my favourite)

So, what we can actually do here, is use another composable operator similar to pipe operator from functional languages (F#, Elixir etc.) - it's a bind operator (NOTE: this operator wasn't proposed to TC39 yet and is implemented by Babel purely for experimental and feedback purposes)

Here's how it works when used as a function call:

context::fn1("a")::fn2("b")::fn3("c");

turns into

fn3.call(fn2.call(fn1.call(context, "a"), "b"), "c");

Beautiful, isn't it?

So, why don't we just put original function itself as a context for operator? In that case, we can decorate it with as much decorators as we wish in a truly composable way!

Here's a simple example of how such pseudo-decorators can be implemented and used:

function debounce(time) {
  var fn = this,
    lastStart = 0;
  return function () {
    var now = Date.now();
    if (now - lastStart >= time) {
      lastStart = now;
      return fn.apply(this, arguments);
    }
  };
}

function measureTime() {
  var fn = this,
    { name } = this;
  return function () {
    console.time(name);
    var result = fn.apply(this, arguments);
    console.timeEnd(name);
    return result;
  };
}

var myFn = ((n) => console.log(`myFn(${n})`))::measureTime()::debounce(200);

myFn(1);
myFn(2);
myFn(3);

setTimeout(() => myFn(4), 500);

See it in action in Babel REPL

UPD: On Twitter I've got a response with another beautiful example of decorator that can be used with :: syntax - and it's proper currying in JavaScript!

That's it! Feel free to ask any questions in comments.

Github project: typescript-estree

2015-07-31T08:41:41Z

Github project: babel-plugin-uglify

2015-07-30T00:01:33Z

Started work on ES6 -> asm.js conversion

2015-07-12T07:41:27Z

View on GitHub

I know this sounds crazy, but I always wondered why do we always give out such a speedup technique as asm.js (it speeds up code in all the modern browsers!) to developers from other languages if we can use it for own code in JavaScript as well?

The answer is that handwriting asm.js is tricky due to limited set of operations and complex rules for satisfying asm.js. So I decided to start this side project. It's goal is to compile statically typed ES6 (Flow-flavoured) module into asm.js while preserving ES6 module interface. So it's still a subset of JavaScript, but much more suitable for hand-writing.

You don't need to use C/C++ -> Emscripten -> asm.js for heavy calculation anymore - just create module, drop Babel with this plugin and you're ready to go!

Currently supported:

ES6 import and named export
automatic var, let, const extraction and conversion
automatic function layout
function parameter annotations
function return type annotation
assignment with automatic type conversion
Flow type cast into asm.js type cast
automatic asm.js imports for stdlib and foreign references
automatic program layout
automatic wrapping into asm.js module with initialize for passing own heap

TODO:

bug fixing
global variable support
string support (literals are already converted to IDs, need to support operations)
better asm.js<->normal code communication
limited object and array support
sky is the limit!

Github project: proxyfill

2015-06-11T15:22:47Z

Github project: better-log

2015-05-22T09:54:26Z

[Potentially ES7] function bind syntax in Babel

2015-05-16T08:01:59Z

View original post

Babel 5.4 was just released and with it comes support for a new
experimental ES7 syntax
proposed by Kevin Smith (@zenparsing) and
implemented in Babel by Ingvar Stepanyan
(@RReverser).

Warning: This syntax is highly experimental and you should not use
it for anything serious (yet). If you do use this syntax, please
provide feedback on GitHub.

The function bind syntax introduces a new operator :: which performs function
binding and method extraction.

Virtual Methods

Using an iterator library implemented as a module of "virtual methods":

/* ES7 */
import { map, takeWhile, forEach } from "iterlib";

getPlayers()
    ::map((x) => x.character())
    ::takeWhile((x) => x.strength > 100)
    ::forEach((x) => console.log(x));

/* ES6 */
import { map, takeWhile, forEach } from "iterlib";

let _val;
_val = getPlayers();
_val = map.call(_val, (x) => x.character());
_val = takeWhile.call(val, (x) => x.strength > 100);
_val = forEach.call(vall, (x) => console.log(x));

Note: Babel's output
looks different than this in order to be more concise.

Using a jquery-like library of virtual methods:

/* ES7 */
// Create bindings for just the methods that we need
let { find, html } = jake;

// Find all the divs with class="myClass", then get all of the
// "p"s and replace their content.
document.querySelectorAll("div.myClass")::find("p")::html("hahaha");

/* ES6 */
let _val;
_val = document.querySelectorAll("div.myClass");
_val = find.call(_val, "p");
_val = html.call(_val, "hahaha");

Method Extraction

Using method extraction to print the eventual value of a promise to the console:

/* ES7 */
Promise.resolve(123).then(::console.log);

/* ES6 */
// Which could be written in ES6 as:
Promise.resolve(123).then(console.log.bind(console));

Using method extraction to call an object method when a DOM event occurs:

/* ES7 */
$(".some-link").on("click", ::view.reset);

/* ES6 */
$(".some-link").on("click", view.reset.bind(view));

Note: You can read more about this syntax in the
Function Bind Syntax proposal.

Usage

Enable by stage:

$ babel --stage 0

Enable by transformer:

$ babel --optional es7.functionBind

The function bind syntax will only make it into ES7 if there is enough interest.
If you would like to see this syntax make it in, please give it a star on
GitHub and provide any
feedback you have on GitHub issues.

Special thanks to Ingvar Stepanyan (@RReverser)
for the implementation in Babel.

— The Babel team

Quickly view Wi-Fi password under OSX and Windows

2015-02-15T18:52:29Z

Today I've found this neat tool called wifi-password by @rauchg and figured out that it's extremely useful. Just look at this:

Simple, isn't? And README says it:

People ask you for the Wi-Fi password. Answer quickly.

So I wanted to answer quickly, too. ~~Unfortunately~~ I'm a Windows user and so had to create own analog for PowerShell:

So now you just:

1. Install it

First of all, install PsGet if you don't have it yet - it's awesome module manager for PowerShell:

(New-Object Net.WebClient).DownloadString("http://psget.net/GetPsGet.ps1") | iex

Now, you can easily install module itself:

Install-Module WiFi-Password

2. Use it:

To get the password for the WiFi you're currently logged onto:

Show-WiFiPassword

or just

wifi-password

To get it for a specific SSID:

$ wifi-password <ssid>

To list all the stored WiFi networks:

Select-WiFi

3. PROFIT!

Tail recursion and 6to5

2015-02-08T14:06:48Z

Just added tail call optimization to 6to5, so it became the first ES6 implementation across all the browsers and transpilers to have this feature:

Unfortunately, generic tail call optimization is still too complex thing as for transpiler level and it introduces significant overhead compared to simple calls, so should be better done on browser level. However, most use cases for TCO involve recursion, where function returns result of itself, but with different arguments (and possibly context), which is perfectly covered by this implementation.

There is still work that can (and eventually will) be done in order to beat performance of other possible solutions by manually resetting state and converting recursion into regular loop, but it's another story.

UPDATE: As per 3.6.0, some investigations were made that allowed to increase performance a bit:

As you can see, all the previous solutions (including hand-written non-recursive variants) were beaten by new heuristics which analyze your code and heavily optimize the most common cases (simple recursion that doesn't use arguments variable nor default/rest parameters inside).

Gitty - unofficial Gitter client for WinRT

2015-01-09T17:53:15Z

Gitter is a great real-time messaging service built specifically to help teams communicate better. It provides free public chat rooms for communities and open source projects and private chat rooms for teams and businesses. All of the rooms are based on Github repos, so you can easily get and use one for your favourite project.

The only downside for me was absence of native app for Windows RT, and always keeping a dedicated tab in browser and switching to it whenever you get message is not so convenient. So I did what any developer does in such situation - created one.

Fortunately, Gitter provides pretty convenient API for all the functions it provides in it's UI: http://developer.gitter.im/. Unfortunately, some parts of it are not documented, but team guys are very responsive and even fixed some pretty specific bugs in real-time, so I could continue development with almost no interruptions!

I'll probably write a bit more about complex implementation details in separate articles, but for now - you can find initial release of the app in the Store and I hope you enjoy it:

http://apps.microsoft.com/windows/en-us/app/gitty/16998cba-b1d7-4d82-a8d0-131a40009168

Github project: Bayeux.NET

2014-12-17T01:34:05Z

Github project: github-editorconfig

2014-10-22T21:00:10Z

Github project: gulp-wrap-js

2014-10-17T09:48:16Z

Github project: acorn-csp

2014-09-06T12:28:04Z

aster - AST-based code builder

2014-08-28T13:37:40Z

View on GitHub

What's that

aster is reactive builder specialized for code processing and transformations. It's built with debugging in mind and makes building JavaScript code more reliable and faster.

You may want to check out presentation below:

Why one more

Source maps are a great invention that is meant to simplify life by allowing developers to debug the original code (the one that they actually write, whatever language it is) on production.

However, using them is pretty hard in any of existing build systems whenever you have at least one plugin in your build pipeline - and you most likely do - that doesn't support emitting them or even consuming from previous step; some plugins even treat code as simple strings discarding all it's inner logic and structure.

Your code isn't just a string. It has a soul and rich inner world and aster is built to treat it like that. As result, it provides complex yet easy and fast transformations that are transparent for browser debugger out of the box.

You can think of aster for JS as of rework for CSS.

But I like X builder! Should I throw it out?

Definitely no! aster doesn't try to fight your favorite build system. It has only one specific area that it's exceptionally good at - code processing. Everything else (CSS, images, publishing to CDN, etc.) is left for generic builders, and you can use them together.

Currently there are following bindings:

grunt-aster - binding for Grunt JavaScript Task Runner.
gulp-aster - binding for Gulp streaming build system.
...more to come!

If you wish, you can define aster pipeline as custom task in any existing build system on your own since aster uses RxJS under the hood, which is interoperable with events, streams, promises, callbacks and any other asynchronous primitives and patterns out of the box.

API

aster is completely modular and main package is just a centralized API wrapper for core parts published as separate modules (check out their documentations for details):

aster.src - Single-pass source files reader.
aster.watch - Continuous source files reader.
aster.dest - File writer.
aster.runner - Build pipeline runner with built-in logger.

Usage

First, install aster as a development dependency:

npm install --save-dev aster

Then, create build script and use it. Example (requires omitted):

aster
    .watch(["src/**/*.js", "src/**/*.coffee", "src/**/*.jsx"])
    .throttle(500)
    .map(
        changed(function (src) {
            return src.map(
                equery({
                    "if ($cond) return $expr1; else return $expr2;":
                        "return <%= cond %> ? <%= expr1 %> : <%= expr2 %>"
                })
            );
        })
    )
    .map(concat("built.js"))
    .map(umd({ exports: "superLib" }))
    .map(aster.dest("dist", { sourceMap: true }))
    .subscribe(aster.runner);

aster doesn't provide task runner - npm is already good one, and we don't want to create one more bicycle. You can simply define every needed task as separate script, or use aster as part of exising builder as mentioned before.

License

MIT License

Github project: TSX

2014-08-10T15:08:24Z

Using Mozilla AST with UglifyJS

2014-08-04T12:23:04Z

Hi there.

I'm continuing my work on tools for manipulating SpiderMonkey AST - as creating new as well as improving existing ones.

This time it was UglifyJS.

It's great JavaScript minifier, but what always bothered me (and other developers) was incompatibility of it's internal AST representation and Mozilla format, and so impossibility of combining it with other tools without dirty and slow hacks like:

Transform Mozilla AST to UglifyJS AST (thanks Mihail, we already got method for this).
Compress, mangle etc. it with UglifyJS API.
Generate code and source map with UglifyJS API.
Parse generated code with either esprima or acorn.
Translate generated source map onto existing AST with sourcemap-to-ast helper.

(I had to use this hackchain for aster-uglify)

So I've decided to add back-conversion from UglifyJS to SpiderMonkey AST format, and it's already landed in UglifyJS!

Now you can use UglifyJS as any other intermediate tool for transforming JavaScript ASTs using it's native API.

Example:

// Parsing
var ast = acorn.parse(code, {
    locations: true,
    ranges: true
});

// Any AST pre-transformations here
ast = preTransform(ast);

// Conversion
var uAST = UglifyJS.AST_Node.from_mozilla_ast(ast);

// Compression
var compressor = UglifyJS.Compressor(options);
uAST.figure_out_scope();
uAST = uAST.transform(compressor);

// Mangling (optional)
uAST.figure_out_scope();
uAST.compute_char_frequency();
uAST.mangle_names();

// Back-conversion
ast = uAST.to_mozilla_ast();

// Any AST post-transformations here
ast = postTransform(ast);

// Generating output code and source map
var output = escodegen.generate(ast, {
    sourceMapWithCode: true,
    sourceMap: true
});

// Done!

UPDATE

In Twitter, I was asked about speed of such conversion, so decided to show timings for above example executed against jQuery v1.11.1 development source without pre-/post-transformations:

Parsing:             175ms   (5%)
Conversion:          148ms   (4%)
Compression:         2155ms  (56%)
Mangling:            502ms   (13%)
Back-conversion:     48ms    (1%)
Generating output:   829ms   (21%)

Or, in grouped view:

I/O             1004ms  (26%)
Conversions     196ms   (5%)
Compressions    2657ms  (69%)

So, as you can see, speed of conversions has negligible performance impact.

Github project: jsx-ast

2014-07-29T11:33:24Z

Added ES6 support to Acorn

2014-07-29T10:16:37Z

Worked on and added following ES6 features support to Acorn - small, fast JavaScript parser:

Unicode Code Point Escape Sequences (\u{ABCDEF}).
Numeric Literals (octal, binary).
Arrow Functions.
Spread Element.
Classes.
Destructuring assignments.
Computed object properties.
Object methods.
Shorthand object properties.
for (... of ...) loops.
Array Comprehensions.
Modules. (newest specs)
Template Strings.
Generators.
Generator Comprehensions.

There are still some pending discussions with guys from Mozilla & Esprima on new AST types & properties used for new syntax features:

Stay tuned - more to come!

Github project: jsx-transpiler

2014-07-28T20:14:33Z

A few more words on Acorn-JSX

2014-07-27T20:59:10Z

Initially this was part of my email response to Lou Husson after mentioning Acorn-JSX in Facebook React Community Round-up #20 and reaching me out, but decided that someone else might find it useful too.

UPD (June 12, 2015): Facebook deprecates own parsing and transformation tools in favor of Acorn, Acorn-JSX and Babel. Looks like I'll need to be more careful about updates starting from now 😜

As for today, I’ve developed complete ES6 support for Acorn (as far as it can be complete in such a dynamic world; anyway it contains even newest specs than Esprima supports). So I’ve merged it to Acorn-JSX; also I’ve added spread attribute support (<div {…attr} />), took all the tests from official parser, adapted them for Acorn test runner - and it passes them all. So caveat about different syntax level supported in my and official parser was removed, and “official parser is maintained by authors of React.js itself, so it's recommended to be used in production” was left as the only reason for not using it.

Everything else works in the same way. Just 2x faster.

Also wanted to notice that it’s not the only tool for JSX AST I’ve worked on, so maybe you find following one useful as well. Earlier from unworking recast-based fork I’ve developed very simple JSX transpiler based on official Esprima-FB parser, own JSX-enabling drop-in for Estraverse and Escodegen AST tools. It allows you to either 1) parse JSX to AST with attached comments (supported in official Esprima, but not in Esprima-FB), 2) convert it directly to code + source map or 3) use as browserify transformer plugin.

Stay tuned for updates about new JS(X) AST tools!

Github project: serialize-js

2014-07-22T10:35:57Z

Github project: biscript

2014-07-21T18:12:21Z

Github project: estraverse-fb

2014-05-31T19:09:15Z

Github project: gh2ost

2014-05-14T00:51:24Z

Github project: stack-displayname

2014-05-05T15:19:32Z

Github project: when-traverse

2014-04-17T17:27:17Z

Github project: apartjs

2014-04-11T21:11:54Z

Github project: promise-array

2014-03-12T13:17:41Z

Github project: winremote

2013-12-10T16:30:41Z

Github project: grunt-pure-cjs

2013-12-08T11:10:26Z

Github project: pure-cjs

2013-12-03T23:37:46Z

Creating own language made easy

2013-12-02T03:33:20Z

Slides from my talk from jQuery Bulgaria 2013:

https://www.slideshare.net/RReverser/creating-own-language-made-easy

Github project: compile2js

2013-11-27T18:57:42Z

HTTP Live Streaming JavaScript player

2013-11-25T19:08:02Z

I've created Apple HTTP Live Streaming JavaScript player created by performing realtime conversion of MPEG-TS video chunks to MPEG-4 in parallel thread using Web Worker and playing them one by one in main thread.

How does it work?

Conversion is done using jBinary library with programmatically described data structures according to ISO 13818-1, ISO-14496-2, ISO-14496-12 and ITU-T H.222.0 specifications.

Where does it work?

Works best in Chrome (stable branch), having more noticable lags when switching videos but still working in latest Firefox versions and IE10+.

Disclaimer

Please note that demo uses 3rd-party HLS demo source and service http://www.corsproxy.com/ for proxying it with needed Cross-Origin-Request headers for browsers to allow chunk downloading, so it may be unstable.

Where I can see that?

Check out http://rreverser.github.io/mpegts/ for live demo.

Screenshot:

Github project: es6-import

2013-10-13T19:40:32Z

Github project: knockout-protector

2013-09-13T12:14:46Z

Github project: bt-parser

2013-09-10T16:50:20Z

Github project: travis-publish-js

2013-09-04T17:09:57Z

"Ingvar Stepanyan: Serious JS" at FrontendConf Zurich 2013

2013-09-02T20:29:32Z

(this used to include a link to the video, but looks like it was since deleted)

Github project: jBinary.Hex

2013-08-01T10:19:48Z

Github project: LikeIE9

2013-06-17T22:28:12Z

Github project: AndroidMouse

2013-06-17T22:18:36Z

Github project: BMPImage

2013-06-17T22:18:31Z

Github project: JSPiano

2013-06-17T22:18:09Z

Github project: FilePipes

2013-06-17T22:17:52Z

Github project: PingPong

2013-06-17T22:17:41Z

Github project: tasks.js

2013-06-17T22:17:36Z

Github project: hubot-skype4com

2013-06-17T22:17:29Z

Github project: GitSync

2013-06-17T22:17:23Z

Github project: SharpScript

2013-06-17T22:17:11Z

Github project: LiveReload.Opera

2013-06-17T22:16:58Z

JS: Audio Data Processing

2012-03-04T15:56:18Z

My lightning talk presentation from JSGroup aka first KyivJS (18.02.2012):

https://www.slideshare.net/RReverser/js-audio-data-processing

Google Translate suggestion in Opera

2010-07-07T11:18:58Z

There is a way to add Google Translate suggestions in Opera (at least, for those who has PHP on local machine; if you don't, please choose any free PHP hosting with allowed HTTP request). So, firstly create and upload such PHP file:

<?php
header('Content-Type: text/javascript');

$q = stripslashes($_GET['q']);
$sl = $_GET['sl']; $tl = $_GET['tl'];

$t = file_get_contents('http://ajax.googleapis.com/ajax/services/language/translate?v=1.0&q=' . urlencode($q) . '&langpair=' . urlencode($sl) . '|' . urlencode($tl));
$a = json_decode($t, true);
echo json_encode($a['responseStatus'] != 200 ? array() : array($q, array($a['responseData']['translatedText'])));

It will return google translate using OpenSearch formatting.

Then, you should open search.ini (in profile folder) and find there your Google Translate's section. Add those two lines to it:

Suggest Protocol=JSON
Suggest URL=http://localhost/gsearch.php?q={SearchTerm} (your URL)

Restart Opera. That's all.

RReverser's

Bringing Sharp to WebAssembly and WebContainers

Github project: serde-ndim

Github project: ascom-alpaca-rs

Github project: eos-remote-web

Drawing to canvas in Emscripten

Porting USB applications to the web. Part 2: gPhoto2

WebAssembly feature detection

Porting USB applications to the web. Part 1: libusb

Embedding JavaScript snippets in C++ with Emscripten

Bundling non-JavaScript resources

Using WebAssembly threads from C, C++ and Rust

Using asynchronous web APIs from WebAssembly

Github project: lutz

Github project: wasm-bindgen-rayon

Github project: sep-sys-rs

WebAssembly shell with a real filesystem access in a browser

How #

Want to learn more? #

wasmbin: a self-generating WebAssembly parser & serializer

Backstory #

Let's build one more #

High-level API #

Trade-offs #

Feedback #

Debugging memory leaks in WebAssembly using Emscripten

Bringing OxiPNG to Squoosh

First attempt #

cloudflare-zlib (C fork) #

cloudflare-zlib (Rust wrapper) #

flate2-rs #

zlib (raw Rust bindings) #

libdeflate #

Time difference #

IDAT size difference #

Total size difference #

What now? #

Github project: cow-utils-rs

A History of HTML Parsing at Cloudflare: Part 1

Github project: asyncify

Github project: lazyhtml

Faster script loading with BinaryAST?

Github project: serde-wasm-bindgen

Github project: astexplorer-syn

Building fast interpreters in Rust

Improving request debugging in Cloudflare Workers

#[derive(Debug)] on serde steroids

Why would I want it? #

How do I use it? #

Why you might not want it #

Github project: xmatch

Github project: ws-proxy

Writing complex macros in Rust: Reverse Polish Notation

Error handling #

Conditional enum variants in Rust

Github project: derex

Github project: emnapi

Parsing JavaScript using SpiderMonkey from Rust

Reducing test cases for JavaScript tooling

Emscripten + Rust: how to interop with real JavaScript

Github project: override-hosts

Github project: esmod

Github project: serde-xml-rs

How we brought HTTPS Everywhere to the cloud

Tabs vs spaces: the ultimate argument

Github project: buffer-utf8-length

GOTOs in JavaScript are now possible

ES6 modules are dead, long live C preprocessor!

ECMAScript Syntax Graph

ECMAScript decorators on functions

Workaround #1 (just call it) #

Workaround #2 (runtime - boring but works) #

Workaround #3 (hacky but with true decorators) #

Workaround #4 (syntactic and my favourite) #

Github project: typescript-estree

Github project: babel-plugin-uglify

Started work on ES6 -> asm.js conversion

Currently supported: #

TODO: #

Github project: proxyfill

How

Want to learn more?

Backstory

Let's build one more

High-level API

Trade-offs

Feedback

First attempt

cloudflare-zlib (C fork)

cloudflare-zlib (Rust wrapper)

flate2-rs

zlib (raw Rust bindings)

libdeflate

Time difference

IDAT size difference

Total size difference

What now?

Why would I want it?

How do I use it?

Why you might not want it

Error handling

Workaround #1 (just call it)

Workaround #2 (runtime - boring but works)

Workaround #3 (hacky but with true decorators)

Workaround #4 (syntactic and my favourite)

Currently supported:

TODO:

Virtual Methods

Method Extraction

Usage

What's that

Why one more

But I like X builder! Should I throw it out?

API

Usage

License

UPDATE

How does it work?

Where does it work?

Disclaimer

Where I can see that?