IPFS

IPFS distributions

2016-06-01T00:00:00+02:00

dist.ipfs.io is the new distributions page for IPFS. This is the new one-stop-shop for finding and downloading all official binaries that the IPFS Team produces.

The IPFS Distributions Website

The distributions website itself is served by, hosted, and distributed through IPFS. The website assets and all of the binaries form one large IPFS content graph. This means that you can view and use this website through any IPFS node, even your own local ipfs node. All you need to do is to run an IPFS daemon and direct your browser to http://localhost:8080/ipns/dist.ipfs.io. Of course, you'll need to be online and connected to the internet, so you can find other ipfs nodes that have this website.

Download Deduplication

If you download files from dist.ipfs.io using your local IPFS node, future downloads may be much faster. When you click to download a file, your browser will ask to download it from your local IPFS node. In turn, your IPFS node will fetch the relevant content from other nodes in the network, and return it to your browser. Your browser will place the file in your Downloads folder, or wherever you directed it to.

However, once your local IPFS node has fetched the content, it will cache it locally for some time. This makes subsequent downloads of the exact same content instantaneous! Your browser asks the IPFS node for the content; the node already has it and simply returns it, without ever having to connect to other nodes. This also means that if other IPFS nodes in your local area network try to download the file, they may be able to fetch it from your node. Once you have the content locally, this can even work while disconnected from the internet!

It may also make downloading new versions much faster, because different versions of large binary files often have lots of duplicated data. IPFS represents files as a Merkle DAG (a datastructure similar to a merkle tree), much like Git or BitTorrent. Unlike them, when IPFS imports files, it chunks them to deduplicate similar data within single files. So when you need to download a new version, you only download the parts that are new or different - this can make your future downloads faster!

Project details

Every distribution has a section, which includes:

The distribution name and a short description;
The current version number and release date;
The software license (usually MIT);
A download button that detects your platform and automatically suggests the appropriate distribution for you;
A grid with download links for all supported platforms (operating system and architectures);
A Changelog, a link to a summary of all version changes;
An All Versions, a link to view and download previous versions.

The All Versions link on each distribution shows directory listings for all the available versions, and a versions file (example). This file can be used by tools, such as ipfs-update, to find all the available versions and download the latest.

The directory listing of each version (example) has all the platform archives (.zip or .tar.gz), a README.md and a dist.json which describe the release for humans and machines. It is meant to be easily consumed and used by tools.

The site is also used directly by ipfs-update to update IPFS.

Project Prerequisites

In order to be added to the distributions page, a product must:

Originate from the IFPS community;
Have high-quality UX and documentation;
Be well maintained and active.

If you think that a product should be there that isn't, get in touch.

Future Plans

In the future, we hope to:

Enable code signing, for progress on this subject you can check this pull request.
Enable closer integration with package managers.
Add more products.
Host more screenshots of the tools directly in the Distributions page.
Import binaries more intelligently in order to enhance change of deduplication.
Add the software license (usually MIT) to each distribution.

Contribute

We welcome any and all sorts of contributions! File issues or send us patches at ipfs/distributions.

Last but not least, a huge thank you to @dignifiedquire for his amazing work with the Dist page. None of this would be possible without him.

IPFS Weekly 10

2016-05-08T00:00:00+02:00

IPFS is a new hypermedia distribution protocol, addressed by content and identities, aiming to make the web faster, safer, and more open. In these posts, we highlight some of the development that has happened in the past week. For anyone looking to get involved, follow the embedded hyperlinks, search the wealth of information on GitHub or join us on IRC (#ipfs on the Freenode network).

If you would like to get this update as an email, sign up for our weekly newsletter!

Here are some of the highlights for the time period from April 6th through April 25th.

Updates

0.4.0!

go-ipfs 0.4.0 has been released! Among the many changes are a revamped implementation of the IPFS communication protocols, increased performance, improvements to IPNS (the Interplanetary Naming System), many bugfixes, and lots of new features to make IPFS even more powerful.

See the blog for more details.

Orbit

@haadcode upgraded all of his projects to use 0.4.0, got orbit-db and ipfs-log working in the browser, and improved Orbit's UX. As a result, Orbit is much faster and more stable than it was under 0.3! We now have working distributables for orbit-db and ipfs-log in the browser. They still require a local daemon to run to work but this will change when js-ipfs ships. @haadcode added some new features to Orbit: preview files directly in the chat with code highlighting, players for audio and video, and improved the files browsing functionality in general. You can now also copy the hash of a file to clipboard. It looks like this:

Finally, @haadcode also created a simple JavaScript logging module called logplease, which works in Node.js and browsers. logplease does two simple things: output log messages to the console and/or to a file (Node.js only) and display the log messages with nice colors. It was inspired by log4js and debug.

aegir

AEgir has been officially released. Formerly called dignified.js, this is our toolset for JavaScript modules, which cuts down on development time immensely by standardizing the process of testing, building, linting, releasing, and generally scaffolding out JS modules. It is now deployed over nearly all active JavaScript projects on IPFS. Check out the npm package or the github repo.

We've also overhauled the community JavaScript guidelines to reflect this change, and to make them more accessible to new developers.

go-libp2p and go-ipfs refactoring

We're modularizing go-libp2p, aiming at module parity with js-ipfs. This makes the codebase less daunting to newcomers, and makes maintenance and testing of everything much easier.

As part of this, @whyrusleeping removed over 9000 (yes, over 9000) lines of unused godeps dependencies from the go-ipfs repo. Its a good deal more manageable now, and we are getting ever closer to having a purely gx managed package.

js-ipld

@dignifiedquire shipped an updated version of js-ipld which now conforms to the latest spec of IPLD. In addition js-ipfs-ipld was created, which implements the building blocks to use IPLD in js-ipfs. The third package that was published is js-ipfs-cli which gives you a cli tool to interact and experiment with IPLD.

`ipfs-blob-store` v1.0.0 released

One of the benefits that comes from the new Files API in 0.4.0, is that we can emulate any kind of file system expectations on top of IPFS, and by doing so, enable applications that use a file system, to use IPFS without any modification. One perfect example of that are all the applications that use a module that implements the abstract-blob-store interface, now with ipfs-blob-store, with a simple npm install, you can swap out your current blob-store implementation by the IPFS one.

`js-ipfs-api` now supports the Files API

On the same lines, js-ipfs-api now offers all the Files API calls through a convenient and easy to use library. You can find all of the function declarations on the js-ipfs-api docs

Making js-ipfs and js-libp2p modules awesome

@JGAntunes, @dignifiedquire, @nginnever, @diasdavid, and @noffle have been working hard to bring the pantheon of js-ipfs modules up to snuff: detailed READMEs with examples and full API docs, as well as 100% test coverage. We made significant inroads last sprint on js-peer-id, js-multiaddr, js-peer-info, js-ipfs-blocks and more! There are a lot more modules that still need good READMEs and docs, so dive in and give us a hand!

go-ipfs and the Prometheus monitoring system

We've had Prometheus integrated with go-ipfs for some time already. In case you're unfamiliar, Prometheus is an open-source systems monitoring and alerting toolkit. It allows you to monitor diverse services, visualize the collected metrics in dashboards, and define rules for alerting. Together with the Grafana dashboard UI, it's the perfect monitoring system.

The scraping endpoint of go-ipfs is exposed at :5001/debug/metrics/prometheus by your go-ipfs daemon. Check Prometheus' Getting Started guide to try it out.

go-ipfs and restricting dialing

go-ipfs has support for restricting peer connections to certain IP address ranges, by using the Swarm.AddrFilters config setting. The following config example illustrates how to exclude all IPv4 peers, and all IPv6 link-local peers.

// .ipfs/config
"Swarm": {
  "AddrFilters": [
    "/ip4/0.0.0.0/ipcidr/0",
    "/ip6/fe80::/ipcidr/64"
  ]
}

Community

Berkman Center

@jbenet and @nicola spoke about IPFS with fellows at the Berkman Center. They discussed the core motivations of the IPFS Project, how it works, the state of the implementations, and a wide array of issues IPFS tackles. The discussion was excellent, starting with important ethical problems present in the web today, going through critical features our web should exhibit to solve them, and ending with how to get there. We look forward to tighter collaboration between our communities. Thank you very much for having us!

MIT Meetup

We had the first ever IPFS meetup in Cambridge, at MIT. It was sponsored by Consensys, and hosted by the MIT Bitcoin Club. @jbenet, @nicola, and @CReckhow each gave talks. A full writeup will appear on the blog as soon as we have edited the videos and posted them on the IPFS YouTube channel. We'll let you know when that happens, here.

Photo by @nicocesar_.

Peergos

Work on Peergos, a decentralised, secure file storage, sharing and social network using IPFS, is progressing well; @ianopolous is close to getting a FUSE binding working. Throughput hasn't been as bad as he was expecting with all the encryption and erasure coding, and it's looking likely that it will end up being network IO bound, instead of CPU bound for non trivial writes.

SpaceX Landing

Things are awesome and getting awesomer. Specifically, SpaceX landed a reusable Falcon 9 booster, which puts us that much closer to having a human colony on Mars. When that happens, we're betting they'll be using IPFS.

Content-Type of The Week

This week's Content-Type is the non-standard chemical/x-chemdraw. ChemDraw is a molecule editor first released in 1985, and still maintained as of today. We don't know which molecules were added to IPFS, but will do our best for IPFS to be around in 30 years just like ChemDraw is still around.

Every week we highlight a Content-Type that we've seen on the public HTTP-to-IPFS gateway at https://ipfs.io. If you're waiting to see one particularly interesting Content-Type in one of the next Weeklies, propose it by ipfs add -w'ing a file of that type, and requesting it from the public gateway: https://ipfs.io/ipfs/<hash>.

Contributors

Across the entire IPFS GitHub organization, the following people have committed code, created issues, or made a comment on GitHub between April 6th (noon, GMT) and April 25th. We're autogenerating this list using this tool and this other tool, so please let us know if your name isn't here.

@AdamStone (Adam Stone)
@amstocker (Andrew Stocker)
@ARezaK
@atommixz
@atondwal (Anish Tondwalkar)
@bpcamac (Brenton Camac)
@c-johnson (Christopher Johnson)
@chriscool (Christian Couder)
@chrisvoncsefalvay (Chris)
@chungy (Mike Swanson)
@ComSecNinja (Timo Virkkunen)
@cpacia (Chris Pacia)
@creckhow (Carolyn Reckhow)
@cusspvz (José Moreira)
@diasdavid (David Dias)
@dignifiedquire (Friedel Ziegelmayer)
@eboto (Erem Boto)
@edrex (Eric Drechsel)
@EliasGabrielsson (Elias Gabrielsson)
@eminence (Andrew Chin)
@fazo96 (Enrico Fasoli)
@felixonmars (Felix Yan)
@FrauBienenstich (Susanne Dewein)
@fzzr- (Alexander "FIZZΞR" Koz.)
@GitCop
@greenkeeperio-bot (Greenkeeper)
@groksie (James McFarland)
@haadcode (Haad)
@hermanjunge (Herman Junge)
@hutenosa
@ianopolous (Ian Preston)
@ikreymer (Ilya Kreymer)
@ingokeck (Ingo Keck)
@interfect
@jbenet (Juan Benet)
@jedahan (Jonathan Dahan)
@jefft0 (Jeff Thompson)
@jerodsanto (Jerod Santo)
@JGAntunes (João Antunes)
@kbtombul
@kevina (Kevin Atkinson)
@kevinmartinmauro
@krl (kristoffer)
@Kubuxu (Jakub Sztandera)
@kyledrake (Kyle Drake)
@lgierth (Lars Gierth)
@lidel (Marcin Rataj)
@machawk1 (Mat Kelly)
@mariusdarila (Marius Darila)
@matshenricson (Mats Henricson)
@mheiber (mheiber)
@MichaelMure (Michael Muré)
@micxjo (Micxjo Funkcio)
@mildred (Mildred Ki'Lya)
@mitar (Mitar)
@mitzip (mitzip)
@mrshu (Marek Šuppa)
@NDuma (NDuma)
@neko259
@nginnever (Nathan Ginnever)
@nicola (Nicola Greco)
@noffle (Stephen Whitmore)
@pfista (michael)
@Piruzzolo
@plungu (Paul Lungu)
@pokeball99
@rht
@RichardLitt (Richard Littauer)
@SCBuergel (Sebastian C. Bürgel)
@SilverWingedSeraph (Leo Tindall)
@sivachandran (Sivachandran)
@slang800 (Sean Lang)
@Stebalien (Steven Allen)
@sterpe (Steve Terpe)
@stevenh512 (Steven Hancock)
@TheNain38
@thomas-gardner
@traverseda (Alex Davies)
@upfs
@vijayee (Vijayee Kulkaa)
@vitzli
@whyrusleeping (Jeromy Johnson)
@wzhd
@xicombd (Francisco Baio Dias)

This newsletter is also a community effort. If you have cool things to share for the next weekly, drop a comment about it in the next weekly sprint issue! The more people mention items they want to see in the weekly there, the easier it is to make this and send it out.

Thanks, and see you next week!

Richard Littauer

Submit feedback about this issue here, or send us feedback about the IPFS Weekly in general.

IPFS Weekly 9

2016-04-20T00:00:00+02:00

If you would like to get this update as an email, sign up for our weekly newsletter!

Here are some of the highlights for the March 21st through March 28th sprint:

Updates

Package Managers

Package managers have been a large topic of discussion recently. Mostly, this is because of an issue with an author of some heavily used npm packages unpublishing all of his modules simultaneously. One of these was left-pad, which was used by thousands of builds globally, all of which broke when the package was removed. A great writeup of what happened is on the npm.js blog here; they took this very seriously, and shortly after changed their unpublish policy as a direct result.

Many people jumped to IPFS as a possible solution to this problem. With a permanent filesystem, unpublishing wouldn't be possible. Here's one post titled How to use IPFS to fix npm; here's an issue on a new GitHub organization, ipmjs, trying to find consensus on how to fix npm using a permanent storage system; here's an npm module, cowpen that publishes modules directly to IPFS; here's another decentralized package manager that sprung up using IPFS and Ethereum.

The IPFS community has been thinking about immutable package managers for a long time. IPFS itself began as an immutable package manager, and it is built to make writing them much easier. @diasdavid has a project called registry-mirror, which allows you to run an npm registry locally that is backed by packages retrieved from IPFS instead of NPM directly. He's written about a presentation he gave for it, here; the source code is here.

On a similar note, gx, a package manager for Go made by @whyrusleeping, was also mentioned in a lot of the discussions about npm and package managers, especially on Hacker News. In the past two weeks, the project went from 50 to 1000 stars, so people are clearly interested in this now.

The discussion about how to best use IPFS as a package manager is ongoing. Jump on GitHub if you have something to say; we're listening in the FAQ and in the notes repo.

DNS outage

We're using DigitalOcean to provide ipfs.io DNS. On Tuesday, March 24th, DigitalOcean DNS was hit by a severe outage lasting hours, which took the public gateway at ipfs.io down. We switched to DNSimple in an ad-hoc fashion and brought ipfs.io back while DigitalOcean was still down, but this incident obviously hit us on the wrong foot a bit. We'll be working to never get taken down this way again. It's HARD not to depend on any single points of failure. Here's a few things we'll do:

codify DNS zones, and tools to upload them to DNS providers
keep one or two backup DNS providers
update our monitoring and failover procedures

We'll post a more detailed post-mortem on our blog in the next few days.

Captain.log

Aye, you might want to check the new js-ipfs Captain.log entry, matey!

Following js-ipfs roadmap, we’re close™ to having a workable js-ipfs version that can be used in the browser and in Node.js. This will mark a very important milestone on the IPFS project and enable a whole set of new distributed web applications to be possible. If you want to be part of this effort, check out our Captain.log entry to get a full update and a list of tasks you can contribute to.

Orbit

@haadcode has been working on improvements to orbit-db, ipfs-log and Orbit. The message history fetching is now more stable and the UI feedback for loading messages is fixed. All this work will improve the user experience of Orbit.

js-ipfs-init

js-ipfs init works! @noffle finished the remaining pieces this week, including CLI usage. This included a handful of auxiliary PRs that cascaded out of that work. This makes the js-ipfs init process produce IPFS repos that are compatible with go-ipfs'.

Dictionary support for the zlib JavaScript Implementation, `pako`

One of the significant contributions made this week was the addition of 'dictionary' support for zlib JavaScript implementation, pako. With this contribution, we are able to have a complete implementation of SPDY 3.1's framing layer running in the browser, the default stream muxing library used in IPFS. You can find more about this contribution in the following issue and PR discussions:

https://github.com/nodeca/pako/issues/69
https://github.com/diasdavid/js-libp2p-spdy/pull/6
https://github.com/nodeca/pako/pull/77

go-ipfs

@whyrusleeping wrote a tool to move content from 0.4.0 to 0.3.11 (see levart-emit). He also discovered a file descriptor leak bug in utp causing connectivity issues, and began work on datastore performance improvements.

`jsipfs object` cli and http-api endpoints are complete

Now you can use jsipfs object in the same way you would use ipfs object. Big thanks to Francisco Dias for leading the last miles of this goal. The complete track of the development can be found at github.com/ipfs/js-ipfs/issues/58.

Nginx metrics

The infrastructure metrics dashboard didn't previously have HTTP request/response metrics from nginx's point of view, but only from IPFS's and multireq's point of view. (Multireq is our v04x/v03x multiplexing proxy). Nginx itself provides finegrained metrics only through their commercial subscriptions. We're now using mtail to parse metrics from nginx access logs and expose them to Prometheus. @lgierth will also contribute the nginx.mtail program upstream with mtail.

Community

Upcoming talks

On April 20th, IPFS will host a joint meetup with ConsenSys at MIT in Cambridge, Massachusetts. Sign up here!

First IPFS Meeting in NYC

We had our first IPFS meetup in New York! It went fantastically; expect an upcoming post on the Blog soon.

Meeting with NYC Mesh

@jbenet and @lgierth met with the fine folks of nycmesh.net. For the past two years they've been building a community Wifi network in New York City. We had lots of great conversation about wireless mesh networking and IPFS. If you live in NYC, you should come attend their meetups!

NYC Blockchain

Last Monday members of the IPFS comminuty attended a blockchain workshop event held by COALA, "a collaboration between academics, lawyers, technologists and entrepreneurs who have been driving research, policy and infrastructure-building in the blockchain ecosystem for the past three years" at the New York University School of Business. @diasdavid @haad @noffle and @nginnever were in attendance as @jbenet was a part of a protocols panel, speaking on scalability and the future of blockchain technology. A recording of the event should be available on youtube in the future here.

Lisbon Research and Development Meetup

The IPFS Lisbon community had their second "Research & Development Meetup", hosted by Uniplaces (https://www.uniplaces.com). The focus was "The Distributed Web" (Slides) and "Machine Learning + Artificial Intelligence for Recommender Algorithms", with talks by David Dias and João Ascensão, respectively. If you are around Lisbon, make sure to join http://www.meetup.com/ipfs-lisbon-meetup to get notified about the next one. Resources for this talk can be found here.

Seattle

@whyrusleeping gave a talk introducing IPFS at ta3m seattle - Techno-Activism 3rd Mondays. Video links to come when they are posted.

BitCoin News

BitCoin news had a discussion on using IPFS and Bitcoin for Decentralised Citizen Journalism. Check it out!

BitDevsNYC

Christian Lundkvist gave a talk on IPFS at BitDevsNYC. Christian works closely with IPFS at ConsenSys.

IPFS Meme of the Week

From https://twitter.com/jplur_/status/712670265919086594. Thanks, jplur_!

Content-Type of the Week

Now that we're using mtail to make better sense of nginx serving the IPFS-to-HTTP gateway, we can graph the frequency of content types served. We'll showcase interesting content types served from the gateway in the coming weeklies.

The first Content-Type of the week is: application/x-chdr, which signifies a C source header file.

Contributors

Across the entire IPFS GitHub organization, the following people have committed code, created issues, or made a comment on GitHub between March 21st (noon, GMT) and March 28th. We're autogenerating this list using this tool and this other tool, so please let us know if your name isn't here.

@achin (Andrew Chin)
@AdamStone (Adam Stone)
@admiral-Guck (Gucky)
@alexAubin (Alexandre Aubin)
@amstocker (Andrew Stocker)
@anarcat (anarcat)
@Anthonyzou (Anthony Ou)
@area
@ARezaK
@atommixz
@Azulan (Frank Flores)
@balupton (Benjamin Lupton)
@bgrieder (Bruno Grieder)
@cb0 (Marcus Puchalla)
@chinshiketsu (Sijie Chen)
@chpio
@chriscool (Christian Couder)
@CMCDragonkai (Roger Qiu)
@ComSecNinja (Timo Virkkunen)
@darkstar (Michael D.)
@davidar (David A Roberts)
@diasdavid (David Dias)
@dignifiedquire (Friedel Ziegelmayer)
@djdv (Dominic Della Valle)
@ehmry (Emery Hemingway)
@fomkin (Aleksey Fomkin)
@gerrickw (Gerrick W)
@greenkeeperio-bot (Greenkeeper)
@haadcode (Haad)
@harlantwood (Harlan T Wood)
@hshimo (Hiroshi Shimo)
@hutenosa
@ianopolous (Ian Preston)
@iav (iav)
@ibnesayeed (Sawood Alam)
@insanity54 (Chris Grimmett)
@jbenet (Juan Benet)
@JGAntunes (João Antunes)
@jmsmcfrlnd
@josephholsten (Joseph Anthony Pasquale Holsten)
@julian-smith-code (Julian Smith)
@jwsher (Justin Sher)
@kalmi (Tarnay Kálmán)
@kevina (Kevin Atkinson)
@knocte (Andres G. Aragoneses)
@KrishnaPG (Gopalakrishna Palem)
@Kubuxu (Jakub Sztandera)
@kumavis (kumavis)
@kyledrake (Kyle Drake)
@kyrias (Johannes Löthberg)
@lgierth (Lars Gierth)
@lidel (Marcin Rataj)
@machawk1 (Mat Kelly)
@matshenricson (Mats Henricson)
@Mec-iS (Lorenzo)
@micahredding (Micah Redding)
@MichaelMure (Michael Muré)
@micxjo (Micxjo Funkcio)
@mildred (Mildred Ki'Lya)
@mitar (Mitar)
@Mithgol
@monoflash (Alex Geer)
@montagsoup
@mpgn (Martial Puygrenier)
@mrshu (Marek Šuppa)
@nginnever (Nathan Ginnever)
@nicola (Nicola Greco)
@NightRa (Ilan Godik)
@noffle (Stephen Whitmore)
@nonchip (Kyra Zimmer)
@novocodev
@palkeo (palkeo)
@pguth (Per)
@Piruzzolo
@pocketmax
@pokeball99
@polkovnikov-ph (Philip Polkovnikov)
@puffnfresh (Brian McKenna)
@rht
@RichardLitt (Richard Littauer)
@rsynnest
@sdockray
@se3000 (Steve Ellis)
@Shikibashi
@SilverWingedSeraph (Leo Tindall)
@slothbag
@Stebalien (Steven Allen)
@syang
@tadfisher (Tad Fisher)
@thomas-gardner
@tinybike (Jack Peterson)
@upfs
@voltrevo (Andrew Morris)
@whyrusleeping (Jeromy Johnson)
@willglynn (Will Glynn)
@wking (W. Trevor King)
@wmhilton (Will Hilton)
@xicombd (Francisco Baio Dias)
@xsnk (Sankrant Chaubey)
@yuvipanda (Yuvi Panda)
@Zogg

Thanks, and see you next week!

Richard Littauer

Submit feedback about this issue here, or send us feedback about the IPFS Weekly in general.

go-ipfs 0.4.0 has been released

2016-04-07T00:00:00+02:00

The IPFS Files API

In 0.4.0, we've added a new feature, the Files API, available through the subcommand: ipfs files. This subcommand allows a program to interact with IPFS using familiar filesystem operations, namely: creating directories, reading, writing and deleting files, listing out different directories, and so on.

This feature enables any application that uses a filesystem-like backend to use IPFS for storage without changing the application logic at all.

It is used like so:

> ipfs files mkdir /cats

> ipfs files ls /
cats

> echo bar | ipfs files write --create /cats/foo

> ipfs files ls /cats
foo

> ipfs files read /cats/foo
bar

> ipfs files stat /
QmNU8HmaeRa8VtfqAoQRJhSE5Zx54vyYf2nT1bDGcYXaNv
...

# /ipfs/QmNU8HmaeRa8VtfqAoQRJhSE5Zx54vyYf2nT1bDGcYXaNv is a snapshot
# of this file system!

# You can see it locally or on the public gateway:
#  https://ipfs.io/ipfs/QmNU8HmaeRa8VtfqAoQRJhSE5Zx54vyYf2nT1bDGcYXaNv

# Subsequent edits will produce a new /ipfs address for the root directory.

One great example of this is ipfs-blob-store, an IPFS-backed storage driver that implements the blob-store-interface, so that any app that uses any other blob-store storage driver (S3, IndexedDB, LevelDB, etc) can now use IPFS. registry-mirror uses this approach to mirror the npm registry onto IPFS.

We are looking forward to seeing more use cases of this new convenient and powerful API.

Why we're changing the protocol

The most important change allows IPFS implementations to use pluggable stream multiplexers, such as yamux, spdystream, or muxado. Instead of locking IPFS permanently into a single multiplexer that won't work for every language or situation, this change allows each IPFS implementation to implement the multiplexers that they choose to, and then negotiate which to use during the initial connection handshake.

This modularity with stream muxing makes it easier for certain languages to improve performance. For example, the Go programming language may have muxado and yamux implementations that are really good, but many languages lack good (or any) implementations of them. Or take Node.js, which works well with spdy-transport, and it would be nice to take advantage of that. And then there are options like multiplex, which may not have the same performance, but are much easier to implement.

So, by supporting as many muxers as we can, we get to choose the best multiplexers for the job. It also makes it much easier to implement the IPFS protocols in a new language. And of course, if a better multiplexers standard comes along, it will be easier to upgrade IPFS to support it in the future.

The same code that allows us to easily select a stream muxer is also being used to select which IPFS sub-protocol to use for any given stream between peers. Now, if we need to make a breaking protocol change to any of those (like the DHT or bitswap) we can do so seamlessly, and provide easy backwards compatibility. We won't need to "break" anything because we will be able to have nodes run multiple protocols at the same time for compatibility.

In addition to the multiplexers changes, the protocol revamp has also improved efficiency and performance in a few important ways, including the elimination of a double wrapping of the length prefixer, and the removal of some unnecessary round trips between nodes.

Breaking changes

This release contains a breaking change to the network wire protocol in the form of a major refactor and upgrade to the libp2p handshake protocol. Because of the refactor, all IPFS daemons earlier than 0.4.0 will not be able to communicate with the newest version. It is strongly recommended that everyone running an IPFS node upgrades to the latest version as soon as possible, as these nodes will, after a certain time, no longer be able to communicate with the majority of the network until they are upgraded. There are instructions on how to update below.

Refactoring the protocol is not something to be done lightly. But at this early stage, this is necessary to ensure we have the right design for IPFS in place for the future. It's better to improve the protocol now during this alpha stage of the project than when there are a lot more people running nodes across a lot of different implementations.

One of the important changes that's been made to the protocol means that there should never be a breaking change like this again. This is due to a change to allow nodes to announce the version of the protocol they are using when connecting to other IPFS nodes. The goal is to roll any future protocol changes into the implementations gradually, so that we can still support legacy protocols for a period of time, making it easier to deprecate old versions over time.

The public gateway and bootstrappers

We provide two essential public services to the IPFS community: the public gateway at https://ipfs.io and the default bootstrap nodes. We're making sure that despite the breaking changes, both will continue to work with 0.4.x and 0.3.x for a while. You can read more about the details of this in an earlier blog post: Migrating ipfs.io from go-ipfs 0.3.x to 0.4.0. We expect to keep this grace period open until the end of April 2016.

Changes to the repo

We made a few changes to our on disk storage format (called the 'repo'). The way object pinning (ipfs pin add) works has also been upgraded to be much more efficient, which will improve the overall speed of adding and downloading IPFS data. Previously, when you pinned a file recursively, it would add a recursive type pin to the root, and then add an indirect type pin to each child node. This made enumeration for GC convenient, but was very slow and made a lot of things needlessly complicated. We have switched to just adding the recursive pin to the root object, and then doing the enumeration of child nodes when we actually run a garbage collection.

Because of this change, you will need to run a migration (from repo version 2 to version 3). If you update with the ipfs-update tool, this will be done for you automatically. If you updated manually, and did not run the migration, ipfs will fail to run, and print a message saying that there is a mismatch in the repo versions.

Other improvements and fixes

In addition to a major protocol improvement and upgrade, this release adds a lot of new functionality, performance speedups, and stability fixes that make this the best version of IPFS to date.

This release also includes performance and usability improvements to IPNS, which is IPFS's mutability layer. IPNS creates a link from an IPFS node's public key to the hash of an arbitrary objects hash, in a way that is cryptographically verifiable. We call this "publishing" a hash, and you can try it out using the ipfs publish command. By allowing users to change what the pubkey hash points to, we provide users with a single hash they can give to their users to get the latest version of their data. This creates a seamless way to use IPFS to verify content, and to distribute content via trustless nodes in a smart, safe way. This brings IPFS closer to the goal of being a global filesystem of data, that can allow everyone in the world to help serve the world's data in a way that enriches and empowers everybody.

How to upgrade

Depending on how you initially installed IPFS, there are several ways to upgrade. If you installed IPFS with a pre-built binary, you can either head over to dist.ipfs.io and grab the latest version from there. Or alternatively, from the same page you can grab the ipfs-update binary, and use it to perform the upgrade for you. If you installed from source, you can simply update your git repo to the v0.4.0 tag, run make toolkit_upgrade && make install.

Please upgrade your IPFS nodes as soon as you can, so you can take advantage of the improvements!

Changelog

This is a major release with plenty of new features and bugfixes. It also includes breaking changes which make it incompatible with v0.3.x on the networking layer.

Major Changes
Multistream
- The addition of multistream is a breaking change on the networking layer, but gives IPFS implementations the ability to mix and match different stream multiplexers, e.g. yamux, spdystream, or muxado. This adds a ton of flexibility on one of the lower layers of the protocol, and will help us avoid further breaking protocol changes in the future.
Files API
- The new files command and API allow a program to interact with IPFS using familiar filesystem operations, namely: creating directories, reading, writing, and deleting files, listing out different directories, and so on. This feature enables any other application that uses a filesystem-like backend for storage, to use IPFS as its storage driver without having change the application logic at all.
Gx
- go-ipfs now uses gx to manage its dependencies. This means that under the hood, go-ipfs's dependencies are backed by IPFS itself! It also means that go-ipfs is no longer installed using go get. Use make install instead.
New Features
Web UI
- Update to new version which is compatible with 0.4.0. (@dignifiedquire)
Networking
- Implement uTP transport. (@whyrusleeping)
- Allow multiple addresses per configured bootstrap node. (@whyrusleeping)
IPNS
- Improve IPNS resolution performance. (@whyrusleeping)
- Have dnslink prefer TXT _dnslink.example.com, allows usage of CNAME records. (@Kubuxu)
- Prevent ipfs name publish when /ipns is mounted. (@noffle)
Repo
- Improve performance of ipfs add. (@whyrusleeping)
- Add Datastore.NoSync config option for flatfs. (@rht)
- Implement mark-and-sweep GC. (@whyrusleeping)
- Allow for GC during ipfs add. (@whyrusleeping)
- Add ipfs repo stat command. (@tmg, @diasdavid)
General
- Add support for HTTP OPTIONS requests. (@lidel)
- Add ipfs diag cmds to view active API requests (@whyrusleeping)
- Add an IPFS_LOW_MEM environment veriable which relaxes Bitswap's memory usage. (@whyrusleeping)
- The Docker image now lives at ipfs/go-ipfs and has been completely reworked. (@lgierth)
Security fixes
The gateway path prefix added in v0.3.10 was vulnerable to cross-site scripting attacks. This release introduces a configurable list of allowed path prefixes. It's called Gateway.PathPrefixes and takes a list of strings, e.g. ["/blog", "/foo/bar"]. The v0.3.x line will not receive any further updates, so please update to v0.4.0 as soon as possible. (@lgierth)
Incompatible Changes
Install using make install instead of go get (@whyrusleeping)
Rewrite pinning to store pins in IPFS objects. (@tv42)
Bump fs-repo version to 3. (@whyrusleeping)
Use multistream muxer (@whyrusleeping)
The default for --type in ipfs pin ls is now all. (@chriscool)
Bug Fixes
Remove msgio double wrap. (@jbenet)
Buffer msgio. (@whyrusleeping)
Perform various fixes to the FUSE code. (@tv42)
Compute ipfs add size in background to not stall add operation. (@whyrusleeping)
Add option to have ipfs add include top-level hidden files. (@noffle)
Fix CORS checks on the API. (@rht)
Fix ipfs update error message. (@tomgg)
Resolve paths in ipfs pin rm without network lookup. (@noffle)
Detect FUSE unmounts and track mount state. (@noffle)
Fix go1.6rc2 panic caused by CloseNotify being called from wrong goroutine. (@rwcarlsen)
Bump DHT kvalue from 10 to 20. (@whyrusleeping)
Put public key and IPNS entry to DHT in parallel. (@whyrusleeping)
Fix panic in CLI argument parsing. (@whyrusleeping)
Fix range error by using larger-than-zero-length buffer. (@noffle)
Fix yamux hanging issue by increasing AcceptBacklog. (@whyrusleeping)
Fix double Transport-Encoding header bug. (@whyrusleeping)
Fix uTP panic and file descriptor leak. (@whyrusleeping)
Tool Changes
Add --pin option to ipfs add, which defaults to true and allows --pin=false. (@eminence)
Add arguments to ipfs pin ls. (@chriscool)
Add dns and resolve commands to read-only API. (@Kubuxu)
Add option to display headers for ipfs object links. (@palkeo)
General Codebase Changes
Check Golang version in Makefile. (@chriscool)
Improve Makefile. (@tomgg)
Remove dead Jenkins CI code. (@lgierth)
Add locking interface to blockstore. (@whyrusleeping)
Add Merkledag FetchGraph and EnumerateChildren. (@whyrusleeping)
Rename Lock/RLock to GCLock/PinLock. (@jbenet)
Implement pluggable datastore types. (@tv42)
Record datastore metrics for non-default datastores. (@tv42)
Allow multistream to have zero-rtt stream opening. (@whyrusleeping)
Refactor ipnsfs into a more generic and well tested mfs. (@whyrusleeping)
Grab more peers if bucket doesn't contain enough. (@whyrusleeping)
Use CloseNotify in gateway. (@whyrusleeping)
Flatten multipart file transfers. (@whyrusleeping)
Send updated DHT record fixes to peers who sent outdated records. (@whyrusleeping)
Replace go-psutil with go-sysinfo. (@whyrusleeping)
Use ServeContent for index.html. (@AtnNn)
Refactor object patch API to not store data in URL. (@whyrusleeping)
Use mfs for ipfs add. (@whyrusleeping)
Add Server header to API responses. (@Kubuxu)
Wire context directly into HTTP requests. (@rht)
Wire context directly into GetDAG operations within GC. (@rht)
Vendor libp2p using gx. (@whyrusleeping)
Use gx vendored packages instead of Godeps. (@whyrusleeping)
Simplify merkledag package interface to ease IPLD inclusion. (@mildred)
Add default option value support to commands lib. (@whyrusleeping)
Refactor merkledag fetching methods. (@whyrusleeping)
Use net/url to escape paths within Web UI. (@noffle)
Deprecated key.Pretty(). (@MichealMure)
Documentation
Fix and update help text for every ipfs command. (@RichardLitt)
Change sample API origin settings from wildcard (*) to example.com. (@Kubuxu)
Improve documentation of installation process in README. (@whyrusleeping)
Improve windows.md. (@chriscool)
Clarify instructions for installing from source. (@noffle)
Make version checking more robust. (@jedahan)
Assert the source code is located within GOPATH. (@whyrusleeping)
Remove mentions of /dns from ipfs dns command docs. (@lgierth)
Testing
Refactor iptb tests. (@chriscool)
Improve t0240 sharness test. (@chriscool)
Make bitswap tests less flaky. (@whyrusleeping)
Use TCP port zero for ipfs daemon in sharness tests. (@whyrusleeping)
Improve sharness tests on AppVeyor. (@chriscool)
Add a pause to fix timing on t0065. (@whyrusleeping)
Add support for arbitrary TCP ports to t0060-daemon.sh. (@noffle)
Make t0060 sharness test use TCP port zero. (@whyrusleeping)
Randomized ipfs stress testing via randor (@dignifiedquire)
Stress test pinning and migrations (@whyrusleeping)

IPFS Weekly 8

2016-03-22T00:00:00+01:00

If you would like to get this update as an email, sign up for our weekly newsletter!

Here are some of the highlights for the March 14 sprint:

Updates

webui

@dignifiedquire has been working hard on the new webui, which will be coming soon. You'll be able to preview images, watch videos, create folders, drag and drop, and more. Here's a pretty gif. To help out, check out the help wanted labels on the repo.

PubSub

There are some active discussions on PubSub, following a good video chat last week during the hangouts. PubSub is the name we're using to talk about a simple protocol which will help facilitate easy publishing and subscribing on top of IPFS. Our requirements are that it must be easy to implement, well-layered, and mesh well with the rest of the IPFS abstractions. To join the conversation about the PubSub API, check out this issue. For more discussions, check out all of the PubSub issues in the notes repo.

ipfs-log

@haadcode has been working on and released ipfs-log, a partially ordered linked list of IPFS hashes. Each entry in the log points to all known heads or leaf nodes. It can be used as a building block for applications that need to track "dynamic content", eg. track a version of a file, create a feed of IPFS hashes, messaging or as a transport for CRDTs. THis was originally created for, and is currently used in, orbit-db, a KV-store and Event Log on IPFS.

ipfs init

ipfs init for js-ipfs is almost complete, thanks to @noffle. This will have go-ipfs compatibility, but run using only JavaScript. If you like building tests and sharpening your semicolons, there's plenty of opportunities to contribute in js-ipfs land.

go-ipfs

This week in go-ipfs, we prepped for shipping version 0.4.0. This included lots of testing, writing of information, and verification of different aspects of IPFS. @whyrusleeping wrote a stress test for the fs-repo-migrations that adds a very large number of objects (over 200,000) and pins a couple thousand of them, runs the migration, verifies everything, runs a gc, and then verifies everything again. Once that got working, he kicked off a test run of that with the numbers bumped by a factor of 10 (over two million objects!) and everything completed just fine. This robustness means that 0.4.0 will be ready to ship very, very soon.

ipfs-firefox-addon

There were some changes in ipfs-firefox-addon since it was mentioned in Weekly No.3 (v1.4.2).

The Firefox addon that provides transparent access to IPFS resources via local HTTP2IPFS gateway has been fully reviewed by Mozilla and updated to v1.5.6. On average it has over 350 daily users.

The 1.5.x series brings various UX improvements such as Realtime Status and Diagnostics, along with experimental features that can be enabled on the Preferences screen. Check the full list at Github. Feature requests and bug reports are welcome!

Community

Mediachain

Our friends at Mine recently released L-SPACE, the Mediachain server. they have also been writing great updates to their blog: How Mediachain works, Dev Update V, Dev Update VI, and more. The press took notice and Mediachain has been featured in Nasdaq, Bitcoin Magazine, CCN, and more! Congratulations, and thanks for the IPFS shoutouts there!

Lisbon

@diasdavid organized a research and development meeting for IPFS in Lisbon. If you're in the area, join this meetup group.

Press

Bitcoin Magazine had a guest post by Zach Ramsay, from Eris Industries, about How Blockchains Can Further Public Science. Zach also published part two on the Eris blog: Public Science: A Slightly More Practical Guide. Both are well worth the read, especially if you're in academia.

Contributors

Across the entire IPFS GitHub organization, the following people have committed code, created issues, or made a comment on GitHub between March 14th (noon, GMT) and March 21st. We're autogenerating this list using this tool and this other tool, so please let us know if your name isn't here.

@amstocker (Andrew Stocker)
@anarcat (anarcat)
@ansuz (ansuz)
@basile-henry (Basile Henry)
@bdunlay (Brian Dunlay)
@brainframe-me (Cox Davy)
@candeira (Javier Candeira)
@chpio
@chriscool (Christian Couder)
@christianlundkvist (Christian Lundkvist)
@Cleric-K
@clkao (Chia-liang Kao)
@davidar (David A Roberts)
@diasdavid (David Dias)
@dignifiedquire (Friedel Ziegelmayer)
@doesntgolf (Nate Dobbins)
@ehd (Stephan Seidt)
@eminence (Andrew Chin)
@fazo96 (Enrico Fasoli)
@GoogilyBoogily (Derek Mayer)
@greenkeeperio-bot (Greenkeeper)
@haadcode (Haad)
@harlantwood (Harlan T Wood)
@ianopolous (Ian Preston)
@ingokeck (Ingo Keck)
@jbenet (Juan Benet)
@jwsher (Justin Sher)
@kalmi (Tarnay Kálmán)
@kevina (Kevin Atkinson)
@klartext
@kseistrup (Klaus Alexander Seistrup)
@Kubuxu (Jakub Sztandera)
@lgierth (Lars Gierth)
@lhenocque
@lidel (Marcin Rataj)
@luigiplr (Luigi Poole)
@matshenricson (Mats Henricson)
@MaxEntropyy
@Mec-iS (Lorenzo)
@mildred (Mildred Ki'Lya)
@Mithgol
@mlbk0
@mnp (Mitchell Perilstein)
@montagsoup
@nginnever (Nathan Ginnever)
@nicola (Nicola Greco)
@noffle (Stephen Whitmore)
@palkeo (palkeo)
@RichardLitt (Richard Littauer)
@rinpoo
@sahib (Chris Pahl)
@se3000 (Steve Ellis)
@sexybiggetje (Martijn de Boer)
@sivachandran (Sivachandran)
@slothbag
@Stebalien (Steven Allen)
@thelinuxkid (Andres Buritica)
@thomas-gardner
@wanderer
@wasserfuhr (RainerWasserfuhr)
@whyrusleeping (Jeromy Johnson)
@willglynn (Will Glynn)
@wking (W. Trevor King)
@xicombd (Francisco Baio Dias)

Thanks, and see you next week!

Richard Littauer

Submit feedback about this issue here, or send us feedback about the IPFS Weekly in general.

IPFS Weekly 7

2016-03-17T00:00:00+01:00

If you would like to get this update as an email, sign up for our weekly newsletter!

Here are some of the highlights for the second week of March:

Updates

Orbit

Orbit, a distributed, peer-to-peer chat application built on IPFS, is back in active development and going through a major code base overhaul. It now uses orbit-db as it's database layer. orbit-db is a KV-store and Event Log on top of IPFS which allows developers to use IPFS as a database. Last week orbit-db got big performance and stability improvements and is now using CRDTs for eventual consistency.

Go-IPFS

A nasty bug in our networking code was fixed this week. An issue in yamux (our primary stream multiplexer) would cause code to hang when opening a new stream if there were too many in flight stream opens. As a result of this, large file transfers (ipfs refs -r, ipfs get, and so on) would hang.

IPFS 0.4.0 is now very close to shipping, we ran @dignifiedquire's randor tool quite extensively and are more confident in the repo operations that have changed since 0.3.11. The release now has a somewhat short checklist of things that are blocking the official release.

As a small side project, @whyrusleeping started benchmarking each of our possible datastore implementations with ds-bench. The results will help us improve the performance of our storage moving forward. On that same topic, he started an experiment in new and exciting datastores and wrote a datastore implementation that can be backed by a sql database. @whyrusleeping then spun up a postgresql database (from docker) and ran the benchmarks against it, without any sort of tuning the initial performance metrics showed that it was around three time as fast as our current on-disk datastore. (Note: the metrics are simply testing the performance of writing random 256k binary blobs to each datastore).

js-multiaddr

The js-multiaddr implementation was upgraded, now that it now matches go-ipfs features. What this means is that js-multiaddr now supports protocols that need to be encoded with varints. A varint is a integer that uses only the number of bytes it needs to be described by using a continuation bit (more details). Before this addition, we were not able to declare http, websockets or even IPFS multiaddrs because these protocol have a code that doesn't fit in a byte - now we can. You can check the encoding table here.

mafmt

mafmt, a module that @whyrusleeping wrote and which stands for multiaddr format, filters out different formats to use with certain transports. Thanks to @daviddias, this now does uTP and WebSockets validation, as well.

libp2p-swarm

The new libp2p-swarm has been released; to reiterate last week, the module has been overauled to have a new API, to be agnostic about which transport an application has to use, and also to have more tests. This also exposes more internal processes, which makes error checking and logging more useful. Libp2p will eventually help users traverse IPFS for any node which allows their desired transport, meaning that we can have a more connected and less brittle web.

Community

@jbenet was in Boston this week for consensus research discussions. While in Boston, @jbenet visited the Digital Currency Initiative at the MIT Media Lab, where he spoke about IPFS and its implications for blockchains, for digital publishing, and for access to information.

c-base/ipfs-ringpin is a new set of tools for setting up an IPFS "file pinning ring" between multiple parties. This can be used, for example, for hackerspaces to provide redundancy by pinning each other's files. pipermerriam/ipfs-persistence-consortium and VictorBjelkholm/pincoop do similar things.

the-gred/jsjob-ethereum is an experiment at creating a computational market on top of Ethereum. Computational jobs are run in a JavaScript sandbox, and both input data, the algorithm to run, and the produced results are shared over IPFS.

@jbenet gave a talk titled Datastructures In and On IPFS at last year's QCon SF. The video was released this week, so now you can all see it! The talk includes a typical introduction (skip if you know IPFS well), and then dives into datastructures, including Merkle Links, Mazieres links, how IPNS works, IPRS records, versioning, Keychain (PKI on IPFS), Persona (identity), and more.

The IPFS powered USB deaddrop at c-base was demonstrated at the LoganCIJ 2016 Symposium, in the Investigative Journalism conference. More pictures here. If you want to build your own IPFS USB deaddrops, check out the c-base project here: c-base/ipfs-deaddrop.

The IPFS Copenhagen Meetup organized by @NeoTeo had another meeting a few days ago. If you're in the area, sign up at the Meetup.com page so you don't miss the next one!

Several core IPFS developers will be in NYC from March 28th to April 8. We will be organizing an IPFS Meetup within that time frame. If you would like to come, please follow this github issue for the final details. If you would like to present, post in that issue.

Contributors

Across the entire IPFS GitHub organization, the following people have committed code, created issues, or made a comment on GitHub between March 7th (noon, GMT) and March 14th. We're autogenerating this list using this tool and this other tool, so please let us know if your name isn't here.

@42 (42 Coffee Cups)
@aaannndddyyy
@adamc199
@anarcat (anarcat)
@atondwal (Anish Tondwalkar)
@basilmon92 (Basil M Kuriakose)
@bergie (Henri Bergius)
@brailateo (Constantin Teodorescu)
@candeira (Javier Candeira)
@chpio
@chriscool (Christian Couder)
@christianlundkvist (Christian Lundkvist)
@clkao (Chia-liang Kao)
@Crest (Crest)
@cryptix (Henry)
@davidar (David A Roberts)
@DavidBooher (David Booher)
@deawar (Dean Warren)
@deb0ch (Thomas de Beauchêne)
@diasdavid (David Dias)
@dignifiedquire (Friedel Ziegelmayer)
@djdv (Dominic Della Valle)
@doesntgolf (Nate Dobbins)
@dontspamm
@dylanPowers (Dylan Powers)
@eminence (Andrew Chin)
@greenkeeperio-bot (Greenkeeper)
@gritzko (Victor Grishchenko)
@haadcode (Haad)
@hutenosa
@ianopolous (Ian Preston)
@icf20
@Iyer-VivekAnand
@jbenet (Juan Benet)
@klauswuestefeld (Klaus Wuestefeld)
@knocte (Andres G. Aragoneses)
@krl (kristoffer)
@Kubuxu (Jakub Sztandera)
@kyledrake (Kyle Drake)
@lamarpavel
@lgarron (Lucas Garron)
@lgierth (Lars Gierth)
@lhenocque
@loadletter
@matshenricson (Mats Henricson)
@mattseh
@micahredding (Micah Redding)
@MichaelMure (Michael Muré)
@micxjo (Micxjo Funkcio)
@mildred (Mildred Ki'Lya)
@Mithgol
@mklarmann (mklarmann)
@mnp (Mitchell Perilstein)
@moritz121
@myleshorton
@Nadziv (Shahinaz)
@nginnever (Nathan Ginnever)
@nicola (Nicola Greco)
@noffle (Stephen Whitmore)
@obvio171 (Helder Ribeiro)
@pietsch (Christian Pietsch)
@plungu (Paul Lungu)
@pokeball99
@polycopter
@rht
@RichardLitt (Richard Littauer)
@rsynnest
@SCBuergel (Sebastian C. Bürgel)
@sdockray
@sexybiggetje (Martijn de Boer)
@shalnoff
@sivachandran (Sivachandran)
@skippa
@slothbag
@sptankard
@Stebalien (Steven Allen)
@suisha (David Mai)
@thomas-gardner
@Unode (Renato Alves)
@wasserfuhr (RainerWasserfuhr)
@whyrusleeping (Jeromy Johnson)
@xicombd (Francisco Baio Dias)
@yncyrydybyl (Yan Minagawa)
@yuvallanger (Yuval Langer)
@yuy168

This newsletter is also a community effort. If you have cool things to share for the next weekly, drop a comment about it in the weekly sprint issue! The more people mention items they want to see in the weekly there, the easier it is to make this and send it out.

Thanks, and see you next week!

Richard Littauer and Andrew Chin

Submit feedback about this issue here, or send us feedback about the IPFS Weekly in general.

IPFS Weekly 6

2016-03-09T00:00:00+01:00

If you would like to get this update as an email, sign up for our weekly newsletter!

Here are some of the highlights for the first week of March:

Updates

station

The new version of station is ready for developer preview! station is one of the easiest ways to get an IPFS daemon running on your machine. It acts as a service and you get many convenient features, such as the ability to turn an IPFS node on through a GUI and drag and drop to share through IPFS. To try it out, you need Node.js 4 (installation instructions here) and npm 3 (which comes with Node) installed. Then, do the following:

> git clone https://github.com/ipfs/station.git && cd station
> npm install
> npm start

website

The list of API Commands on the website has been updated. This provides a single place to look through all of the CLI commands for go-ipfs, at once; it is a good reference point if you're not sure exactly which command to use next, and ipfs commands seems sparse.

AUR

@Kubuxu has made sure that gx, gx-go, and ipget are available on AUR, the arch-linux user repository. This means that it will be easier to get and install these on some linux platforms.

karma-peer

@dignifiedquire's work on karma-peer means that it now has the ability to dynamically launch browsers, which will help @diasdavid (and hopefully more people!) write better tests for P2P browser applications. See an example of some tests, here. You can also read the discussion that lead to this module and the original tool to test P2P browser applications.

randor

@dignifiedquire has been working on randor, a testing framework that will be able to send huge files and lots of requests into IPFS in order to test how it works for edge cases and how it scales. Randor is now able to rerun tests predictably based on stored data, so it's easy to find and fix bugs. @whyrusleeping is already working on the first bug that randor detected. To contribute, check out the repository.

WebRTC Explorer

WebRTC Explorer 2.0.0 has been alpha released! WebRTC Explorer is a P2P Routing Overlay Network, using WebRTC Data Channels as the transportation layer between nodes. WebRTC Explorer enables communication between browsers without needing mediators (servers), enabling users to route packets between machines, using only Web technologies. WebRTC Explorer is inspired by the Chord DHT, to create a routing scheme with finger tables that are evenly balanced across nodes. @diasdavid wrote up a more extensive blog post here, and has an introductory video here.

libp2p

@diasdavid released versions of two modules for stream multiplexing: libp2p-spdy and libp2p-multiplex. As well, libp2p-swarm has a new API, more tests, less magic, and more flexibility. There is an open PR to track these changes; if you'd like to get involved, follow it.

js-ipfs

Some of you have been asking about ways to contribute to the JavaScript implementation of IPFS: well, wait no more! Now you can read the latest captain.log entry and learn about the state of the project and have a list of issues of things that you can contribute. We appreciate all your help.

ipfs-pad

We want to build an Etherpad-like product on top of IPFS. There's a lot of technical ground to cover to do this: how do we know how to sort edits? How do we deal with high latencies (days/weeks), or with simultaneous edits? How is the data transported? @noffle has been spearheading the process behind building this, and this week made a host of modules to resolve some of its dependencies:

bisecting-numbers - Integer-like number system where any number can be bisected to form infinite integer subsystems.
bisecting-between - Produces a unique value that sorts between two other given values.
hyperswarm - Create a p2p webrtc swarm around a hyperlog.

(He also wrote a little README generator to help him out with his JS module READMEs.)

pubsub

@noffle also published a couple of early mad science modules that enable a peer-to-peer overlay network for gossip-based message exchange. These aren't strictly built for IPFS, but are an experimental shim to enable projects like @haad's orbit-db to operate without a central server for message exchange between peers.

secure-gossip - Secure, transport agnostic, message gossip protocol. Any peer in the network can publish messages, which will eventually propagate to the entire network via rounds of gossip between each node's peers.
pubsub-swarm - Form a p2p swarm of nodes around a topic and exchange messages, with a traditional pubsub API.

go-ipfs

@whyrusleeping set up Teamcity. This cuts down on the long waits for Travis to run, and hopefully will mean faster CI tests. Teamcity also gives us awesome metrics on our tests and nice statistics on failures and failure rates. Teamcity has nice integration with a large array of test runners, from go tests to karma and sharness. It will give us more detailed feedback about our test runs, even when successful.

fc00

@lgierth spent a productive week in Paris, and chatted with @cjdelisle and @ansuz at @xwiki about the state and future of cjdns/fc00, layed out ideas for routing improvements, and drafted spec documents for the switch and cryptoauth layers. You can find those specs here (they'll be updated soon). Work will continue on these for the rest of March. The switch and routing layers of fc00 might be the foundation of a smarter swarm for IPFS/libp2p, so this is all very exciting.

Community

name-your-contributors

@RichardLitt gave a talk at BostonJS on Thursday to around fifty people, about how the name-your-contributors module is used in this newsletter. The theme of the meetup was community building; if you have any ideas for improving the Contributor list below, check out this repository and let us know. Since the talk, @RichardLitt also added get-pr-creators to the module, so that everyone who opens a PR (even if it isn't merged) gets added to the contributors list. Has your name been included?

dignifed hacks

@dignifiedquire launched a live-stream of him coding, which he is calling "dignified hacks". Last Monday he recorded himself doing a new feature for the WebUI in the first episode. He'll do another one this week. One of the viewers, @nginnever, said it "was helpful for a quick view of our components and data flow in the webui." He will announce regular showtimes on Twitter and you can subscribe to IPFS on YouTube where future episodes will be hosted.

Lisbon Blockchain Workshop

@diasdavid participated on the last Lisbon Blockchain Workshop on March 5, hosted by Kwamecorp. The workshop gathered many Blockchain, IPFS, ethereum and zerocash enthusiasts that were organized in working groups to build solutions with these distributed technologies.

IPFS Dead drop

Some members of c-base have written a dead drop-like system that automatically uploads files from a USB memory stick to IPFS. When you plug a USB memory into the device, it will automatically access the memory stick and publish all the files on the web. Thanks to IPFS the files are instantly available to the whole world. Check out deaddrops.com for more information about dead drops.

They also have an installation available: the node is run in the above device. When you plug it in, you get to see this, too. If you're in Berlin or visiting soon, make sure to drop by.

Interplanetary Wayback (IPWB)

At Archives Unleashed, a Web Archive Hackathon in Toronto, Mat Kelly (@machawk1) and Sawood Alam (@ibnesayeed) from ODU WSDL Research Group developed an IPFS based distributed and persistent archive replay system called Interplanetary Wayback.

Science Data

@jbenet visited the Janelia Research Campus to learn about scientific data tools and use cases. He gave a talk about IPFS, data versioning, package management, and more (video forthcoming). He learned about DVID, and the requirements of the amazing FlyEM brain imaging effort. He got to see many fantastic open source research tools made by the FlyEM Team (github), the Freeman Lab (github), and other groups. Go check out their GitHub repos, and help them improve brain research!

Press

Jeff Smith of Sitepoint released a great article on IPFS: "HTTP vs IPFS: is Peer-to-Peer Sharing the Future of the Web?".

Contributors

Across the entire IPFS GitHub organization, the following people have committed code, created issues, or made a comment on GitHub between February 29th (noon, GMT) and March 7th. We're autogenerating this list using this tool and this other tool, so please let us know if your name isn't here.

@adrian-bl (Adrian Ulrich)
@amstocker (Andrew Stocker)
@anarcat (anarcat)
@Beligertint
@bergie (Henri Bergius)
@cevin (cevin)
@chriscool (Christian Couder)
@cinderblock (Cameron Tacklind)
@clkao (Chia-liang Kao)
@daveajones (Dave Jones)
@davidar (David A Roberts)
@diasdavid (David Dias)
@dignifiedquire (Friedel Ziegelmayer)
@greenkeeperio-bot (Greenkeeper)
@hjoest (Holger Joest)
@jbenet (Juan Benet)
@jedahan (Jonathan Dahan)
@knocte (Andres G. Aragoneses)
@Kolomona (Kolomona Myer)
@Kubuxu (Jakub Sztandera)
@lgierth (Lars Gierth)
@mappum (ᴍᴀᴛᴛ ʙᴇʟʟ)
@mattseh
@MichaelMure (Michael Muré)
@micxjo (Micxjo Funkcio)
@mildred (Mildred Ki'Lya)
@moritz121
@NeoTeo (Teo Sartori)
@Neurone (Giuseppe Bertone)
@nginnever (Nathan Ginnever)
@noffle (Stephen Whitmore)
@peteygao (Peter Gao)
@randomshinichi
@RichardLitt (Richard Littauer)
@richardschneider (Richard Schneider)
@sivachandran (Sivachandran)
@suisha (David Mai)
@thelinuxkid (Andres Buritica)
@tinybike (Jack Peterson)
@whyrusleeping (Jeromy Johnson)
@xicombd (Francisco Baio Dias)
@yangwao (Matej Nemček)
@yncyrydybyl (Yan Minagawa)
@zignig (Simon Kirkby)
@Zogg

This newsletter is also a community effort. If you have cool things to share for the next weekly, drop a comment about it in the weekly repo! The more people mention items they want to see in the weekly there, the easier it is to make this and send it out.

Thanks, and see you next week!

Richard Littauer and Andrew Chin

Submit feedback about this issue here, or send us feedback about the IPFS Weekly in general.

IPFS Weekly 5

2016-03-04T00:00:00+01:00

If you would like to get this update as an email, sign up for our weekly newsletter!

This weekly covers the last month. Here are some of the highlights for the February sprints!

Highlight

Our friends and users at ConsenSys wrote an excellent "Introduction to IPFS", which starts with a less-technical preface, and then dives deep into a full explanation of how the IPFS object model works. It walks through multiple examples, including directory structures, version control systems, and blockchains. This is a great post to familiarize yourself with how the low level IPFS objects work, with graph visualizations and in-depth explanations. It also features a fantastic cover image! Thanks @ChrisLundkvist and @ConsenSysAndrew!

Updates

go-ipfs

@lgierth improved the automated docker image for go-ipfs, which is now named ipfs/go-ipfs. Give it a try: docker run -i --net=host ipfs/go-ipfs. The new image is built automatically for every commit, and has automatic tags for the coming releases. It's also a few MB smaller, and doesn't require mounting a volume for the IPFS repository anymore. Instead, if there's no volume mounted, it'll generate an ephemeral identity and configuration which will be lost when the container exits. This is ideal for testing or just trying something out real quick. Images for go-ipfs 0.3.x are not provided, because the Dockerfile shipped with these version isn't suitable for Docker Hub automatic builds.
@noffle, @whyrusleeping and @chriscool have been improving the test suite, fixing flaky tests which used to fail inaccurately. This will let us merge PRs faster, as CI should correctly categorize success and failure.
@noffle also made progress toward getting go-ipfs compatible with vanilla go get installation. Hope to have more tangible results to show by end of the month.
@RichardLitt made a lot of documentation fixes to the CLI while working on the IPFS HTTP API.
@chriscool rewrote the build documentation for Windows, which enabled @slothbag to get Windows builds working again with minimal fuss.
@lgierth posted a writeup on the migration from 0.3.x to 0.4.0.
@noffle improved FUSE connection termination.

GX

@whyrusleeping shipped a PR that introduces the use of a tool called gx, for vendoring our project dependencies. Previously we used godeps, and saved all of the code required indirectly to build ipfs in the go-ipfs repository itself. This was very difficult to work with for a few different reasons, first off, it made the size of the repository bloat way more than the original size of our codebase, causing clones to take longer, and making CI slower all around. Second, updating these dependencies was a hassle: this was partially the fault of go's poor package management choices and partially that we found godeps UX to be unfriendly. To solve the problem, @whyrusleeping create gx. Gx is a package management tool based on ipfs. Package references are all hashes linked in a merkletree, and resolving all the dependencies of a given project is as easy as an ipfs fetch. Now that we are using gx, the main go-ipfs repo is much smaller, dependencies can be easily fetched and installed (and shared across projects), and we also get to essentially bootstrap ipfs with ipfs.

From the average users perspective, there are a few small changes; go get is no longer a viable way to install ipfs, and users will now need to run make install as is common on other large golang projects (docker and kubernetes among them). You can read more about gx over here at its repo: github.com/whyrusleeping/gx and about gx-go (the subtool of gx specifically for go) here: github.com/whyrusleeping/gx-go.

@RichardLitt improved the gx README document to better explain its goals, and help people get started. Take a look and give us feedback through an issue if you think anything could be improved.

ipns-pub

@whyrusleeping wrote a tool called ipns-pub to allow people to publish IPNS entries to the network without actually running a node. You can generate keypairs with ipfs-key and then use those keys to publish any ipfs path you like. Be aware though that entries published with this tool expire every 24 hours, so to keep them alive on the network, the tool has a --daemon option that will republish your entry every twelve hours automatically.

specs

The IPLD spec was merged, after months of thoughtful design. The bulk of the work was achieved by @mildred and @jbenet, with lots of comments and design opinions from many other contributors. The IPLD spec. The "thin-waist" Merkle DAG format, defines merkle-links, -dags, and -paths, as well as the IPLD Data model and formats. In short: JSON documents with named merkle-links that can be traversed. Stay tuned for more information in the future.

http-api-spec

@RichardLitt finished logging all existing ipfs commands for the HTTP API Spec. This means that if you have any questions about how the HTTP API should work, you can see them either in master or as open PRs to that issue. If you are interested in how the HTTP API should work, or have any specific questions, please see the current version, and explore the open PRs on that repo.

distributions

dist.ipfs.io almost has signed releases thanks to @dignifiedquire. This will be coming soon.

js-ipfs

Thanks to @diasdavid, the DAG object manipulation commands now work, with tests and all. As well, thanks to @dignifiedquire's efforts, the js-ipfs API now returns promises if there isn't a specified callback, allowing both major methods used by the Javascript community to work equally well.

registry-mirror

@diasdavid improved registry-mirror performance and robustness by removing the dependency on registry-static. The few necessary parts were copied in. This is a huge step in reliability and performance cloning the registry with registry-mirror.

station

@dignifiedquire fixed drag and drop file uploading, and some dependency issues. So go ahead and try it out.

ipfs-geoip

@dignifiedquire rewrote the generation script and cleaned up the code so that now the data is consistently reproducible and fully stored on IPFS. This ensures that geoip lookups over IFPS will work in all future releases.

fs-repo-migrations

@chriscool improved important tests for fs-repo-migrations -- tests verify more edge cases when migrating forward and backward, through various sample workloads.

ipfs-hyperlog

@noffle built ipfs-hyperlog, and ipfs-compatible fork of hyperlog, a DAG that replicates based on scuttlebutt logs and causal linking. ipfs-hyperlog is a drop-in replacement for @mafintosh's hyperlog. Its key difference is that it creates a Merkle DAG that is binary compatible with IPFS objects. This means any node of any DAG built using ipfs-hyperlog can be replicated to and from the IPFS network as well!

Logo

@Kubuxu worked on a new IPFS logo. Check it out.

Community

If you have any suggestions for conferences, there is a new issue where you can submit your proposals. The IPFS community will see if we can get to that conference and have a presence there.
We now have a list of community events for IPFS on the ipfs/community README. Do you have any that you'd like to add? Check out past events, too.
@RichardLitt proposed using the new GitHub Templates for IPFS. What do you think about this idea? Chime in here.

IPFS In The Wild

Marmot Image Checker: This takes an image, asks the Google Cloud Vision API for three descriptions of the image, and compares those descriptions to a chosen list of words. If there is match, the image is added to the toadserver, a smart contract and IPFS-Backed download server. From Twitter
asciienema now allows playback on IPFS. From Twitter

Contributors

Across the entire IPFS GitHub organization, the following people have committed code, created issues, or made a comment on GitHub between February 1st (noon, GMT) and February 29th. We're autogenerating this list using this tool and this other tool, so please let us know if your name isn't here.

@abacon (Bacon)
@almereyda (jon r)
@amstocker (Andrew Stocker)
@anacrolix (Matt Joiner)
@anarcat (anarcat)
@Ape (Lauri Niskanen)
@area
@ARezaK
@Asgraf (Michal Turek)
@Balancer (Balancer)
@balupton (Benjamin Lupton)
@bierlingm
@BigBlueHat (BigBlueHat)
@boergsen
@boxxa (Boxxa)
@briantigerchow (Brian Tiger Chow)
@brimstone (Matt)
@bussiere (bussiere)
@bzz (Alexander)
@chpio
@chris-martin (Christopher Martin)
@chriscool (Christian Couder)
@christianlundkvist (Christian Lundkvist)
@ChristopherA (Christopher Allen)
@cjcase (Cj Case)
@cleichner (Chas)
@codeburd
@Crest (Crest)
@cryptix (Henry)
@daveajones (Dave Jones)
@David-Leudolph (David Leudolph)
@david415 (David Stainton)
@davidar (David A Roberts)
@denisnazarov (Denis Nazarov)
@dginev (Deyan Ginev)
@diasdavid (David Dias)
@dignifiedquire (Friedel Ziegelmayer)
@djdv (Dominic Della Valle)
@dominictarr (Dominic Tarr)
@Dumptel
@dylanPowers (Dylan Powers)
@edent (Terence Eden)
@ehd (Stephan Seidt)
@EliasGabrielsson (Elias Gabrielsson)
@emardee
@eminence (Andrew Chin)
@faebser (Fabian Frei)
@fazo96 (Enrico Fasoli)
@Fil (Fil)
@frabrunelle (Francis Brunelle)
@GitCop
@GravisZro (Gravis)
@greenkeeperio-bot (Greenkeeper)
@halseth
@harlantwood (Harlan T Wood)
@hitchcott (Chris Hitchcott)
@hosh (Ho-Sheng Hsiao)
@hutenosa
@IanCal (Ian Calvert)
@ianopolous (Ian Preston)
@ingokeck (Ingo Keck)
@insanity54 (Chris Grimmett)
@ion1 (Johan Kiviniemi)
@j-h-scheufen
@j4nu5 (Kushagra Sinha)
@jamescarlyle (James Carlyle)
@jbenet (Juan Benet)
@jbshirk (Joe)
@jedahan (Jonathan Dahan)
@jeffscottward (Jeff Ward)
@jefft0 (Jeff Thompson)
@johncant (John Cant)
@kalmi (Tarnay Kálmán)
@Kolomona (Kolomona Myer)
@krl (kristoffer)
@Kubuxu (Jakub Sztandera)
@kyledrake (Kyle Drake)
@kyrias (Johannes Löthberg)
@lamarpavel
@lernisto (Terrel Shumway)
@lgierth (Lars Gierth)
@lidel (Marcin Rataj)
@lockedshadow
@lovelaced
@mappum (ᴍᴀᴛᴛ ʙᴇʟʟ)
@matshenricson (Mats Henricson)
@MichaelMure (Michael Muré)
@micxjo (Micxjo Funkcio)
@mildred (Mildred Ki'Lya)
@mindhog
@Mithgol
@mnp (Mitchell Perilstein)
@montagsoup
@MrChrisJ (Chris Ellis)
@mseri
@NDuma (NDuma)
@nginnever (Nathan Ginnever)
@NightRa (Ilan Godik)
@NodeGuy (David Braun)
@noffle (Stephen Whitmore)
@odipar (rapido)
@olizilla (Oli Evans)
@palesz (Palesz)
@palkeo (palkeo)
@parkan (Arkadiy Kukarkin)
@peerchemist
@pietsch (Christian Pietsch)
@prusnak (Pavol Rusnak)
@randomshinichi
@reit-c
@rethore (Pierre-Elouan Réthoré)
@rht
@RichardLitt (Richard Littauer)
@richardschneider (Richard Schneider)
@rsynnest
@rubiojr (Sergio Rubio)
@rwcarlsen (Robert Carlsen)
@se3000 (Steve Ellis)
@Shaaah (Shaaah)
@shtukas (Pascal Honoré)
@sivachandran (Sivachandran)
@sleep-walker (Tomáš Čech)
@slothbag
@Stebalien (Steven Allen)
@suisha (David Mai)
@tcyrus (Timothy Cyrus)
@thelinuxkid (Andres Buritica)
@tidux (Todixu)
@tinybike (Jack Peterson)
@tomgg (tmg)
@tv42 (Tv)
@void4
@wanderer
@wasserfuhr (RainerWasserfuhr)
@whyrusleeping (Jeromy Johnson)
@xicombd (Francisco Baio Dias)
@yangwao (Matej Nemček)
@zignig (Simon Kirkby)
@Zogg

This newsletter is also a community effort. If you have cool things to share for the next weekly, drop a comment about it in [the weekly repo](//github.com/ipfs/weekly! The more people mention items they want to see in the weekly there, the easier it is to make this and send it out.

Richard Littauer and Andrew Chin

Submit feedback about this issue here, or send us feedback about the IPFS Weekly in general.

Migrating ipfs.io from go-ipfs 0.3.x to 0.4.0

2016-02-12T00:00:00+01:00

Good news everyone! We'll soon release go-ipfs 0.4.0, which contains lots of great changes and enhancements. An upcoming post will detail the improvements of 0.4.0. This post is about important things to be aware of when upgrading. Users who already rely on go-ipfs or the ipfs.io services should pay special attention.

There are breaking changes which prevent 0.4.x nodes from communicating with 0.3.x nodes. We know breaking changes are painful. We avoid them. In this case, the improvement makes IPFS substantially more upgradable. It's one of those things that "should've been different" from the start. Having the freedom to improve the core protocol is why IPFS is still in alpha phase.

Nevertheless, we take service disruption very seriously even in these early days. We know thousands of developers use IPFS, and hundreds of thousands of people rely on IPFS links to our gateway. So we planned a smooth upgrade path.

The breaking change is at the wire protocol level, which means there will be, and already are, two separate networks. We'll call these networks v03x and v04x. IPFS nodes built from the master branch are already part of v04x. The Docker image ipfs/go-ipfs is built from master (until 0.4.0 is released) and thus also part of v04x.

ipfs.io provides two essential services to the community, which are affected by this: the public HTTP-to-IPFS gateway and the default bootstrappers. We'll continue to support them for the v03x network until the end of April 2016.

Please note that we won't support 0.3.x with patches or new features. All development effort is directed towards 0.4.0, and you should update as soon as possible.

How do I update go-ipfs to 0.4.0?
The public gateways at ipfs.io
Solarnet: the default bootstrappers
How does this affect me?

How do I update go-ipfs to 0.4.0?

Until we release 0.4.0, you can update by building from source.

After the release, you can:

Use the brand new ipfs-update tool to update go-ipfs (recommended)
Download the new go-ipfs binaries and install them.
Download the new source and build it yourself.

Please note that installation with go get does not work at this time. We are experimenting with gx, the new IPFS-based package manager.

The public gateways at ipfs.io

For the time being, https://ipfs.io uses both v03x and v04x to service requests. Content available in either network will be served just fine. Whichever responds successfully first (2xx or 3xx status code), gets to serve its content. The other response is discarded. If the first response is not successful (4xx/5xx, connection errors), it is discarded and the second response is served, regardless of its status code.

We're using the multireq proxy to accomplish this multiplexing behaviour. multireq is a new tool, so if you notice any weirdness with gateway requests, please let us know.

If you want to target a specific network, use v03x.ipfs.io or v04x.ipfs.io. These domain names will stay around as long as the respective network is supported by the public gateway.

All of the above also applies to the /api endpoint (the readonly API), which is part of the gateway.

Expect the v03x gateway to be supported until the end of April 2016. After that day, ipfs.io will be served by v04x exclusively, and v03x.ipfs.io will no longer work.

Solarnet: the default bootstrappers

We call the 8 bootstrap nodes in go-ipfs's default config the default bootstrappers. Their PeerIDs start with QmSoL (for Solarnet), and all of them use /tcp/4001 and/or /udp/4002/utp. Use ipfs bootstrap to see or modify the bootstrappers currently used by your IPFS node.

In order to balance the default bootstrappers over v03x and v04x, a few of them bind v04x to these ports, and a few bind v03x. The respective other network is bound to /tcp/14001 and /udp/14002/utp. This means that all 8 hosts run bootstrappers for both networks, but are available as default bootstrappers only to one.

We'll gradually shift the v03x/v04x ratio to v04x. Expect at least two v03x bootstrappers to be supported until the end of April 2016.

How does this affect me?

Check which version of go-ipfs you are running: ipfs version

If you rely on the public gateways, we ask you to please upgrade to 0.4.0 as soon as you can. You can subscribe to this issue to be notified when 0.4.0 is released.

If you're running an 0.3.x node, it won't be able to communicate with any node which has updated to 0.4.x, let alone exchange data. You can access data added by 0.4.0 nodes using the public gateway. Likewise, data added by your 0.3.x node is still available on the public gateway.

If you're running an 0.4.0 node, all is well in the world. You don't need to do anything.

Thank you for bearing with this important change. We seek to provide a smooth transition for everyone, and wish we didn't have to bother you with this at all.

IPFS Weekly 4

2016-02-05T00:00:00+01:00

If you would like to get this update as an email, sign up for our weekly newsletter!

Here are some of the highlights for the January 25th sprint:

Updates

js-ipfs: @diasdavid and @vijayee created js-ipfs-merkle-dag and js-ipfs-blocks, which will help IPFS deal with MerkleDAGs and are extensible enough to allow others to build their own MerkleDAG structures. Also, js-ipfs is now able to import a file into the MerkleDAG, an important milestone. An architecture graph has been spec'ed out here to help newcomers understand how js-ipfs works. Finally, lots of new issues and low-hanging fruit have been tagged for contributors to tackle. idb-plus-blob-store: The substack/idb-blob-store had issues regarding emitting the finish event on the createWritableStream result due to internal issues in streams and the way .end is handled. So @dignifiedquire wrote this module that fixes this and now allows us to test all parts of js-ipfs in the browser using IndexedDB as the storage.
go-ipfs: Thanks to @Kubuxu, dnslink was extended by dedicated subdomain (_dnslink.). This allows users to alias (using CNAME) their domain to gateway.ipfs.io while still being able to to set the dnslink to content they wish. Also, @whyrusleeping has an open PR to pull libp2p out of go-ipfs and put it into a module on gx, the IPFS native package manager. This is part of a larger effort to whittle go-ipfs down into smaller, extensible modules. Finally, @whyrusleeping did a clean up of the go-ipfs pull requests, closing all pull requests or pinging their authors for updates as needed.
notes: @noffle began a discussion about revamping the ipfs mount interface. Join in to voice your thoughts here.
community: @diasdavid has written a guide to writing captain.logs, which are short gists written by maintainers about IPFS projects, their status, and how to help. We plan to have more of these in different projects.

Community

Lisbon: @diasdavid hosted a Lisbon IPFS Sunday Hack Day this week. A few people came, and @xicombd made this Chrome Extension, which lets you access IPFS urls from your local IPFS daemon (source code).
Coffee Talks: @whyrusleeping has hosted another "Coffee Talk", this time on gx, the IPFS package manager.
NYC Meet-up: A NYC meet-up is planned for the end of February. Help us plan it and show your interest here.
Desert Blockchain: IPFS was presented at the last Desert Blockchain miniconference.

Tools

There was an unfortunate GitHub outage and many people started using IPFS to move around their repos, with @whyrusleeping's simple git-ipfs-rehost (view on IPFS) and @cryptix's excellent git-remote-ipfs (view on IPFS).
@pipermerriam created ipfs-persistence-consortium to create networks of IPFS nodes which persist each other's content. This is similar to @victorbjelkholm's pincoop. Perhaps there can be some cooperation in these efforts? 😎
The FullNode Project -- which includes IPFS, Tor, Bitcoin, OpenVPN and more -- is having a crowdfund -- go get one! Thanks for making it @MrChrisJ!

Other

It was Ralph Merkle's birthday on February 2nd! He is the creator of the Merkle tree, which are a crucial part of how IPFS works. Happy Birthday, Ralph! 🎂
Our friends at Mine are using IPFS for the excellent Mediachain metadata protocol (protocol proposal here), a protocol to track media creation, attribution, and more. They have published several great articles about it. This week @denisnazarov wrote about The GIF That Fell To Earth, and @parkan gave a Developer Update that discusses pHash, IPLD, and more.
@lexansoft created a new Ethereum name registrar called EtherID (repo here), which uses IPFS for storing users' content. Then @btsfav wrote an article about Personal Websites on EtherID with IPFS.

Contributors

Across the entire IPFS GitHub organization, the following people have committed code, created issues, or made a comment on GitHub between January 25th (noon, GMT) and February 1st. We're autogenerating this list using this tool, so please let us know if your name isn't here.

@alexAubin (Alexandre Aubin)
@andreiamatuni (Andrei Amatuni)
@area
@AtnNn (Etienne Laurin)
@bdunlay (Brian Dunlay)
@BigBlueHat (BigBlueHat)
@chriscool (Christian Couder)
@ConsciousCode (Conscious Code)
@cryptix (Henry)
@davidar (David A Roberts)
@diasdavid (David Dias)
@dignifiedquire (Friedel Ziegelmayer)
@dysbulic (Will Holcomb)
@eminence (Andrew Chin)
@fazo96 (Enrico Fasoli)
@GitCop
@greenkeeperio-bot (Greenkeeper)
@harlantwood (Harlan T Wood)
@Hexagon6
@IanCal (Ian Calvert)
@ion1 (Johan Kiviniemi)
@JAremko
@jbenet (Juan Benet)
@jedahan (Jonathan Dahan)
@kazarena
@kpcyrd
@Kubuxu (Jakub Sztandera)
@lgierth (Lars Gierth)
@lidel (Marcin Rataj)
@lockedshadow
@Luzifer (Knut Ahlers)
@MartinThoma (Martin Thoma)
@MichaelMure (Michael Muré)
@mildred (Mildred Ki'Lya)
@mindhog
@Mithgol
@mortonfox (Morton Fox)
@MrChrisJ (Chris Ellis)
@NDuma (NDuma)
@NeoTeo (Teo Sartori)
@nikhilshekhawat
@noffle (Stephen Whitmore)
@palesz (Palesz)
@Patagonicus (Philipp Adolf)
@ralphbean (Ralph Bean)
@randomshinichi
@rht
@RichardLitt (Richard Littauer)
@Shaaah (Shaaah)
@sivachandran (Sivachandran)
@slothbag
@thelinuxkid (Andres Buritica)
@tilgovi (Randall Leeds)
@tommg (Thomas Gardner)
@VertigoRay (Raymond Piller)
@w33tmaricich (Alexander Maricich)
@whyrusleeping (Jeromy Johnson)
@whyun7892
@willeponken (William Wennerström)
@willglynn (Will Glynn)
@xicombd (Francisco Baio Dias)

Thanks, and see you next week! If you have cool things to share for the next weekly, drop us a line in the next weekly sprint issue!

Richard Littauer and Andrew Chin

Submit feedback about this issue here, or send us feedback about the IPFS Weekly in general.

IPFS Weekly 3

2016-02-01T00:00:00+01:00

If you would like to get this update as an email, sign up for our weekly newsletter!

This is a double weekly: here are some of the highlights for the January 12th and the January 19th sprints:

Updates

dist.ipfs.io A distributions page has been shipped! This was largely spearheaded by @dignifiedquire. This is the new one-stop-shop for finding and downloading all official binaries that IPFS produces. Each project has:
The distribution name and a short description;
The current version number and release date;
A download button that detects your platform and automatically suggests the appropriate distribution for you;
A grid with download links for all supported platforms (operating system and architectures);
A Changelog, a link to a summary of all version changes;
An All Versions, a link to view and download previous versions.

The site is also hosted on IPFS, and is used by ipfs-update to update IPFS. Check it out at http://dist.ipfs.io. go-ipfs 0.3.11 is shipped! Also, we merged 0.4.0 into master. @jbenet, @Dignifiedquire and others fixed appveyor. And finally the latest webui by @Dignifiedquire was pushed in 0.3.11. Check out the changelog. js-ipfs-merkle-dag @diasdavid worked on merkle-dag implementation interoperabolity, by enabling you to successfully read (and write) MerkleDAG nodes that were stored into an IPFS Repo through go-ipfs, in JavaScript. py-ipfs Thanks to @candeira and @ivilata, we now have a much more filled out plan for the Python implementation of IPFS. Join in if you are a pythonista. js-mafmt @whyrusleeping is working on a multiaddr validator in js, after his go work in mafmt infrastructure The public gateway at ipfs.io is now backed by both v0.4.0-dev and v0.3.11 at the same time. Requests are proxied to both, and the first successful response wins. ipfs.io will be backed by v0.4.x alone eventually. What this means is that if you want to target your requests at one specific version, use v04x.ipfs.io or v03x.ipfs.io, respectively. The latter will stop returning DNS records when v03x isn't supported on the public gateway. Also, while we're releasing v0.4.0 and for a while after, the default bootstrap nodes are divided among v03x and v04x. Most can be used by v04x nodes, and a few by v03x nodes. The latter will be phased out over time. ipfs-firefox-addon The Firefox addon that provides transparent access to IPFS resources via local HTTP2IPFS gateway has been updated to release 1.4.2. This is the first time we've mentioned this in this newsletter; go check it out. The addon enables you to load content via IPFS instead of HTTP. For example, if you open a resource from a public gateway (eg. https://ipfs.io/ipfs/QmW2WQi7j6c7UgJTarActp7tDNikE4B2qXtFCfLPdsgaTQ/cat.jpg) and have addon enabled, the request will not hit public gateway, but data will be loaded from IPFS swarm, in a decentralized and distributed way. It also now supports the IPFS protocol scheme, meaning that can look up a hash by going to fs:/ipfs/<hash> directly. go-multiaddr go-multiaddr now supports the adding of new protocols on the fly. This allows us to build in new protocols without having to force everyone to update the multiaddr library itself. community The labels 'help wanted' and 'difficulty:easy', ':hard', and ':moderate' have been added to every IPFS repository, to help out contributors know what are some low-hanging fruit they can help with. As well, @RichardLitt has created a Docs Styleguide to help standardize how we use English in all of our tools. It'll grow as time goes on. ipfs-dht @whyrusleeping wrote a 'just a dht' binary. This means you can help out the network without running a full ipfs node. It also takes less memory and bandwidth, and having more dht nodes will speed up lookups. js-ipfsd-ctl The Node module able to control an IPFS daemon was updated to 0.3.11. * webui The webui no long depends on jQuery, thanks to a lot of work from @luigiplr, @travisperson, and @dignifiedquire.

Work in Progress

js-ipfs @diasdavid has merged in the roadmaps for js-ipfs and libp2p, which he'll update as more details become apparent. Take a look.
weekly @RichardLitt put a lot of work in getting various sorts of attribution for contributors to IPFS, currently available as name-your-contributors. This is almost done, but there are a few tweaks left to do.

Meetups and Conferences

ArcticJS @diasdavid, @whyrusleeping, @RichardLitt, and @noffle were all able to meet up for the first time as a group and hack together at ArcticJS, the first JavaScript conference above the Arctic circle, in Svalbard. Around 15 attended, and there was a lot of hacking, snow, talking about buffers, and swimming in the ocean at midnight underneath the Aurora Borealis (no, really). Most of the work done at the conference was conversations, but included @diasdavid and @RichardLitt working on the API together, @whyrusleeping learning how to javascript, and @diasdavid and @xicombd working on IPFS together. All in all, it was an incredible trip. The videos for the actual talks (given in the most Northernmost church in the world) are here.

Shoutouts

Joe Armstrong, inventor of Erlang, mentioned the importance of permanence and IPFS in a recent, and fantastic series of posts about organizing software. Thanks for the kind mention, Joe!

Contributors

Across the entire IPFS GitHub organization, the following people have committed code, created issues, or made a comment on GitHub between January 11th and January 25. (We're autogenerating this list using this tool, so please let us know if your name isn't here.)

@adrian-bl (Adrian Ulrich)
@alexeicolin
@ali (Ali Ukani)
@amstocker (Andrew Stocker)
@anarcat (anarcat)
@ansuz (ansuz)
@Ape (Lauri Niskanen)
@area
@AtnNn (Etienne Laurin)
@Azulan (Frank Flores)
@Balancer (Balancer)
@bcg-didi
@benjaminbollen (Benjamin Bollen)
@brailateo (Constantin Teodorescu)
@btrask (Ben Trask)
@CaioAlonso (Caio Alonso)
@chpio
@chriscool (Christian Couder)
@ConsciousCode (Conscious Code)
@CrowdHailer (Peter Saxton)
@dandart (Dan Dart)
@david415 (David Stainton)
@davidar (David A Roberts)
@diasdavid (David Dias)
@digital-dreamer
@dignifiedquire (Friedel Ziegelmayer)
@djdv (Dominic Della Valle)
@dylanPowers (Dylan Powers)
@eminence (Andrew Chin)
@Faleidel
@fazo96 (Enrico Fasoli)
@findkiko
@Foxcool (Alexander Babenko)
@GitCop
@goog (Jay Cheng)
@greenkeeperio-bot (Greenkeeper)
@harlantwood (Harlan T Wood)
@ikreymer (Ilya Kreymer)
@jbenet (Juan Benet)
@jbshirk (Joe)
@jedahan (Jonathan Dahan)
@Kubuxu (Jakub Sztandera)
@kyledrake (Kyle Drake)
@lgierth (Lars Gierth)
@lidel (Marcin Rataj)
@lovelaced
@luigiplr (Luigi Poole)
@Luzifer (Knut Ahlers)
@matshenricson (Mats Henricson)
@MChabez
@Mec-iS (Lorenzo)
@MichaelMure (Michael Muré)
@mildred (Mildred Ki'Lya)
@Mithgol
@NDuma (NDuma)
@NeoTeo (Teo Sartori)
@nginnever (Nathan Ginnever)
@nicola (Nicola Greco)
@NightRa (Ilan Godik)
@noffle (Stephen Whitmore)
@Patagonicus (Philipp Adolf)
@peerchemist
@peteygao (Peter Gao)
@PlanetPlan
@pra85 (Prayag Verma)
@prusnak (Pavol Rusnak)
@ralphtheninja (Lars-Magnus Skog)
@ReadmeCritic
@Red5d
@reit-c
@rht
@RichardLitt (Richard Littauer)
@richardschneider (Richard Schneider)
@rugk (rugk)
@SCBuergel (Sebastian C. Bürgel)
@seclorum (seclorum)
@thelinuxkid (Andres Buritica)
Thomas Gardner
@travisperson (Travis Person)
@tv42 (Tommi Virtanen)
@void4
@WeMeetAgain (Cayman)
@whyrusleeping (Jeromy Johnson)
@windemut
@wking (W. Trevor King)
@xicombd (Francisco Baio Dias)
@zignig (Simon Kirkby)

Thanks, and see you next week! If you have cool things to share for the next weekly, drop us a line in the next weekly sprint issue!

Richard Littauer and Andrew Chin

Submit feedback about this issue here, or send us feedback about the IPFS Weekly in general.

IPFS Weekly 2

2016-01-13T00:00:00+01:00

Here are some of the highlights for the January 5th Sprint:

Updates

(specs) There was much debate in the IPLD spec about pathing notation. @mildred in particular did a lot of good work.
(go-ipfs) The IPFS Docker image is now tested automatically on Travis, thanks to @ChristianKniep, @whyrusleeping, and @chriscool. For more, see the go-ipfs#2064 pull request.
(js-ipfs) Now you can do npm i -g js-ipfs and use jsipfs (the javascript impl of IPFS) with bootstrap + id + version commands, fully compatible with the go-ipfs repo, thanks to @diasdavid!
(project-repos) A very cool organization-wide dashboard that provides a status overview of all the IPFS repos was launched. By @harlantwood.
(community) @RichardLitt merged the pull request for JavaScript guidelines for IPFS repos. This is a major step towards standard JavaScript repositories within IPFS.
(community) @NeoTeo, @whyrusleeping, and @diasdavid hosted a small meetup in Copenhagen! Eight people in total showed up for an intro to IPFS, a Q&A session, and some good conversation!

Work in Progress

(distributions) @Dignifiedquire further developed the distributions page. Click here for a preview!
(go-ipfs) @lgierth continued the work towards the dev040 migration. Notably we have two new gateways to aid in the transition: http://v04x.ipfs.io and http://v03x.ipfs.io
(archives) @Dignifiedquire has added to IPFS a dump of the Stackexchange archives! Details are in the archives repo. We are always looking for more help with archiving important datasets, so feel free to join us in the archives repo!

Contributors

Across the entire IPFS GitHub organization, the following people have committed code since January 4th. (We're autogenerating this list using this tool, so please let us know if your name isn't here.) In the future, we will also include people who comment, as they are also super important; bear with us while we develop that technology.

@chriscool (Christian Couder)
@ChristianKniep (Christian Kniep)
@diasdavid (David Dias)
@Dignifiedquire (Friedel Ziegelmayer)
@eminence (Andrew Chin)
@greenkeeperio-bot
@ianopolous (Ian Preston)
@jbenet (Juan Benet)
@Kubuxu (Jakub Sztandera)
@lgierth (Lars Gierth)
@noffle (Stephen Whitmore)
@ralphtheninja (Lars-Magnus Skog)
@ReadmeCritic
@RichardLitt (Richard Littauer)
@whyrusleeping (Jeromy Johnson)
@yuvallanger (Yuval Langer)

Thanks, and see you next week! If you have cool things to share for the next weekly, drop us a line in the next weekly sprint issue!

Richard Littauer and Andrew Chin

Submit feedback about this issue here, or send us feedback about the IPFS Weekly in general.

IPFS Weekly 1

2016-01-11T00:00:00+01:00

Welcome to the first edition of IPFS Weekly!

IPFS is a new hypermedia distribution protocol, addressed by content and identities, aiming to make the web faster, safer, and more open. In these posts, we will try to highlight some of the development that happened in the past week. For anyone looking to get involved, follow the embedded hyperlinks, search the wealth of information on github or join us on IRC (#ipfs on the Freenode network).

Since this is our first time launching the Weekly, we've included several past weeks. This is partially because we've been refining our process, and wanted to make the first weekly a great one. In the future, they will be released weekly. If you have any feedback about this process in general, let us know here. Thanks!

December 21

Here are some of the highlights for the December 21 Sprint:

32c3

Not too much happened during this sprint, because it was the holidays - however, it was also the 32nd CCC. @whyrusleeping, @diasdavid, @lgierth, @Dignifiedquire and more of the team were over there in Hamburg.

@whyrusleeping gave a lightning talk (see above).
@whyrusleeping @diasdavid and @Dignifiedquire led an install party and live demo.
@MrChrisJ led a discussion about his Full Node project, which includes IPFS.
We met lots of our users, and other groups (including Tahoe-LAFS, tor, etc).

Updates

(go-ipfs) @whyrusleeping and @diasdavid did amazing work closing around 200 issues as part of the "Great go-ipfs Issue Party of 2015".
(registry-mirror) npm on IPFS was furthered by @diasdavid by making it work with 0.4.0. Currently, there are 130,000 modules and it's going strong.
(go-ipfs) @lgierth set up namaka and hiiaka (NUC nodes at our table at 32c3). This is pretty cool. CF. ipfs/infrastructure#142.
(readme-standard) @RichardLitt had his own great issue party. He's most proud of breaking ground on the future readme standard for IPFS.
(webui) @Dignifiedquire did some awesome work improving the webui using webpack.
He also managed to release rust-multiaddr and rust-multihash, which means more languages now implement the orginal multihash and multiaddr specs.

For more updates, see the sprint issue.

December 14

Here are some of the highlights for the December 14 Sprint:

Releases

@whyrusleeping shipped fs-repo-migrations! These are migrations for the filesystem repository of ipfs clients. This tool is written in Go, and developed alongside go-ipfs, but it should work with any repo conforming to the fs-repo specs.
@Dignifiedquire did some great work with rust-multiaddr, which is a Rust implementation of @jbenet's multiaddr.

Updates

(registry-mirror) @diasdavid worked on the npm on IPFS project. This involved some new features, moving the mirror to a different server, and making it work better with larger dirs and with 0.4.0.
[(js-ipfs)](//github.com/ipfs/js-ipfs @diasdavid pushed some major updates for js-ipfs, too.

Not much else to report this week; a lot of people are off to enjoy CCC, and the holidays.

December 7th

Here are some highlights of what happened during the December 7 Sprint :

Releases

@whyrusleeping shipped IPFS version 0.3.10! It contains 74 new commits since the previous version and you can get it here.
npm on IPFS! registry-mirror is a new tool that enables distributed discovery of npm modules by fetching and caching the latest state of npm through IPNS. For more info, see this blog post by @diasdavid .
@jbenet released a new tool/library called dnslink that makes it easy to resolve dns links (special TXT records in a domain name that can point to paths, like an IPFS path)

Updates

(infrastructure) On the infrastructure side of things, @lgierth has bootstrapped two new storage, each with 17 TB of disk space!
(api) @RichardLitt has [reached a draft 1]((//github.com/ipfs/api/pull/13) of the much needed API documentation.
@harlantwood wrote a bit of nodejs code that spins up a fresh IPFS node, sets it to a known ID, and publishes to IPNS using that node.
(specs) The new IPFS Linked Data (IPLD) spec is actively being iterated on in the specs repository. Join the discussion here!

Active stuff

@robcat and @fazo96 have done great work integrating IPFS with pacman (the package manager for Arch Linux). They can now install arch packages straight from IPFS! For more details, see this active discussion.
@Dignifiedquire has been working on an attractive new distribution page for IPFS, which will be the new landing page to download all things IPFS. You can see the latest screenshots here.

Contributors

Across the entire IPFS GitHub organization, the following people have committed code since December 7th. (We're autogenerating this list using this tool, so please let us know if your name isn't here.) In the future, we will also include people who comment, as they are also super important; bear with us while we develop that technology.

@alexmingoia (Alex Mingoia)
@chriscool (Christian Couder)
@diasdavid (David Dias)
@digital-dreamer
@Dignifiedquire (Friedel Ziegelmayer)
@djdv (Dominic Della Valle)
@fazo96 (Enrico Fasoli)
@greenkeeperio-bot
@harlantwood (Harlan T Wood)
@hcs64 (Adam Gashlin)
@ianopolous (Ian Preston)
@ivilata (Ivan Vilata-i-Balaguer)
@jbenet (Juan Benet)
@jedahan (Jonathan Dahan)
@Kubuxu (Jakub Sztandera)
@kyledrake (Kyle Drake)
@lgierth (Lars Gierth)
@NeoTeo (Teo Sartori)
@ralphtheninja (Lars-Magnus Skog)
@ReadmeCritic
@rht
@RichardLitt (Richard Littauer)
@travisperson (Travis Person)
@wasserfuhr (RainerWasserfuhr)
@whyrusleeping (Jeromy Johnson)
@zignig (Simon Kirkby)

Thanks, and see you next week!

Richard Littauer and Andrew Chin

Send us feedback about the Weekly

Stellar Module Management - Install your Node.js modules using IPFS

2015-12-08T00:00:00+01:00

Node.js Interactive, the first Node.js conference organized by the Linux Foundation, happened on Dec 8-9 of 2015. There were hundreds of participants, and dozens of really amazing talks divided in 3 specific tracks: backend, frontend and IoT.

I was fortunate to attend and present a project we've been developing at Protocol Labs, that builds on on top of IPFS, the InterPlanetary FileSystem.

You can learn about that project in this blog post, check out the talk slides or wait for the video recording of the talk. I will update this blog post when that happens.

Enter registry-mirror

registry-mirror enables distributed discovery of npm modules by fetching and caching the latest state of npm through IPNS, the InterPlanetary Naming System. With this state, a node in the network is capable of querying IPFS network for an npm module's cryptographic hash, fetching it from any peer that has it available.

registry-mirror is open source, MIT licensed and available at github.com/diasdavid/registry-mirror.

Getting started

In order to get started, you must first be sure that you are running IPFS 0.4.0. IPFS 0.4.0 is not yet released, but you can already use it by compiling from source or downloading the pre-built binary.

Compiling from source

You can find a tutorial on how to compile and install IPFS from source at https://github.com/ipfs/go-ipfs#build-from-source. Just make sure to change to the dev0.4.0 branch, as 0.4.0 isn't released yet.

Please make sure you have go 1.5.2 or above installed.

Downloading pre-built Binary

Download the pre-built binary for your OS and Arch at gobuilder.

Installing and running registry-mirror

Once you have IPFS 0.4.0 available, install registry-mirror by running the following command (you should have Node.js 4 and npm 2 or above available):

$ npm i registry-mirror -g
# ...

Then start your IPFS daemon, run:

$ ipfs daemon
Initializing daemon...
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/172.19.248.69/tcp/4001
Swarm listening on /ip6/::1/tcp/4001
API server listening on /ip4/127.0.0.1/tcp/5001
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready

After, run registry-mirror daemon with the --ipfs option:

$ registry-mirror daemon --ipfs --port=9595
IPFS mode ON
registry-mirror [info] output dir: /npm-registry/
registry-mirror [info] listening:127.0.0.1:9595
registry-mirror [info] Updated /npm-registry to: /ipfs/QmSjG9fadu4mPdtRsQYtXhwwCBouFEPiYHtVf8f4iH6vwj

Now, to install a module using IPFS, you only need to set this local registry when running an npm install. This can be done through config or a command line argument:

$ npm i bignumber --registry=http://localhost:9595
npm http request GET http://localhost:9595/bignumber
npm http 200 http://localhost:9595/bignumber
npm http fetch GET http://localhost:9595/bignumber/-/bignumber-1.1.0.tgz
npm http fetch 200 http://localhost:9595/bignumber/-/bignumber-1.1.0.tgz
/Users/david/Documents/code/ipfs/ip-npm/node-interactive
└── bignumber@1.1.0

Features

registry-mirror itself is quite a simple application, as most of the heavy lifting is done by IPFS. IPFS's distributed nature affords a set of really nice features as a transport layer that registry-mirror leverages to create its service.

Find where the module lives without having to hit the backbone

With registry-mirror, a registry becomes a curated list of hashes. While the modules live in the network, as soon as registry-mirror caches this list locally (which it gets from the IPFS network), it has a list of the hashes of the modules that a user might need in the future. With this list, a user doesn't have to know of the whereabouts of a module until it needs to request it from the network.

This list is fetched and kept up to date through IPNS. This ensures secure distribution, as IPNS records and validated with the publisher's priate key.

Work offline/disconnected

Just like git, registry-mirror is able to work offline and/or in a disconnected scenario. As long as the module you are looking for exists in the network you are currently in, IPFS would be able to find it through its Peer and Content Routing (e.g. with a DHT).

Enable several registries to coexist

Once the notion of a registry becomes a curated list of modules available, enabling more than one registry to exist becomes simpler. This scenario can be especially interesting for private networks such as the ones within companies and organizations that don't want their modules to be publicly known and available.

Run only what you were looking for

Just like git, IPFS verifies the content received using cryptographic hashing, making sure it is exactly what was requested -- you can always be sure that what you are running is what you asked for.

Faster

By leveraging local and network caches efficiently, downloading your dependencies can be much faster as it avoids going to npm's servers or CDN all the time. This can be crucial in high latency networks or more remote areas.

Demo Video

registry-mirror demo from David Dias on Vimeo.

A special thanks

A very big thank you goes to Bryan English and everyone that was involved in the discussion and contributed to make this possible.

ipscend - Publish static web content to IPFS

2015-11-27T00:00:00+01:00

ipscend is a new tool to help developers publish their static web content to IPFS and share it easily, while keeping history and more. It is heavily inspired by previous static web content publishing tools, like GitHub Pages and surge.

Features

Currently, ipscend offers a set of features, accessible through a CLI, and installable through npm (npm i -g ipscend), which enable a simple workflow for working in your Web page/app and publishing it to the IPFS network.

ipscend browse - Opens the last published version of your application in the browser.
ipscend init - Initializes your project. Asks for the folder where the web application will be available and stores an ipscend.json object in your current path to store all the metadata it generates, such as published versions and taken screenshots.
ipscend preview - Serves your application on a local static file server, so that you can try it out before you feel ready to publish it.
ipscend publish - Publishes the current state of your application to IPFS and stores a reference to it.
ipscend versions - Prints out the published versions for the app and its respective timestamp.
ipscend screenshot - Opens a screenshot preview of all the published versions of your app. In order to generate the screenshots, you must first run ipscend screenshot --gen.

An ipscend.json object for a project with some published versions will look like:

$ cat ipscend.json
{
  "versions": [
    {
      "hash": "QmQhNMwk7fThpwRNUR3bStb1eA7aeFJaApZjDaQCnykowL",
      "timestamp": "2015-11-14T14:50:10.998Z",
      "snapshot": "QmNMqiKZG7gCsnaQFTqG3AUeVhA1n8byy974Yqn3qRGZcJ"
    },
    {
      "hash": "QmSAmgQPCWjbrpbYHZQ2rVkH7a9vavubG1Jzv5CjDWrUmt",
      "timestamp": "2015-11-14T14:51:00.860Z",
      "snapshot": "QmcCNrn72FuHWkXtpJuUYfbH87d61qa6PSagUbLiK6VfLJ"
    },
    {
      "hash": "QmVNgdUoBQHiBhSeDe2z8LttJaDZq7JZi17sR1SPnJmjMh",
      "timestamp": "2015-11-14T14:51:24.379Z",
      "snapshot": "QmP5NuGdozeWaZqEdY1zBpcupB6qQ66AWReqm4L2vJzt73"
    }
  ],
  "path": "src/public"
}

Workflow

In order to get started, all you need to do is initiate your ipscend in your web page/app project:

$ ipscend init
This utility will walk you through creating a ipscend.json file.
Path of your Web Application (project)? (public) src/public
$ cat ipscend.json
{
  "versions": [],
  "path": "src/public"
}

Once you are ready to publish, run the ipscend publish command:

$ ipscend publish
{ hash: 'QmVNgdUoBQHiBhSeDe2z8LttJaDZq7JZi17sR1SPnJmjMh',
  timestamp: Fri Nov 27 2015 10:23:37 GMT+0000 (WET) }
published src/public QmVNgdUoBQHiBhSeDe2z8LttJaDZq7JZi17sR1SPnJmjMh
$ cat ipscend.json
{
  "versions": [
    {
      "hash": "QmVNgdUoBQHiBhSeDe2z8LttJaDZq7JZi17sR1SPnJmjMh",
      "timestamp": "2015-11-27T16:23:37.971Z"
    }
  ],
  "path": "src/public"

Grab that hash and share it with your friends, by sending them a link to ipfs.io, appending "/ipfs/Hash" (e.g. https:/ipfs.io/ipfs/ QmVNgdUoBQHiBhSeDe2z8LttJaDZq7JZi17sR1SPnJmjMh).

If you want to use your awesome.domain.com to load your page from IPFS, you can check how to do it now at https://github.com/diasdavid/ipscend#use-ipfs-to-host-your-webpage-using-a-standard-domain-includes-cool-dns-trick.

Awesome (FUTURE)!

ipscend is still in its humble beginnings. Some of the ideas and plans to build in the future include being able to:

Extract the version from the VCS itself (https://github.com/ipfs/notes/issues/23), so that every commit can be a different working version, allowing you to test every commit in your CI.
Roll back history, following the 'time machine' analogy.
Update your DNS provider automatically, avoiding having to use an external tool like dnslink-deploy.
Enable reviewers to write notes.
Take screenshots in every browser version, so that we can use the timeline to see if there was any regression at a point in time, which might happen in a specific browser.
moaaaar :D If you have ideas or want to contribute, ipscend is fully MIT Licensed, so feel free to open a issue or PR on https://github.com/diasdavid/ipscend.

A big thank you to Andrés Gutgon who made the screenshot preview look really good

Run IPFS in a Docker container

2015-07-11T00:00:00+02:00

In recent years, Docker and a few other projects have redefined how we run server applications. In the future, we might be running containerized apps in our personal devices. At its core, this fast-paced improvement is a combination of good interfaces to standardize how to do things, and great tooling to make using containers easy.

The IPFS Project has many things planned for the world of containers. The most interesting is using IPFS to distribute containers hyper efficiently across data-centers and the internet. We will be discussing many of these things in upcoming posts, but first things first. This post is a quick guide for running an IPFS node directly within Docker.

The IPFS team has provided an IPFS Docker image, which is syncronized with the latest commits to go-ipfs. It only takes a few commands to try it out!

> mkdir /tmp/ipfs-docker-staging
> mkdir /tmp/ipfs-docker-data
> docker run -d --name ipfs-node \
  -v /tmp/ipfs-docker-staging:/export -v /tmp/ipfs-docker-data:/data/ipfs \
  -p 8080:8080 -p 4001:4001 -p 127.0.0.1:5001:5001 \
  jbenet/go-ipfs:latest
faa8f714398c7a1a5a29adc2aed01857b41444ed53ec11863a3136ad37c8064c

Port 8080 is the HTTP Gateway, which allows you to query ipfs data with your browser (see this example), port 4001 is what swarm port IPFS uses to communicate with other nodes, and port 5001 is used for the local API. We bind 5001 only on 127.0.0.1 because it should not be exposed to the outside world. The faa8f7143... is the docker container id.

We've mounted a data and staging volume. The data volume is used to store the IPFS local repo (config and database), and staging is a directory you can use for staging files for command line usage (such as ipfs add). If you're only using the API, you can omit the staging directory volume. And of course, feel free to put those directories somewhere other than /tmp.

Now what? Your node is running. You can issue commands directly to the containerized ipfs with docker exec <container-id> <ipfs-cmd>. For example, you can try ipfs swarm peers to see who you are connected to:

# let's set $cid = <container-id> for easy access
> cid=faa8f714398c7a1a5a29adc2aed01857b41444ed53ec11863a3136ad37c8064c
> docker exec $cid ipfs swarm peers
/ip4/104.236.179.241/tcp/4001/ipfs/QmSoLpPVmHKQ4XTPdz8tjDFgdeRFkpV8JgYq8JVJ69RrZm
/ip4/128.199.219.111/tcp/4001/ipfs/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu
/ip4/162.243.248.213/tcp/4001/ipfs/QmSoLueR4xBeUbY9WZ9xGUUxunbKWcrNFTDAadQJmocnWm
/ip4/178.62.61.185/tcp/4001/ipfs/QmSoLMeWqB7YGVLJN3pNLQpmmEk35v6wYtsMGLzSr5QBU3

And of course, you can add or cat content as usual:

> echo "hello from dockerized ipfs" >/tmp/ipfs-docker-staging/hello
> docker exec $cid ipfs add /export/hello
added QmcDge1SrsTBU8b9PBGTGYguNRnm84Kvg8axfGURxqZpR1 /export/hello
> docker exec $cid ipfs cat /ipfs/QmSvCqazpuuib8qyRyddyFemLc2qmRukLLy8YfkdRPEXoQ
hello there!

Your dockerized IPFS is now also running a Gateway at http://<ip-address-of-the-computer>:8080. You can try it out with curl, or with your browser:

> curl http://localhost:8080/ipfs/QmcDge1SrsTBU8b9PBGTGYguNRnm84Kvg8axfGURxqZpR1
hello from dockerized ipfs

Kubernetes 1.0 comes out next week, so after that, we'll try using it to build a cluster of IPFS nodes that can store any kind of data and be able to retreive it from any other IPFS node. Not just with IPFS nodes in your cluster, but with everyone!

asciicast powered by asciinema

Hello Worlds

2015-05-05T00:00:00+02:00

> echo "hello worlds" | ipfs add
added QmZ4tDuvesekSs4qM5ZBKpXiZGun7S2CYtEZRB3DYXkjGx
> ipfs cat QmZ4tDuvesekSs4qM5ZBKpXiZGun7S2CYtEZRB3DYXkjGx
hello worlds

Greetings Internet!

This post kicks off the official IPFS (InterPlanetary File System) Blog. This is long overdue-- the project is many months old. We will be making a series of posts explaining various aspects of IPFS, its development, our growing community, and so on. This blog will also be used to make all important announcements henceforth.

You can't yet "follow" the blog with ipfs, but we're working on it and expect to have it working soon. For now, you can follow one of these ways:

HTTP: http://ipfs.io/blog
Git: git clone https://github.com/ipfs/blog
GitHub: click watch at https://github.com/ipfs/blog
RSS: follow RSS Feed
IPFS: coming soon

Don't miss any InterPlanetary updates!

IPFS

IPFS distributions

The IPFS Distributions Website

Download Deduplication

Project details

Project Prerequisites

Future Plans

Contribute

IPFS Weekly 10

Updates

0.4.0!

Orbit

go-libp2p and go-ipfs refactoring

js-ipld

ipfs-blob-store v1.0.0 released

js-ipfs-api now supports the Files API

Making js-ipfs and js-libp2p modules awesome

go-ipfs and the Prometheus monitoring system

go-ipfs and restricting dialing

Community

Berkman Center

MIT Meetup

Peergos

SpaceX Landing

Content-Type of The Week

Contributors

IPFS Weekly 9

Updates

Package Managers

DNS outage

Captain.log

Aye, you might want to check the new js-ipfs Captain.log entry, matey!

js-ipfs-init

Dictionary support for the zlib JavaScript Implementation, pako

jsipfs object cli and http-api endpoints are complete

Nginx metrics

Community

Upcoming talks

First IPFS Meeting in NYC

Meeting with NYC Mesh

Lisbon Research and Development Meetup

Seattle

BitCoin News

BitDevsNYC

IPFS Meme of the Week

Content-Type of the Week

Contributors

go-ipfs 0.4.0 has been released

The IPFS Files API

Why we're changing the protocol

Breaking changes

The public gateway and bootstrappers

Changes to the repo

Other improvements and fixes

How to upgrade

Changelog

IPFS Weekly 8

Updates

PubSub

ipfs init

Community

Lisbon

Press

Contributors

IPFS Weekly 7

Updates

Community

Contributors

IPFS Weekly 6

Updates

AUR

ipfs-pad

pubsub

Community

dignifed hacks

Interplanetary Wayback (IPWB)

Science Data

Press

Contributors

IPFS Weekly 5

`ipfs-blob-store` v1.0.0 released

`js-ipfs-api` now supports the Files API

Dictionary support for the zlib JavaScript Implementation, `pako`

`jsipfs object` cli and http-api endpoints are complete