https://petermolnar.net/feed - petermolnar.net 2018-04-17T18:00:00+00:00 Peter Molnar mail@petermolnar.net python-feedgen https://petermolnar.net/favicon.png https://petermolnar.net/how-to-make-a-print-css Guide on how to make your website printable with CSS 2018-04-17T18:00:00+00:00 Printing?! It's 2018!

"Printing" doesn't always mean putting it on paper. When people print a web article, sometimes it ends up as a PDF, because the HTML save it not usable. The reasons for this differ: JavaScript rendered content, unsaved scripts in the end result, the lack of MHTML support in browsers, etc. What's important is that providing a print-friendly format for your site makes it possible for people to save it in a usable way.

Printing might still be relevant, because that's the only method that gives you a physical object. I have long entries about journeys, visits of foreign places. At a certain point in time I was tempted to put together a photobook from the images there, but the truth is: it's a lot of work, especially if you've more or less done it already once by writing your entry.

There's also the completely valid case of archiving: hard copies have a life of decades if not centuries, when stored properly, unlike any electronic media we currently have as an option.

That little extra CSS

Before jumping into the various hacks that helps printers it's important to mention, how to add printer-only instructions to your CSS. There are two ways, either using:

@media print {

}

inside an existing CSS, or by adding another CSS file specifically for print media into the HTML <head> section:

<head>
    <link rel="stylesheet" type="text/css" href="print.css" media="print">
</head>

White background, black(ish) text

Most printers operate with plain, white paper, so unless there's a very, very good reason for printing background color, just get rid of it.

It also applies to the font: a bit lighter from black, so saves tint.

* {
    background-color: #fff !important;
    color: #222;
}

Use printer and PDF safe fonts

If you take a look at the history of printers vs fonts there used to be many problems around this topic - even so they might still require a font cartridge to be able to properly print fonts out of the basic options.1

To avoid rendering problems, aliasing issues, generally speaking: unreadable fonts, stick to one of the base 14 fonts:

  • Courier, Courier-Bold, Courier-Oblique, Courier-BoldOblique
  • Helvetica, Helvetica-Bold, Helvetica-Oblique, Helvetica-BoldOblique
  • Times-Roman, Times-Bold, Times-Italic, Times-BoldItalic
  • Symbol
  • ZapfDingbats

which are, by definition, part of the PDF standard2. So for example:

* {
    font-size: 11pt !important;
    font-family: Helvetica, sans-serif !important;
}

If you do insist on special fonts, eg. you have icons in fonts, you might want to consider using SVG instead of fonts for icons - otherwise printing them properly will become a problem.

Besides the potential printing issues one more reason to go with a standard, base font is that if for any reason the text needs to go through character recognition for scanning it back - say it's an archival hard copy and the only one left after a data loss indicent - the simpler and wider known the font, the better your chances for getting the characters properly recognized.

Pages and page breaks

It's very annoying to find a heading at the bottom of a printed page, or a paragraph broke into separate pages, although this latter depends on paragraph length. I generally recommend disallowing page breaks at these locations.

Apart from this it's a good idea to have a margin around the edges so you have an area where you can handle the page, not covering any of the text, or where it can be glued together as pages in a book.

@page {
    margin: 0.5in;
}

h1, h2, h3, h4, h5, h6 {
    page-break-after: avoid !important;
}

p, li, blockquote, figure, img {
    page-break-inside: avoid !important;
}

Images

Printing images is tricky: most of the images are sized for the web and those sizing are too small by resolution, too large by percentage of space taken for printing. The alt-text and the image headline, which is usually in alt and title are also something to consider printing, but unfortunately the href trick doesn't work with them: that is because you can't add ::before or ::after to self-closing tags, such as images.

Lately, instead of using simple img tags, I switched to using figure, along with figcaption - this way the headline became possible to print.

Apart from this I've limited the size of the images by view-width and view-height, so they never become too large and occupy complete pages.

figure {
    margin: 1rem 0;
}

figcaption {
    text-align: left;
    margin: 0 auto;
    padding: 0.6rem;
    font-size: 0.9rem;
}

figure img {
    display: block;
    max-height: 35vh;
    max-width: 90vw;
    outline: none;
    width: auto;
    height: auto;
    margin: 0 auto;
    padding: 0;
}

This is how images inside figure (should) look in print with the styling above:

This is how images can look like when some width/height limitations are applied in printing
This is how images can look like when some width/height limitations are applied in printing

Source codes

If you have code blocks in your page it's useful to have them coloured, but still dark-on-light.

I'm using Pandoc's built-in syntax highlighting3 and the following styling for printing:

code, pre {
    max-width: 96%;
    border: none;
    color: #222;
    word-break: break-all;
    word-wrap: break-word;
    white-space: pre-wrap;
    overflow:initial;
    page-break-inside: enabled;
    font-family: "Courier", "Courier New", monospace !important;
}

pre {
    border: 1pt dotted #666;
    padding: 0.6em;
}

/* code within pre - this is to avoid double borders */
pre code {
    border: none;
}

code.sourceCode span    { color: black; }
code.sourceCode span.al { color: black; }
code.sourceCode span.at { color: black; }
code.sourceCode span.bn { color: black; }
code.sourceCode span.bu { color: black; }
code.sourceCode span.cf { color: black; }
code.sourceCode span.ch { color: black; }
code.sourceCode span.co { color: darkgray; }
code.sourceCode span.dt { color: black; }
code.sourceCode span.dv { color: black; }
code.sourceCode span.er { color: black; }
code.sourceCode span.ex { color: darkorange; }
code.sourceCode span.fl { color: black; }
code.sourceCode span.fu { color: darkorange; }
code.sourceCode span.im { color: black; }
code.sourceCode span.kw { color: darkcyan; }
code.sourceCode span.op { color: black; }
code.sourceCode span.ot { color: black; }
code.sourceCode span.pp { color: black; }
code.sourceCode span.sc { color: black; }
code.sourceCode span.ss { color: black; }
code.sourceCode span.st { color: magenta; }
code.sourceCode span.va { color: darkturquoise; }

It should result in something similar:

Color printing source code
Color printing source code

The basic CSS solution

Links are the single most important things on the internet; they are the internet. However, when they get printed, the end result usually looks something like this:

Before showing URLs - example showing Wikipedia entry
Before showing URLs - example showing Wikipedia entry "Mozilla software rebranded by Debian"

In order to avoid this problem, the URLs behind the links need to be shown as if they were part of the text. There is a rather simple way to do it:

a::after {
    content: " (" attr(href) ") ";
    font-size: 90%;
}

but unfortunately it makes the text rather ugly and very hard to read:

After showing URLs
After showing URLs

Aaron Gustafson's solution4

There is a very nice, minimalistic Javascript solution5 that collects all links on the page and converts them into footnotes on the fly, when it detects a print request.

This solution is way nicer, so I certainly recommend using this as well (it's a supplement for the CSS solution above) even if it requres Javascript: (this is a copy-paste solution, just put it in your header)

<script type="text/javascript">
    // <![CDATA[
    /*------------------------------------------------------------------------------
    Function:       footnoteLinks()
    Author:         Aaron Gustafson (aaron at easy-designs dot net)
    Creation Date:  8 May 2005
    Version:        1.3
    Homepage:       http://www.easy-designs.net/code/footnoteLinks/
    License:        Creative Commons Attribution-ShareAlike 2.0 License
                    http://creativecommons.org/licenses/by-sa/2.0/
    Note:           This version has reduced functionality as it is a demo of
                    the script's development
    ------------------------------------------------------------------------------*/
    function footnoteLinks(containerID,targetID) {
      if (!document.getElementById ||
          !document.getElementsByTagName ||
          !document.createElement) return false;
      if (!document.getElementById(containerID) ||
          !document.getElementById(targetID)) return false;
      var container = document.getElementById(containerID);
      var target    = document.getElementById(targetID);
      var h2        = document.createElement('h2');
      addClass.apply(h2,['printOnly']);
      var h2_txt    = document.createTextNode('Links');
      h2.appendChild(h2_txt);
      var coll = container.getElementsByTagName('*');
      var ol   = document.createElement('ol');
      addClass.apply(ol,['printOnly']);
      var myArr = [];
      var thisLink;
      var num = 1;
      for (var i=0; i<coll.length; i++) {
        var thisClass = coll[i].className;
        if ( coll[i].getAttribute('href') ||
             coll[i].getAttribute('cite') ) {
          thisLink = coll[i].getAttribute('href') ? coll[i].href : coll[i].cite;
          var note = document.createElement('sup');
          addClass.apply(note,['printOnly']);
          var note_txt;
          var j = inArray.apply(myArr,[thisLink]);
          if ( j || j===0 ) {
            note_txt = document.createTextNode(j+1);
          } else {
            var li     = document.createElement('li');
            var li_txt = document.createTextNode(thisLink);
            li.appendChild(li_txt);
            ol.appendChild(li);
            myArr.push(thisLink);
            note_txt = document.createTextNode(num);
            num++;
          }
          note.appendChild(note_txt);
          if (coll[i].tagName.toLowerCase() == 'blockquote') {
            var lastChild = lastChildContainingText.apply(coll[i]);
            lastChild.appendChild(note);
          } else {
            coll[i].parentNode.insertBefore(note, coll[i].nextSibling);
          }
        }
      }
      target.appendChild(h2);
      target.appendChild(ol);
      addClass.apply(document.getElementsByTagName('html')[0],['noted']);
      return true;
    }
    window.onload = function() {
      footnoteLinks('content','content');
    }
    // ]]>
  </script>
  <script type="text/javascript">
    // <![CDATA[
    /*------------------------------------------------------------------------------
    Excerpts from the jsUtilities Library
    Version:        2.1
    Homepage:       http://www.easy-designs.net/code/jsUtilities/
    License:        Creative Commons Attribution-ShareAlike 2.0 License
                    http://creativecommons.org/licenses/by-sa/2.0/
    Note:           If you change or improve on this script, please let us know.
    ------------------------------------------------------------------------------*/
    if(Array.prototype.push == null) {
      Array.prototype.push = function(item) {
        this[this.length] = item;
        return this.length;
      };
    };
    // ---------------------------------------------------------------------
    //                  function.apply (if unsupported)
    //           Courtesy of Aaron Boodman - http://youngpup.net
    // ---------------------------------------------------------------------
    if (!Function.prototype.apply) {
      Function.prototype.apply = function(oScope, args) {
        var sarg = [];
        var rtrn, call;
        if (!oScope) oScope = window;
        if (!args) args = [];
        for (var i = 0; i < args.length; i++) {
          sarg[i] = "args["+i+"]";
        };
        call = "oScope.__applyTemp__(" + sarg.join(",") + ");";
        oScope.__applyTemp__ = this;
        rtrn = eval(call);
        oScope.__applyTemp__ = null;
        return rtrn;
      };
    };
    function inArray(needle) {
      for (var i=0; i < this.length; i++) {
        if (this[i] === needle) {
          return i;
        }
      }
      return false;
    }
    function addClass(theClass) {
      if (this.className != '') {
        this.className += ' ' + theClass;
      } else {
        this.className = theClass;
      }
    }
    function lastChildContainingText() {
      var testChild = this.lastChild;
      var contentCntnr = ['p','li','dd'];
      while (testChild.nodeType != 1) {
        testChild = testChild.previousSibling;
      }
      var tag = testChild.tagName.toLowerCase();
      var tagInArr = inArray.apply(contentCntnr, [tag]);
      if (!tagInArr && tagInArr!==0) {
        testChild = lastChildContainingText.apply(testChild);
      }
      return testChild;
    }
    // ]]>
  </script>
  <style type="text/css" media="screen">
    .printOnly {
      display: none;
    }
  </style>
  <style type="text/css" media="print">
    a:link:after,
    a:visited:after {
      content: " (" attr(href) ") ";
      font-size: 90%;
    }
    html.noted a:link:after,
    html.noted a:visited:after {
      content: '';
    }
  </style>

Alternative approach: always using footnotes for URLs

I little while ago I made a decision to put all links into footnotes by default - no in-text-links which will bring you to another site. This is a design decision and doesn't apply to most of the already existing sites, but if you, just as me, think, there is value in it, consider it as an option. It also makes the two hacks above obsolete, however, it has it's own problems, such as reading the site entries via RSS.

Avoid

opacity and transparency: it can get blurry

A simple and sort of lazy solution to, instead of figuring out the proper color code, just apply opacity to a text to make it slightly different from the rest. Unfortunately some of these opacity settings can result in blurry or unusable text:

CSS opacity resulting in blurry text
CSS opacity resulting in blurry text

Therefore I suggest to avoid opacity and transparency on all elements for your printing styles.

Happy printing!

  1. https://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/print_c_fonts.mspx↩

  2. https://en.wikipedia.org/wiki/Portable_Document_Format#Standard_Type_1_Fonts_.28Standard_14_Fonts.29↩

  3. http://pandoc.org/MANUAL.html#syntax-highlighting↩

  4. https://alistapart.com/article/improvingprint↩

  5. https://alistapart.com/d/improvingprint/files/final.html↩

]]> 2018-04-17T18:00:00+00:00 2018 Peter Molnar CC BY 4.0 https://petermolnar.net/internet-emotional-core The internet that took over the Internet 2018-03-25T21:20:00+00:00 There is a video out there, titled "The Fall of The Simpsons: How it Happened"1. It starts by introducing a mediocre show that airs every night, called "The Simpsons", and compares it to a genius cartoon, that used to air in the early 90s, called "The Simpsons". Watch the video, because it's good, and I'm about to use it's conclusion.

It reckons that the tremendous difference is due to shrinking layers in jokes, and, more importantly, in the characters after season 7. I believe something similar happened online, which made the Internet become the internet.

Many moons ago, while still living in London, the pedal of our flatmate's sewing machine broke down, and I started digging for replacement parts for her. I stumbled upon a detailed website about ancient capacitors2. It resembled other, gorgeous sources of knowledge: one of my all time favourite is leofoo's site on historical Nikon equipment3. All decades old sites, containing specialist level knowledge on topics only used to be found in books in dusty corners of forgotten libraries.

There's an interesting article about how chronological ordering destroyed the original way of curating content4 during the early online era, and I think the article got many things right. Try to imagine a slow web: slow connection, slow updates, slow everything. Take away social networks - no Twitter, no Facebook. Forget news aggregators: no more Hacker News or Reddit, not even Technorati. Grab your laptop and put in down on a desk, preferably in a corner - you're not allowed to move it. Use the HTML version of DuckDuckGo5 to search, and navigate with links from one site to another. That's how it was like; surfing on the information highway, and if you really want to experience it, UbuWeb6 will allow you to do so.

Most of the content was hand crafted, arranged to be readable, not searchable; it was human first, not machine first. Nearly everything online had a lot of effort put into it, even if the result was eye-blowing red text on blue background7; somebody worked a lot on it. If you wanted it out there you learnt HTML, how to use FTP, how to link, how to format your page.

We used to have homepages. Homes on the Internet. Not profiles, no; profile is something the authorities make about you in dossier.

6 years ago Anil Dash released a video, "The web we lost"8 and lamented the web 2.0 - I despise this phrase; a horrible buzzword everyone used to label anything with; if you put 'cloud' and 'blockchain' together, you'll get the level of buzz that was 'web 2.0' -, that fall short to social media, but make no mistake: the Internet, the carefully laboured web 1.0, had already went underground when tools made it simple for anyone to publish with just a few clicks.

The social web lost against social media, because it didn't (couldn't?) keep up with making things even simpler. Always on, always instant, always present. It served the purpose of a disposable web perfectly, where the most common goal is to seek fame, attention, to follow trends, to gain followers.

There are people who never gave up, and are still tirelessly building tools, protocols, ideas, to lead people out of social media. The IndieWeb9's goals are simple: own your data, have an online home, and connect with others through this. And so it's completely reasonable to hear:

I want blogging to be as easy as tweeting.10

But... what will this really achieve? This may sound rude and elitist, but the more I think about it the more I believe: the true way out of the swamp of social media is for things to require a little effort.

To make people think about what they produce, to make them connect to their online content. It's like IKEA11: once you put time, and a minor amount of sweat - or swearing - into it, it'll feel more yours, than something comfortably delivered.

The Internet is still present, but it's shrinking. Content people really care about, customised looking homepages, carefully curated photo galleries are all diminishing. It would be fantastic to return to a world of personal websites, but that needs the love and work that used to be put into them, just like 20 years ago.

At this point in time, most people don't seem to relate to their online content. It's expendable. We need to make them care about it, and simpler tooling, on it's own, will not help with the lack of emotional connection.

  1. https://www.youtube.com/watch?v=KqFNbCcyFkk↩

  2. http://www.vintage-radio.com/repair-restore-information/valve_capacitors.html↩

  3. http://www.mir.com.my/rb/photography/↩

  4. https://stackingthebricks.com/how-blogs-broke-the-web/↩

  5. https://duckduckgo.com/html/↩

  6. http://www.slate.com/articles/technology/future_tense/2016/12/ubuweb_the_20_year_old_website_that_collects_the_forgotten_and_the_unfamiliar.html↩

  7. http://code.divshot.com/geo-bootstrap/↩

  8. http://anildash.com/2012/12/the-web-we-lost.html↩

  9. https://indieweb.org↩

  10. http://www.manton.org/2018/03/indieweb-generation-4-and-hosted-domains.html↩

  11. https://en.wikipedia.org/wiki/IKEA_effect↩

]]> 2018-03-25T21:20:00+00:00 2018 Peter Molnar CC BY 4.0 https://petermolnar.net/re-eli-20180318015703 RE: https://eli.li/entry.php?id=20180318015703 2018-03-18T15:30:00+00:00

Anyone with me? Am I totes off base?

You are certainly not; the tools provided right now are indeed technical. However, you have to keep it in mind, that none of the specifications are finalised in any form - only webmentions are in W3C recommendation stage. As it has been pointed out, my wording and knowledge was off: recommendation is the final, released stage, so I fixed it.

The vast majority of users aren’t going to read the spec., nor care to ever do so. [...] This should be our (the IndieWeb’s) holy mission — empowering all sorts of folks to post content that they get to control.

This is where we disagree. There are other, interesting movements, two of them with very similar goals: Repair Café and Restart Party. As the name suggests, it's about repairing things, instead of throwing them away, but both of them share the ideology of teaching people to fix their own things; to learn about their tools, to value, to own them.

Owning the content is the first step, but it shouldn't be the only step. I wholeheartedly disagree with WordPress' attitude, making people avoid technical responsibility1 - people should care, they should be at least be aware of what's happening when they press a publish button. They needn't have to be capable of doing it from scratch, but providing the tools only is not a goal I can align with.

If we keep up the attitude of making everything so simple that people don't have to understand, we'll very soon find ourselves in the chapter of The Foundation, where the priests keep running technology, without anyone understanding it.

  1. https://www.rarst.net/wordpress/technical-responsibility/↩

]]> 2018-03-18T15:30:00+00:00 2018 Peter Molnar CC BY-NC-ND 4.0 https://petermolnar.net/instant-messenger-hell We are living in instant messenger hell 2018-03-03T15:00:00+00:00 Note: I have updated some parts of this entry. This is due to the fact that I wrote about XMP without spending enough time exploring what it's really capable of, for which I'm sorry. I made changes to my article according to these finds.

Me vs. IM

Before the dawn of the always online era (pre 2007) the world of instant messengers was completely different. For me, it all started with various IRC1 rooms, using mIRC2, later extended with ICQ3 in 1998.

I loved ICQ. I loved it's notifications sound - I have it as notification sound on my smartphone and it usually results in very confused expressions from people who haven't heard the little 'ah-oooh' for a decade -, it's capability of sending and receiving files, the way you could search for people based on location, interest tags, etc.

The sixth protocol version appeared in ICQ 2000b and faced a complete rework. Encryption was significantly improved. Thanks to the new protocol, ICQ learned how to call phones, and send SMS and pager messages. Users also got the option of sending contact requests to other users.4

Around this time, Windows included an instant messenger in their operating systems: MSN Messenger5, later renamed to Windows Live Messenger. It was inferior, but because it was built in to Windows, it took all the ICQ users away. It's completely dead now.

The multiplication of messengers had one useful effect though: people who got fed up running multiple clients for the same purpose - to message people - came up with the idea if multi-protocol applications. I used Trillian6 for many years, followed by Pidgin7 once I switched to linux.

With the help of these multi-protocol miracles it wasn't an issues when newcomers like Facebook or Google released their messaging functionality: both were built in top of XMPP8, an open standard for instant messaging, and they were both supported out of the box in those programs.

Around this time came Skype and it solved all the video call problems with ease. It was fast, p2p, encrypted, ran on every platform, supported more or less everything people needed, including multiple instances for multiple accounts. Skype was on a good way to eliminate everything else. Unfortunately none of the multi-protocol messengers ever had a native support to it: it only worked if a local instance of Skype was running.

A few years later iPhone appeared and it ate the consumer world; not long before that, BlackBerry did the same to the business. Smartphones came with their own, new issues: synchronization, and resource (battery and bandwidth) limitations. ICQ existed for Symbian S60, Windows CE, and a bunch of other, ancient platforms, but by the time iPhones and BlackBerries roamed the mobile land, it was in a neglected state in AOL and missed a marvellous opportunity.

Both of those problems were known and addressed in the XMPP specification. The protocol was low on resources by design, it supported device priority, and XEP-0280: Message Carbons9 took care of delivering messages to multiple clients. There was a catch though: none of the well known XMPP providers supported any of these additions, so you ended up using either your mobile device or your computer exclusively at the same time. Most of the big system - AOL, Yahoo!, MSN, Skype, etc - didn't even have a client for iOS, let alone for Android that time.

This lead to a new type of messenger generation: mobile only apps. WhatsApp10, BlackBerry Messenger11, Viber, etc - none of them offered any real way to be used from the desktop, and they all required - they still do - a working mobile phone number even to register.

For reasons I'm yet to comprehend, both Google and Facebook abandoned XMPP instead of extending fully implementing it. Google went completely proprietary and replaced gtalk12 with Hangouts13; Facebook started using MQTT14 for their messenger applications. Both of them were simple enough to be reverse engineered and added to libpurple, but they both tried to reinvent something that already existed.

For Skype, this was a turning point: it was bought by Microsoft, and they slowly moved it from p2p to a completely centralised webapp. The official reasoning included something about power hungry p2p connections... Soon, Skype lost all of it's appeal from it's previous iterations: video and voice was lagging, it was consuming silly amount of resources, it was impossible to stop it on Android, etc. Today, it resembles nothing from the original, incredible, p2p, secure, decentralised, resource-aware application it used to be.

I had to install WhatsApp yesterday - I resisted it as long as I could. It completely mangled competition in the UK and the Netherlands: nobody is willing to use anything else, not even regular text (SMS) or email. It did all this despite it's lack of multi-device support, and the fact that it's now owned by one of the nastiest, people-ignorant businesses around the globe15.

So, all together, in February 2018, for work and personal communication, I need to be able to connect to:

  • IRC
  • Skype
  • Facebook
  • Telegram
  • XMPP
  • Workplace by Facebook16
  • WhatsApp
  • ICQ*
  • Google Hangouts*
  • WeChat**

* I still have some contacts on ICQ, though it's a wasteland, and I can't even remember the last time I actually talked to anyone on it. This sort of applies to Hangouts: those who used to use it are now mostly on Facebook.

** WeChat is, so far, only a thing if you have Chinese contacts or if you live in/visit China. It's dominating China so far that other networks, like QQ, can be more or less ignored, but WeChat is essential.

If I install all those things on my phone, I'll run out of power in a matter of hours and the Nomu has an internal 5000mAh brick. They will consume any RAM I throw at them, and I don't even want to think about the privacy implications: out of curiosity I checked the ICQ app, but the policy pushed into my face on the first launch is rather scary. As for Facebook: I refuse to run Facebook in any form on my phone apart from 'mbasic', the no javascript web interface.

Typing on a touchscreen inefficient, and I'm very far from being a keyboard nerd; my logs will be application specific and probably not in any readable/parsable format.

On top of all this, a few days ago Google announced Google Hangouts Chat17. Right now, Google has the following applications to cover text, voice, and video chat:

  • Hangouts
  • Allo
  • Duo
  • Hangouts Meet
  • Hangouts Chat

That's 5 applications. 5. Only from Google.

Words for the future

I really, really want one, single thing, which allows me:

  • native voice and video
  • private and group messaging
  • multiple concurrent login from varios platforms
  • libpurple plugin option
  • all devices get all messages

I sort of liked is Telegram18: cross device support, surprisingly fast and low on resources, but it gets attacked because they dared to roll their own crypto, and, in the end, it's still a centralised service, ending up as just another account to connect to, and just another app to run. Since I wrote this entry, a few has tried to point out, that Telegram is not better, than WhatsApp or Signal, but I have to disagree. Yes, WhatsApp is encrypted by default - this also means I need to run my phone as a gateway all the time. No phone = no desktop user. The desktop "app" is a power and resource eater Electron app.

Others asked about Signal. It's doing encryption the paranoid, aggressive way, but the same time, it depends on Google Services Framework on Android, Twilio, AWS, requires a smartphone app, eliminates 3rd party client options, and will only run the "desktop" Electron app if you pair it with a phone app - in which case it's very similar to WhatsApp. Like it or not, it's also a silo, with centralised services, even though you could, in theory, be able to install a complicated server of your own, that relies on the services listed above. It might be better, then WhatsApp, definitely not better from a usability point of view, than Telegram. Privacy wise... unless I can run my own server, without those kind of dependencies, no, thanks - it's just another silo.

I also believe OTR-like encryption is overrated, or at least not as important as many presses. Most of the messages will tell you less, than their metadata, so what's the point? Most of the encryption protocols are exclusive per connected client, meaning you can't have multiple devices with the same account exchanging the same messages - hence the need for the phone apps as gateways. XMPP with OMEMO19 is tacking this - if that's on by default, that could work. Note: TLS, infrastructure level encryption is a must, that is without question.

While Matrix20 looks promising, it's an everything-on-HTTP solution, which I still find odd and sad. HTTP is not a simple protocol - yes, it's omnipresent, but that doesn't make it the best for a particular purpose. There's another problem with it: no big player bought in which could bring the critical mass of users, and without that, it's practically impossible to get people to use it.

Video and voices calls are, in general, in a horrible shape: nearly everything is doing WebRTC, which, while usually works, is a terrible performer, insanely heavy on CPU, and, most of the time, always tries to go for the highest quality, consuming bandwidth like there is no tomorrow.

All this leaves me with XMPP and SIP.

XMPP is and could be able to cover everything, and, on top of it, it's federated, like email: anyone can run their own instance. I'm still a fan of email (yes, you read that right), and a signficant part of it is due to the options you can choose from: providers, clients, even being your own email service.

Unlike with most solutions and silos, the encryption problem (namely that if encryption is on, only one of the devices can get the messages, our you need to use a router device, like WhatsApp does) is covered and done with the XMPP extension OMEMO21. It's a multi-client encryption protocoll, that allows simultaneous devices to connect and encrypt at once.

In case of XMPP, voice and video could be handled by a P2P protocol, Jingle22, but, unfortunately, it's rarely supported. On Android, I found Astrachat23 which can do it, but it lacks many features when it comes to text based communications, unlike Conversations24. On desktop, I'm having serious problems getting Pidgin use video, so not everything is working yet.

This is where SIP comes in: an old, battle tested, proven VOIP protocol, which, so far, worked for me without any glitch in 2018. A few years ago many mobile providers were blocking SIP (among other VOIP protocols), but it's getting much better. Unfortunately I have not started running my own VOIP exchange yet, and ended up using Linphone25 as software and provider - for now. The unfortunate part of SIP is that Pidgin doesn't support it in any form.

There is one, very significant problem left: conformist people. I understand WhatsApp is simple and convenient, but it's a Facebook owned, phone only system.

I'd welcome thoughts and recommendations on how to make your friends use something that's not owned by Facebook.

Until then, I'll keep using Pidgin, with a swarm of plugins that need constant updating.

Adding networks to Pidgin (technical details)

Pidgin, which I mentioned before, is a multi protocol client. Out of the box, it's in a pretty bad shape: AIM, MSN, and Google Talk are dead as doornail, most of the systems it supports are arcane (eg. Zephyr) or sort of forgotten (ICQ). The version 3 of pidgin, and it's library, libpurple, has been in the making for a decade and it's still far ahead; the current 2.x line is barely supported.

There is hope however: people keep adding support for new systems, even to ones without proper or documented API.

For those who want to stick to strictly text interfaces, Bitlbee has a way to be compiled with libpurple support, but it's a bit weird to use when you have the same contact or same names present on multiple networks.

The guides below are made for Debian and it's derivatives, like Ubuntu and Mint. In order to build any of the plugins below, some common build tools are needed, apart from the per plugin specific ones:

sudo apt install libprotobuf-dev protobuf-compiler build-essential
sudo apt-get build-dep pidgin

How to conect to Skype with Pidgin (or libpurple)

The current iteration of the Skype plugin uses the web interface to connect to the system. It doesn't offer voice and video calls, but it supports individual and group chats alike.

If you have 2FA on, you'll need to use your app password as password and tick the Use alternative login method on the Advanced tab when adding the account.

git clone https://github.com/EionRobb/Skype4pidgin
cd Skype4pidgin/Skypeweb
cmake .
make
sudo make install

How to connect to Google Hangouts with Pidgin (or libpurple)

I've taken the instructions from the author's bitbucket site26:

sudo apt install -y libpurple-dev libjson-glib-dev libglib2.0-dev libprotobuf-c-dev protobuf-c-compiler mercurial make
hg clone https://bitbucket.org/EionRobb/purple-hangouts/
cd purple-hangouts
make
sudo make install

How to connec to Facebook and/or Workplace by Facebook with Pidgin (or libpurple)

The Workplace support is not yet merged into the main code: it's in the wip-work-chat branch. More information in the support ticket27.

Workplace and it's 'buddy' list is sort of a mystery at this point in time, so don't expect everything to run completely smooth, but it's much better, than nothing.

In order to log in to a Workplace account, tick Login as Workplace account on the Advanced tab.

git clone https://github.com/dequis/purple-facebook
cd purple-facebook
git checkout wip-work-chat
./autogen.sh
./configure
make
sudo make install

How to conect to Telegram with Pidgin (or libpurple)

The Telegram plugin works nicely, including inline images and and to end encrypted messages. Voice supports seems to be lacking unfortunately.

sudo apt install libgcrypt20-dev libwebp-dev
git clone https://github.com/majn/telegram-purple
cd telegram-purple
git submodule update --init --recursive
./configure
make
sudo make install

How to connect to WhatsApp with Pidgin (or libpurple)

Did I mention I hate this network? First of all a note: WhatsApp doesn't allow 3rd party applications at all. They might ban the phone number you use for life. This ban may be extended to Facebook with the same phone number but this has never been officially confirmed.

Apart from that it needs a lot of hacking around: the plugin is not enough, because WhatsApp doesn't tell you your password. In order to get your password, you need to fake a 'registration' from the computer.

Even if you do this, only one device will work: the other instances will get logged out, so there is no way to use WhatsApp from your phone and from your laptop. It's 2007 again, except it's mobile only instead of desktop only.

Please stop using WhatsApp and use something with a tad more openness in it; XMPP, Telegram, SIP, ICQ... basically anything.

If you're stuck with needing to communicate with stubborn and lazy people, like I am, continue reading, and install the plugin for pidgin:

sudo apt install libprotobuf-dev protobuf-compiler
git clone https://github.com/jakibaki/whatsapp-purple/
cd whatsapp-purple
make
sudo make install

However, this is not enough: the next step is yowsup, a command line python utility that allows you to 'register' to WhatsApp and reveals that so well hidden password.

sudo pip3 install yowsup

Once done, you need to first request an SMS, meaning you'll need a number that's able to receive SMS. Replace the COUNTRYCODE and PHONENUMBER string with your country code and phone number without prefixes, so for United Kingdom, that would be:

  • country code: 44
  • phone number: 441234567890

No 00, or + before the full international phone number.

$ yowsup-cli registration --requestcode sms -p PHONENUMBER --cc COUNTRYCODE --env android

    yowsup-cli  v2.0.15
    yowsup      v2.5.7

    Copyright (c) 2012-2016 Tarek Galal
    http://www.openwhatsapp.org

    This software is provided free of charge. Copying and redistribution is
    encouraged.

    If you appreciate this software and you would like to support future
    development please consider donating:
    http://openwhatsapp.org/yowsup/donate


    INFO:yowsup.common.http.warequest:b'{"login":"PHONENUMBER","status":"sent","length":6,"method":"sms","retry_after":78,"sms_wait":78,"voice_wait":65}\n'
    status: b'sent'
    length: 6
    method: b'sms'
    retry_after: 78
    login: b'PHONENUMBER'

Once you got the SMS, use the secret code:

$ yowsup-cli registration --register SECRET-CODE -p PHONENUMBER --cc COUNTRYCODE --env android

    yowsup-cli  v2.0.15
    yowsup      v2.5.7

    Copyright (c) 2012-2016 Tarek Galal
    http://www.openwhatsapp.org

    This software is provided free of charge. Copying and redistribution is
    encouraged.

    If you appreciate this software and you would like to support future
    development please consider donating:
    http://openwhatsapp.org/yowsup/donate

    INFO:yowsup.common.http.warequest:b'{"status":"ok","login":"PHONENUMBER","type":"existing","edge_routing_info":"CAA=","chat_dns_domain":"sl","pw":"[YOUR WHATSAPP PASSWORD YOU NEED TO COPY]=","expiration":4444444444.0,"kind":"free","price":"$0.99","cost":"0.99","currency":"USD","price_expiration":1520591114}\n'
    status: b'ok'
    login: b'PHONENUMBER'
    pw: b'YOUR WHATSAPP PASSWORD YOU NEED TO COPY'
    type: b'existing'
    expiration: 4444444444.0
    kind: b'free'
    price: b'$0.99'
    cost: b'0.99'
    currency: b'USD'
    price_expiration: 1520591114

That YOUR WHATSAPP PASSWORD YOU NEED TO COPY is the password you need to put in the password field of the account; the username is your PHONENUMBER.

How to connect to WeChat with Pidgin (or libpurple)

If there is something worse, than WhatsApp, it's WeChat: app only and rather agressive when it comes to accessing private data on the phone. If you want to use it, but avoid actually serving data to it, I recommend getting the Xposed Framework28 with XPrivacyLua29 on your phone before WeChat and restricting WeChat with it as much as possible.

sudo apt install cargo clang
git clone https://github.com/sbwtw/pidgin-wechat
cd pidgin-wechat
cargo build
sudo cp target/debug/libwechat.so /usr/lib/purple-2/

Pidgin will only ask for a username - fill that in with you WeChat username and connect. Pidgin will soon pop up a window with a QR code - scan it with the WeChat app and follow the process on screen.

Other networks

Pidgin has a list of third party plugins30, but it's outdated. I've been searching for forks and networks missing from the list on Github.

Extra Plugins for Pidgin

Purple Plugin Pack

There are a few useful plugins for Pidgin that can make life simpler; the Purple Plugin Pack31 contains most of the ones in my list:

  • Highlight
  • IRC helper
  • Message Splitter
  • XMPP Priority
  • Join/Part Hiding
  • Markerline
  • Message Timestamp Formats
  • Nick Change Hiding
  • Save Conversation Order
  • Voice/Viceo Settings
  • XMPP Service Discovery
  • Mystatusbox (Show Statusboxes)

XMPP Message Carbons

XEP-0280 Message Carbons32 is an extension that allows multiple devices to receive all messages.

sudo apt install libpurple-dev libglib2.0-dev libxml2-dev
git clone https://github.com/gkdr/carbons
cd carbons
make
sudo make install

Once installed, open a chat or conversation that happens on the relevant server and type:

/carbons on

This will not be delivered as message but executed on the server as command. Unfortunately not all of the XMPP servers support this.

OMEMO

OMEMO33 is a multi-legged encryption protocol that allows encrypted messages across multiple devices. It's built-in into Conversations34, one of the best XMPP clients for Android - Pidgin doesn't have it by default.

sudo apt install git cmake libpurple-dev libmxml-dev libxml2-dev libsqlite3-dev libgcrypt20-dev
git clone https://github.com/gkdr/lurch/
cd lurch
git submodule update --init --recursive
make
sudo make install

Message Delivery Receipts35

Yet another missing by default XMPP extension, which is quite useful.

git clone https://git.assembla.com/pidgin-xmpp-receipts.git 
cd pidgin-xmpp-receipts/
make
sudo cp xmpp-receipts.so /usr/lib/purple-2/

Porting old logs to Pidgin

I wrote a Python script which can port some old logs into Pidgin. It can deal with unmodifies logs from:

  • Trillian (v3.x)
  • MSN Plus! HTML logs
  • Skype (v2.x)

As for ZNC and Facebook, a lot of handywork is needed - see the comments in the script.

Requirements:

pip3 install arrow bs4

And the script:

import os
import sqlite3
import logging
import re
import glob
import sys
import hashlib
import arrow
import argparse
from bs4 import BeautifulSoup
import csv


def logfilename(dt, nulltime=False):
    if nulltime:
        t = '000000'
    else:
        t = dt.format('HHmmss')

    return "%s.%s%s%s.txt" % (
        dt.format("YYYY-MM-DD"),
        t,
        dt.datetime.strftime("%z"),
        dt.datetime.strftime("%Z")
    )


def logappend(fpath,dt,sender,msg):
    logging.debug('appending log: %s' % (fpath))
    with open(fpath, 'at') as f:
        f.write("(%s) %s: %s\n" % (
        dt.format('YYYY-MM-DD HH:mm:ss'),
        sender,
        msg
    ))
    os.utime(fpath, (dt.timestamp, dt.timestamp))
    os.utime(os.path.dirname(fpath), (dt.timestamp, dt.timestamp))


def logcreate(fpath,contact, dt,account,plugin):
    logging.debug('creating converted log: %s' % (fpath))
    if not os.path.exists(fpath):
        with open(fpath, 'wt') as f:
            f.write("Conversation with %s at %s on %s (%s)\n" % (
                contact,
                dt.format('ddd dd MMM YYYY hh:mm:ss A ZZZ'),
                account,
                plugin
            ))


def do_facebook(account, logpathbase):
    plugin = 'facebook'

    # the source for message data is from a facebook export
    #
    # for the buddy loookup: the  pidgin buddy list xml (blist.xml) has it, but
    # only after the alias was set for every facebook user by hand
    # the file contains lines constructed:
    # UID\tDisplay Nice Name
    #
    lookupf = os.path.expanduser('~/tmp/facebook_lookup.csv')
    lookup = {}
    with open(lookupf, newline='') as csvfile:
        reader = csv.reader(csvfile, delimiter='\t')
        for row in reader:
            lookup.update({row[1]: row[0]})

    # the csv file for the messages is from the Facebook Data export
    # converted with https://pypi.python.org/pypi/fbchat_archive_parser
    # as: fbcap messages.htm -f csv > ~/tmp/facebook-messages.csv
    dataf = os.path.expanduser('~/tmp/facebook-messages.csv')
    reader = csv.DictReader(open(dataf),skipinitialspace=True)
    for row in reader:
        # skip conversations for now because I don't have any way of getting
        # the conversation id
        if ', ' in row['thread']:
            continue

        # the seconds are sometimes missing from the timestamps
        try:
            dt = arrow.get(row.get('date'), 'YYYY-MM-DDTHH:mmZZ')
        except:
            try:
                dt = arrow.get(row.get('date'), 'YYYY-MM-DDTHH:mm:ssZZ')
            except:
                logging.error('failed to parse entry: %s', row)

        dt = dt.to('UTC')
        contact = lookup.get(row.get('thread'))
        if not contact:
            continue
        msg = row.get('message')
        sender = row.get('sender')

        fpath = os.path.join(
            logpathbase,
            plugin,
            account,
            contact,
            logfilename(dt, nulltime=True)
        )

        if not os.path.isdir(os.path.dirname(fpath)):
            os.makedirs(os.path.dirname(fpath))
        logcreate(fpath, contact, dt, account, plugin)
        logappend(fpath, dt, sender, msg)


def do_zncfixed(znclogs, logpathbase, znctz):
    # I manually organised the ZNC logs into pidgin-like
    # plugin/account/contact/logfiles.log
    # structure before parsing them
    LINESPLIT = re.compile(
        r'^\[(?P<hour>[0-9]+):(?P<minute>[0-9]+):(?P<second>[0-9]+)\]\s+'
        r'<(?P<sender>.*?)>\s+(?P<msg>.*)$'
    )
    searchin = os.path.join(
        znclogs,
        '**',
        '*.log'
    )
    logs = glob.glob(searchin, recursive=True)
    for log in logs:
        contact = os.path.basename(os.path.dirname(log))
        account = os.path.basename(os.path.dirname(os.path.dirname(log)))
        plugin = os.path.basename(os.path.dirname(os.path.dirname(os.path.dirname(log))))
        logging.info('converting log file: %s' % (log))
        dt = arrow.get(os.path.basename(log).replace('.log', ''), 'YYYY-MM-DD')
        dt = dt.replace(tzinfo=znctz)


        if contact.startswith("#"):
            fname = "%s.chat" % (contact)
        else:
            fname = contact

        fpath = os.path.join(
            logpathbase,
            plugin,
            account,
            fname,
            logfilename(dt)
        )

        if not os.path.isdir(os.path.dirname(fpath)):
            os.makedirs(os.path.dirname(fpath))

        with open(log, 'rb') as f:
            for line in f:
                line = line.decode('utf8', 'ignore')
                match = LINESPLIT.match(line)
                if not match:
                    continue
                dt = dt.replace(
                    hour=int(match.group('hour')),
                    minute=int(match.group('minute')),
                    second=int(match.group('second'))
                )
                logcreate(fpath, contact, dt, account, plugin)
                logappend(fpath, dt, match.group('sender'), match.group('msg'))


def do_msnplus(msgpluslogs, logpathbase, msgplustz):
    NOPAR = re.compile(r'\((.*)\)')
    NOCOLON = re.compile(r'(.*):?')

    searchin = os.path.join(
        msgpluslogs,
        '**',
        '*.html'
    )
    logs = glob.glob(searchin, recursive=True)
    plugin = 'msn'
    for log in logs:
        logging.info('converting log file: %s' % (log))
        contact = os.path.basename(os.path.dirname(log))

        with open(log, 'rt', encoding='UTF-16') as f:
            html = BeautifulSoup(f.read(), "html.parser")
            account = html.find_all('li', attrs={'class':'in'}, limit=1)[0]
            account = NOPAR.sub('\g<1>', account.span.string)
            for session in html.findAll(attrs={'class': 'mplsession'}):
                dt = arrow.get(
                    session.get('id').replace('Session_', ''),
                    'YYYY-MM-DDTHH-mm-ss'
                )
                dt = dt.replace(tzinfo=msgplustz)
                seconds = int(dt.format('s'))

                fpath = os.path.join(
                    logpathbase,
                    plugin,
                    account,
                    contact,
                    logfilename(dt)
                )

                if not os.path.isdir(os.path.dirname(fpath)):
                    os.makedirs(os.path.dirname(fpath))

                for line in session.findAll('tr'):
                    if seconds == 59:
                        seconds = 0
                    else:
                        seconds = seconds + 1

                    tspan = line.find(attrs={'class': 'time'}).extract()
                    time = tspan.string.replace('(', '').replace(')','').strip().split(':')

                    sender = line.find('th').string
                    if not sender:
                        continue

                    sender = sender.strip().split(':')[0]
                    msg = line.find('td').get_text()

                    mindt = dt.replace(
                        hour=int(time[0]),
                        minute=int(time[1]),
                        second=int(seconds)
                    )

                    logcreate(fpath, contact, dt, account, plugin)
                    logappend(fpath, mindt, sender, msg)


def do_trillian(trillianlogs, logpathbase, trilliantz):
    SPLIT_SESSIONS = re.compile(
        r'^Session Start\s+\((?P<participants>.*)?\):\s+(?P<timestamp>[^\n]+)'
        r'\n(?P<session>(?:.|\n)*?)(?=Session)',
        re.MULTILINE
    )

    SPLIT_MESSAGES = re.compile(
        r'\[(?P<time>[^\]]+)\]\s+(?P<sender>.*?):\s+'
        r'(?P<msg>(?:.|\n)*?)(?=\n\[|$)'
    )

    searchin = os.path.join(
        trillianlogs,
        '**',
        '*.log'
    )

    logs = glob.glob(searchin, recursive=True)
    for log in logs:
        if 'Channel' in log:
            logging.warn(
                "Group conversations are not supported yet, skipping %s" % log
            )
            continue

        logging.info('converting log file: %s' % (log))
        contact = os.path.basename(log).replace('.log', '')
        plugin = os.path.basename(os.path.dirname(os.path.dirname(log))).lower()

        with open(log, 'rb') as f:
            c = f.read().decode('utf8', 'ignore')

            for session in SPLIT_SESSIONS.findall(c):
                participants, timestamp, session = session
                logging.debug('converting session starting at: %s' % (timestamp))
                participants = participants.split(':')
                account = participants[0]
                dt = arrow.get(timestamp, 'ddd MMM DD HH:mm:ss YYYY')
                dt = dt.replace(tzinfo=trilliantz)
                fpath = os.path.join(
                    logpathbase,
                    plugin,
                    participants[0],
                    contact,
                    logfilename(dt)
                )

                if not os.path.isdir(os.path.dirname(fpath)):
                    os.makedirs(os.path.dirname(fpath))

                seconds = int(dt.format('s'))
                curr_mindt = dt
                for line in SPLIT_MESSAGES.findall(session):
                    # this is a fix for ancient trillian logs where seconds
                    # were missing
                    if seconds == 59:
                        seconds = 0
                    else:
                        seconds = seconds + 1

                    time, sender, msg = line
                    try:
                        mindt = arrow.get(time,
                        'YYYY.MM.DD HH:mm:ss')
                    except:
                        time = time.split(':')
                        mindt = dt.replace(
                            hour=int(time[0]),
                            minute=int(time[1]),
                            second=int(seconds)
                        )

                    # creating the filw with the header has to be here to
                    # avoid empty or status-messages only files
                    logcreate(fpath, participants[1], dt, account, plugin)
                    logappend(fpath, mindt, sender, msg)

            if params.get('cleanup'):
                print('deleting old log: %s' % (log))
                os.unlink(log)


def do_skype(skypedbpath, logpathbase):
    db = sqlite3.connect(skypedbpath)

    cursor = db.cursor()
    cursor.execute('''SELECT `skypename` from Accounts''')
    accounts = cursor.fetchall()
    for account in accounts:
        account = account[0]
        cursor.execute('''
        SELECT
            `timestamp`,
            `dialog_partner`,
            `author`,
            `from_dispname`,
            `body_xml`
        FROM
            `Messages`
        WHERE
            `chatname` LIKE ?
        ORDER BY
            `timestamp` ASC
        ''', ('%' + account + '%',))

        messages = cursor.fetchall()
        for r in messages:
            dt = arrow.get(r[0])
            dt = dt.replace(tzinfo='UTC')
            fpath = os.path.join(
                logpathbase,
                account,
                r[1],
                logfilename(dt, nulltime=True)
            )

            if not os.path.isdir(os.path.dirname(fpath)):
                os.makedirs(os.path.dirname(fpath))

            logcreate(fpath, r[1], dt, account, 'skype')
            logappend(fpath, dt, r[3], r[4])


if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Parameters for Skype v2 logs to Pidgin logs converter')

    parser.add_argument(
        '--skype_db',
        default=os.path.expanduser('~/.skype/main.db'),
        help='absolute path to skype main.db'
    )

    parser.add_argument(
        '--pidgin_logs',
        default=os.path.expanduser('~/.purple/logs/skype'),
        help='absolute path to Pidgin skype logs'
    )

    parser.add_argument(
        '--facebook_account',
        default='',
        help='facebook account name'
    )

    parser.add_argument(
        '--loglevel',
        default='warning',
        help='change loglevel'
    )

    for allowed in ['skype', 'trillian', 'msnplus', 'znc', 'facebook']:
        parser.add_argument(
            '--%s' % allowed,
            action='store_true',
            default=False,
            help='convert %s logs' % allowed
        )

        if allowed != 'skype' or allowed != 'facebook':
            parser.add_argument(
                '--%s_logs' % allowed,
                default=os.path.expanduser('~/.%s/logs' % allowed),
                help='absolute path to %s logs' % allowed
            )

            parser.add_argument(
                '--%s_timezone' % allowed,
                default='UTC',
                help='timezone name for %s logs (eg. US/Pacific)' % allowed
            )

    params = vars(parser.parse_args())

    # remove the rest of the potential loggers
    while len(logging.root.handlers) > 0:
        logging.root.removeHandler(logging.root.handlers[-1])

    LLEVEL = {
        'critical': 50,
        'error': 40,
        'warning': 30,
        'info': 20,
        'debug': 10
    }

    logging.basicConfig(
        level=LLEVEL[params.get('loglevel')],
        format='%(asctime)s - %(levelname)s - %(message)s'
    )

    if params.get('facebook'):
        logging.info('facebook enabled')
        do_facebook(
            params.get('facebook_account'),
            params.get('pidgin_logs')
        )


    if params.get('skype'):
        logging.info('Skype enabled; parsing skype logs')
        do_skype(
            params.get('skype_db'),
            params.get('pidgin_logs')
        )

    if params.get('trillian'):
        logging.info('Trillian enabled; parsing trillian logs')
        do_trillian(
            params.get('trillian_logs'),
            params.get('pidgin_logs'),
            params.get('trillian_timezone'),
        )

    if params.get('msnplus'):
        logging.info('MSN Plus! enabled; parsing logs')
        do_msnplus(
            params.get('msnplus_logs'),
            params.get('pidgin_logs'),
            params.get('msnplus_timezone'),
        )

    if params.get('znc'):
        logging.info('ZNC enabled; parsing znc logs')
        do_zncfixed(
            params.get('znc_logs'),
            params.get('pidgin_logs'),
            params.get('znc_timezone'),
        )

  1. http://www.irc.org/↩

  2. https://www.mirc.com/↩

  3. https://icq.com/↩

  4. https://medium.com/@Dimitryophoto/icq-20-years-is-no-limit-8734e1eea8ea↩

  5. https://en.wikipedia.org/wiki/Windows_Live_Messenger↩

  6. https://www.trillian.im/↩

  7. http://pidgin.im/↩

  8. https://xmpp.org/↩

  9. https://xmpp.org/extensions/xep-0280.html↩

  10. https://en.wikipedia.org/wiki/Whatsapp↩

  11. https://en.wikipedia.org/wiki/BlackBerry_Messenger↩

  12. https://en.wikipedia.org/wiki/Google_talk↩

  13. https://en.wikipedia.org/wiki/Google_Hangouts↩

  14. https://en.wikipedia.org/wiki/MQTT↩

  15. http://www.salimvirani.com/facebook/↩

  16. https://www.facebook.com/workplace↩

  17. https://www.blog.google/products/g-suite/move-projects-forward-one-placehangouts-chat-now-available/↩

  18. https://telegram.org/↩

  19. https://xmpp.org/extensions/xep-0384.html↩

  20. https://matrix.org/↩

  21. https://xmpp.org/extensions/xep-0384.html↩

  22. https://xmpp.org/extensions/xep-0166.html↩

  23. https://play.google.com/store/apps/details?id=com.mailsite.astrachat↩

  24. https://f-droid.org/packages/eu.siacs.conversations/↩

  25. https://www.linphone.org/↩

  26. https://bitbucket.org/EionRobb/purple-hangouts/src#markdown-header-compiling↩

  27. https://github.com/dequis/purple-facebook/issues/371↩

  28. http://repo.xposed.info/↩

  29. https://lua.xprivacy.eu/↩

  30. https://developer.pidgin.im/wiki/ThirdPartyPlugins↩

  31. https://bitbucket.org/rekkanoryo/purple-plugin-pack/↩

  32. https://xmpp.org/extensions/xep-0280.html↩

  33. https://xmpp.org/extensions/xep-0384.html↩

  34. https://f-droid.org/packages/eu.siacs.conversations/↩

  35. https://xmpp.org/extensions/xep-0184.html↩

]]> 2018-03-03T15:00:00+00:00 2018 Peter Molnar CC BY 4.0 https://petermolnar.net/odexed-android-6-marshmallow-with-microg How to install microG an odexed stock android ROM 2018-02-24T10:00:00+00:00 Why would anyone want an android without Google?

About 1.5 years ago I attended to a wedding. It took place outside of the city, at a restaurant with a very nice garden, where I've never before. In about 2 hours into the happening, my phone buzzed. I took it out, expecting a text message, or similar, but no: it was Google Maps, asking me if I'm at the place where I am, and since I'm there, could I upload some pictures of the place?

Since then this had become regular, until the point it became obstructive and annoying. I'm not alone: Brad Frost's1 entry talks about the same problem. I've tried everything to turn Google location tracking off. I went to Location History2 and disabled it. I went through the Google Maps application and removed every notification. This latter one cleared the madness for a while - until a new version of Maps came up, which introduced some extra notification setting, which then showed yet another popup out of the blue.

Google fails to respect user settings lately, turning into a desperately annoying data hoarding monster. They've been doing nasty things, like silently collecting cell tower information, even with location settings disabled for years3, but with the coming of GDPR4 they need to get consent - hence the silly amount of notifications they are bombarding you with.

Once I set a setting in a service, I expect it to stay they way I set it. I expect backwards compatibility, backfilled data, if needed. Google and Facebook are both failing at this; Facebook always had, Google only recently started. New app, we renamed all the settings, let's reset them to the default level of annoyance!

The whole problem on android can be traced back to one omnipotent application: Google Services Framework. This silent, invisible beast upgrades itself and Play Store whenever, wherever it wants it to. It does this all in the background, not even letting you know. If you happen to run an ancient phone, like my HTC Desire5, it will fill up that generous 150MB you have for user apps without a blink of an eye, and let you wonder why your phone can't boot any more.

The extremely sad part is that everyone started depending on GMS - Google Mobile Services - for convenience: it provides push services, you don't have to run you own. It all leads to the point that android, while in theory is Open Source, it will never be Free from Google, in it's current form.

Enter microG6: a few enthusiasts with the same feelings as me, but with actual results. microG is a service level replacement for Google; it's Free and Open Source, it's transparent. There's only one problem: it's very tricky to install it on nieche phones, with odexed ROMs.

So I made a guide. While this guide was made using a Nomu S107, but given it's an AOSP based ROM with tiny modifications, I'm fairly certain it can be applied to any similar, no-name less known brands and phones.

Important notes

The methods below might void your warranty. It might brick your phone. It will take while. It can cause an unexpected amount of hair pulled out.

Never do it on your only phone, or a phone you value high. I take no responsibility if anything goes wrong.

It was done on a Nomu S10, with Android 6.0 Marshmallow. It will most probably need to be altered for other versions.

The heavy lifting, the work, are all done by magnificent people out there; this article is merely summary of existing knowledge.

The only thing I can assure is that it worked for me, but it took a weekend to get to the bottom of it.

Prerequisites

Operating system, adb, fastboot

The guide was made for Debian based linux, including Ubuntu and Mint.

I assume you have a general understanding and familiarity with fastboot and adb - these are both needed during the process.

It's doable on Windows as well, with very similar steps, but I don't have a Windows, so I can't make that guide.

SP Flash Tool (flashing stock ROMs on MediaTek devices)

The stock ZIPs Nomu provide can't be flashed via the regular recoveries, like TWRP. As a workaround, I used to extract it and flash the pieces by fastboot - that is because I wasn't aware of a tool, called SP Flash Tool and the MediaTek download mode.

When the phone is turned off and connected to a computer via USB, it shows up as modem (!) device, as ttyACM. The SP Flash tool uses this to flash the ROM, but in order to do that - even if the flash tool is run by root - needs some tweaking on the linux side.

In order to get this supported on Debian, some udev rules need to be added:

Run (as root):

cat > etc/udev/rules.d/20-mediatek-blacklist.rules << EOF
ATTRS{idVendor}=="0e8d", ENV{ID_MM_DEVICE_IGNORE}="1"
ATTRS{idVendor}=="6000", ENV{ID_MM_DEVICE_IGNORE}="1"
EOF

cat > etc/udev/rules.d/80-mediatek-usb.rules << EOF
SUBSYSTEM=="usb", ACTION=="add", ATTR{idVendor}=="0e8d", ATTR{idProduct}=="*"
EOF

systemctl restart udev.service

Once done add your user to the dialout and uucp groups as:

usermod -a -G dialout,uucp YOUR_USERNAME

Sp Flash tool needs an old version of libpng12, so get that from the Debian packages, or from the jessie (oldstable) repository:

wget http://ftp.uk.debian.org/debian/pool/main/libp/libpng/libpng12-0_1.2.50-2+deb8u3_amd64.deb
dpkg -i libpng12-0_1.2.50-2+deb8u3_amd64.deb
rm libpng12-0_1.2.50-2+deb8u3_amd64.deb

This should make it possible to flash, using the SP flash tool, which can be downloaded from spflashtool.com8.

Credit due to Miss Montage on needrom.com9 for finding these out.

TWRP recovery for Nomu S10

Do not flash TWRP recovery on the Nomu S10. There is some kind of safety check, which wants to trigger a factory reset, so what happens in short is the following: - you flash system, boot, cache from stock - you flash TWRP as recovery - the phone boots - it re-locks the OEM unlock switch - it removes TWRP

At this point it's stuck with a boot logo on screen (not even a bootloop); it can't even be turned off, it's without a working system, no recovery, and a locked fastboot. If you reach this point, use SP Flash Tool described above.

Instead of flashing, a method called dirty boot will be used: via fastboot, the TWRP image will be booted from the PC, not from the phone, because TWRP is still neede in order to flash custom ZIPs. Jemmini10 made one for the Nomu S10; it in a zip11 ; extract it to get a .img file.

SuperSu flashable ZIP

Many things have changed since the early days of rooting android (2.3 and 4.x). Nowadays fiddling with the system /bin might result in mayhem, so clever people came up with the idea to put su, the actual binary needed for rooting into the boot image, ending up in /sbin, without triggering any "security". One of these systemless rooting apps is SuperSu12, which you'll need in a flashable zip13 format.

Xposed framework ZIP and apk

A vast amount of Android's real potential is lock behind bars - the reason is "security". I'm putting this in quotes, because it's smoke and mirrors: malware can use vulnerabilities to install itself deep inside the system that it's impossible to even detect it, yet you're not allowed to have full access on your phone. Not even the security suits and the malware scanners ask for root, and without that, they are not much more than a paid, bad joke.

Anyway: the XPosed Framework14 is here to help. It's a utility which lets you install modules which modules tweak low level android behaviour. For example, Volumesteps+ will let you change the granularity of volume steps, which is very useful for someone who'd find a volume level between the factory 8 and 9 the best. For us, the important module will be FakeGApps, which allows signature spoofing15, which is a hard requirement for microG to work.

For reasons I'm yet to understand, I had to both flash the zip16 and install the apk17 version to get XPosed on the phone. In theory, only one should have been enough, but for now, get both.

NanoDroid microG ZIP

NanoDroid18 is originally a Magisk module (it's another systemless rooting and framework, but I could never get it running on the Nomu), but it's also available as flashable ZIPs.

For our needs, only the NanoDroid microG zip19 is required.

The actual un-googling

Summary

  1. root the phone and install xposed
  2. enable signature spoofing via xposed module fakegapps
  3. removed all google related apps, libs, entries
  4. install microg

Now: in details.

1. Rooting the Nomu S10 with SuperSu and Xposed

Important: OEM unlocking will trigger a factory reset, it will wipe every user and app setting from the phone.

  1. Start android
  2. Enable Developer Options:
    • go into Settings
    • About Phone
    • tap on the Build Number until you get the message 'you are now a developer!'
      Build number in About Phone
      Build number in About Phone
  3. Enable OEM unlocking
    • go into Settings
    • Developer
    • allow oem unlocking
  4. Enable USB Debugging (ADB)
    • go into Settings
    • Developer
    • allow usb debugging
      OEM unlock and USB debugging options
      OEM unlock and USB debugging options
  5. reboot into fastboot:
    • run adb reboot bootloader from the host, or:
    • power off the phone, then press and hold Volume Up and Power

Fastboot menu

Fastboot menu
  1. in fastboot, issue the oem unlock command:
    • fastboot oem unlock from the host
    • press Volume up as asked on the phones

Fastboot menu

Fastboot menu
  1. reboot the phone: fastboot reboot
  2. let the factory reset flush through
  3. re-enable Developer Options (see 2.)
  4. verify that OEM unlocking is on - is should be. If not, go back to 1. and start again.
  5. re-enable USB debugging (see 3.)
  6. boot into fastboot (see 5.)
  7. "dirty" boot TWRP recovery:
    • fastboot boot [path to the twrp image - the one extracted from the zip]
  8. install SuperSu via TWRP
    • either put the zips on a microSD card and flash it as regular zips
    • or use adb sideload from the Advanced menu in TWRP
  9. install Xposed via TWRP (see 14.)
  10. reboot the phone
  11. install the Xposed apk

2. Enable signature spoofing with FakeGapps

Once the Xposed Installer is up and running it will look like this:

Running and enabled Xposed
Running and enabled Xposed

Under Menu, Download search for FakeGApps20, click Download and enable it:

FakeGApps in Xposed
FakeGApps in Xposed

For FakeGApps to take effect, the phone has to be rebooted, but it can be done together with the next step.

3. Remove all google apps and libraries

These commands should be run either via an adb shell, or from a Terminal Emulator on the phone.

adb shell

# become root - SuperSu will prompt for verification
su -
# remount / and /system for read-write
mount -o remount,rw /
mount -o remount,rw /system
# create a file with the list it items to delete
cat > /sdcard/ungoogle.sh << EOF
rm -rf /system/app/CarHomeGoogle*
rm -rf /system/app/ChromeBookmarksSyncAdapter*
rm -rf /system/app/ConfigUpdater*
rm -rf /system/app/FaceLock*
rm -rf /system/app/GenieWidget*
rm -rf /system/app/Gmail*
rm -rf /system/app/Gmail2
rm -rf /system/app/GmsCore*
rm -rf /system/app/Google*
rm -rf /system/app/LatinImeTutorial*
rm -rf /system/app/LatinImeDictionaryPack*
rm -rf /system/app/MarketUpdater*
rm -rf /system/app/MediaUploader*
rm -rf /system/app/NetworkLocation*
rm -rf /system/app/OneTimeInitializer*
rm -rf /system/app/Phonesky*
rm -rf /system/app/PlayStore*
rm -rf /system/app/SetupWizard*
rm -rf /system/app/Talk*
rm -rf /system/app/Talkback*
rm -rf /system/app/Vending*
rm -rf /system/app/VoiceSearch*
rm -rf /system/app/VoiceSearchStub*
rm -rf /system/etc/permissions/com.google.android.maps.xml
rm -rf /system/etc/permissions/com.google.android.media.effects.xml
rm -rf /system/etc/permissions/com.google.widevine.software.drm.xml
rm -rf /system/etc/permissions/features.xml
rm -rf /system/etc/preferred-apps/google.xml
rm -rf /system/etc/g.prop
rm -rf /system/addon.d/70-gapps.sh
rm -rf /system/framework/com.google.android.maps.jar
rm -rf /system/framework/com.google.android.media.effects.jar
rm -rf /system/framework/com.google.widevine.software.drm.jar
rm -rf /system/lib/libfilterpack_facedetect.so
rm -rf /system/lib/libfrsdk.so
rm -rf /system/lib/libgcomm_jni.so
rm -rf /system/lib/libgoogle_recognizer_jni.so
rm -rf /system/lib/libgoogle_recognizer_jni_l.so
rm -rf /system/lib/libfacelock_jni.so
rm -rf /system/lib/libfacelock_jni.so
rm -rf /system/lib/libgtalk_jni.so
rm -rf /system/lib/libgtalk_stabilize.so
rm -rf /system/lib/libjni_latinimegoogle.so
rm -rf /system/lib/libflint_engine_jni_api.so
rm -rf /system/lib/libpatts_engine_jni_api.so
rm -rf /system/lib/libspeexwrapper.so
rm -rf /system/lib/libvideochat_stabilize.so
rm -rf /system/lib/libvoicesearch.so
rm -rf /system/lib/libvorbisencoder.so
rm -rf /system/lib/libpicowrapper.so
rm -rf /system/priv-app/CarHomeGoogle*
rm -rf /system/priv-app/ChromeBookmarksSyncAdapter*
rm -rf /system/priv-app/ConfigUpdater*
rm -rf /system/priv-app/FaceLock*
rm -rf /system/priv-app/GenieWidget*
rm -rf /system/priv-app/Gmail*
rm -rf /system/priv-app/GmsCore*
rm -rf /system/priv-app/Google*
rm -rf /system/priv-app/LatinImeTutorial*
rm -rf /system/priv-app/LatinImeDictionaryPack*
rm -rf /system/priv-app/MarketUpdater*
rm -rf /system/priv-app/MediaUploader*
rm -rf /system/priv-app/NetworkLocation*
rm -rf /system/priv-app/OneTimeInitializer*
rm -rf /system/priv-app/Phonesky*
rm -rf /system/priv-app/PlayStore*
rm -rf /system/priv-app/SetupWizard*
rm -rf /system/priv-app/Talk*
rm -rf /system/priv-app/Talkback*
rm -rf /system/priv-app/Vending*
rm -rf /system/priv-app/VoiceSearch*
rm -rf /system/priv-app/VoiceSearchStub*
EOF
# execut the created list
sh /sdcard/ungoogle.sh

3. Install NanoDroid microG

Once Google is cleaned up and the FakeGapps module is ready, reboot into recovery (see 12. and 13.) and install the NanoDroid zip via TWRP.

If you done everything right, there will be no Google Services or apps left, if not - as I did - a few leftovers will need to be manually cleaned up.

If the microG flashing was successful, an app, called microG settings will show up:

FakeGApps in Xposed
FakeGApps in Xposed

Done!

  1. http://bradfrost.com/blog/post/google-you-creepy-sonofabitch/↩

  2. https://myaccount.google.com/activitycontrols/location↩

  3. https://qz.com/1131515/google-collects-android-users-locations-even-when-location-services-are-disabled/↩

  4. https://www.eugdpr.org/↩

  5. https://en.wikipedia.org/wiki/HTC_Bravo↩

  6. https://microg.org/↩

  7. http://www.nomu.hk/pro/s10_product_show/↩

  8. https://spflashtool.com/download/SP_Flash_Tool_v5.1744_Linux.zip↩

  9. https://www.needrom.com/download/how-to-setup-sp-flash-tool-linux-mtk↩

  10. https://forum.xda-developers.com/showthread.php?t=3482755↩

  11. https://forum.xda-developers.com/attachment.php?attachmentid=3947063↩

  12. http://www.supersu.com/↩

  13. https://s3-us-west-2.amazonaws.com/supersu/download/zip/SuperSU-v2.82-201705271822.zip↩

  14. https://forum.xda-developers.com/showthread.php?t=3034811↩

  15. https://github.com/microg/android_packages_apps_GmsCore/wiki/Signature-Spoofing↩

  16. https://dl-xda.xposed.info/framework/sdk23/arm/xposed-v89-sdk23-arm.zip↩

  17. https://forum.xda-developers.com/attachment.php?attachmentid=4393082↩

  18. http://nanolx.org/nanolx/nanodroid↩

  19. https://downloads.nanolx.org/NanoDroid/Stable/NanoDroid-microG-16.1.20180209.zip↩

  20. http://repo.xposed.info/module/com.thermatk.android.xf.fakegapps↩

]]> 2018-02-24T10:00:00+00:00 2018 Peter Molnar CC BY 4.0 https://petermolnar.net/hol-vasmu-allott-most-kohalom Hol vasmű állott, most kőhalom 2018-02-15T10:00:00+00:00 Just another abandoned, decaying factory in Budapest.

]]> 2018-02-15T10:00:00+00:00 2018 Peter Molnar CC BY-NC-ND 4.0 https://petermolnar.net/hol-vasmu-allott-most-kohalom-2 Hol vasmű állott, most kőhalom 2 2018-02-14T10:00:00+00:00 Empty windows, nature slowly taking over, rubble everywhere - a few decades of decay.

]]> 2018-02-14T10:00:00+00:00 2018 Peter Molnar CC BY-NC-ND 4.0 https://petermolnar.net/linkedin-public-settings-ignored LinkedIn is ignoring user settings 2018-01-14T12:00:00+00:00 A few days ago, on the #indieweb Freenode channel1 one of the users asked if we knew an indieweb-friendly way of getting data out of LinkedIn. I wasn't paying attention to any recent news related to LinkedIn, though I've heard a few things, such as they are struggling to prevent data scraping: the note mentioned that they believe it's a problem that employers keep an eye on changes in LinkedIn profiles via 3rd party. This, indeed, can be an issue, but there are ways to manage this within LinkedIn: your public profile settings2.

In my case, this was set to visible to everyone for years, and by the time I had to set it up (again: years), it was working as intended. But a few days ago, for my surprise, visiting my profile while logged out resulted in this:

LinkedIn showing a paywall-like 'authwall' for profiles set explicitly to public for everyone
LinkedIn showing a paywall-like 'authwall' for profiles set explicitly to public for everyone

and this:

$ wget -O- https://www.linkedin.com/in/petermolnareu
--2018-01-14 10:26:12--  https://www.linkedin.com/in/petermolnareu
Resolving www.linkedin.com (www.linkedin.com)... 91.225.248.129, 2620:109:c00c:104::b93f:9001
Connecting to www.linkedin.com (www.linkedin.com)|91.225.248.129|:443... connected.
HTTP request sent, awaiting response... 999 Request denied
2018-01-14 10:26:12 ERROR 999: Request denied.

or this:

$ curl https://www.linkedin.com/in/petermolnareu
<html><head>
<script type="text/javascript">
window.onload = function() {
  // Parse the tracking code from cookies.
  var trk = "bf";
  var trkInfo = "bf";
  var cookies = document.cookie.split("; ");
  for (var i = 0; i < cookies.length; ++i) {
    if ((cookies[i].indexOf("trkCode=") == 0) && (cookies[i].length > 8)) {
      trk = cookies[i].substring(8);
    }
    else if ((cookies[i].indexOf("trkInfo=") == 0) && (cookies[i].length > 8)) {
      trkInfo = cookies[i].substring(8);
    }
  }

  if (window.location.protocol == "http:") {
    // If "sl" cookie is set, redirect to https.
    for (var i = 0; i < cookies.length; ++i) {
      if ((cookies[i].indexOf("sl=") == 0) && (cookies[i].length > 3)) {
        window.location.href = "https:" + window.location.href.substring(window.location.protocol.length);
        return;
      }
    }
  }

  // Get the new domain. For international domains such as
  // fr.linkedin.com, we convert it to www.linkedin.com
  var domain = "www.linkedin.com";
  if (domain != location.host) {
    var subdomainIndex = location.host.indexOf(".linkedin");
    if (subdomainIndex != -1) {
      domain = "www" + location.host.substring(subdomainIndex);
    }
  }

  window.location.href = "https://" + domain + "/authwall?trk=" + trk + "&trkInfo=" + trkInfo +
      "&originalReferer=" + document.referrer.substr(0, 200) +
      "&sessionRedirect=" + encodeURIComponent(window.location.href);
}
</script>
</head></html>
So I started digging. According to the LinkedIn FAQ3 there is a page where you can set your profile's public visibility. Those settings, for me, were still set to:
LinkedIn public profile settings
LinkedIn public profile settings

Despite the settings, there is no public profile for logged out users.

I'd like to understand what it going on, because so far, this looks like a fat lie from LinkedIn. Hopefully just a bug.

UPDATE

I tried setting referrers and user agents, used different IP addresses, still nothing. I can't type today and managed to mistype https://google.com - the referrer ended up as https:/google.com. So, following the notes on HN, setting a referrer to Google sometimes works. After a few failures it will lock you out again, referrer or not. This is even uglier if it was a proper authwall for everyone.

curl 'https://www.linkedin.com/in/petermolnareu' \
-e 'https://google.com/' \
-H 'accept-encoding: text' -H \
'accept-language: en-US,en;q=0.9,' \
-H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'
<!DOCTYPE html>...

  1. https://chat.indieweb.org↩

  2. https://www.linkedin.com/public-profile/settings↩

  3. https://www.linkedin.com/help/linkedin/answer/83?query=public↩

]]> 2018-01-14T12:00:00+00:00 2018 Peter Molnar CC BY 4.0 https://petermolnar.net/nomu-s10 Living with a rugged, cheap Chinese Android phone, the Nomu S10 2018-01-07T11:00:00+00:00 Disclaimer: I didn't get paid for this entry: I simply liked this phone enough to share the whys and the fixes for it's problems. It also serves as notes for myself.

How the Nomu S10 found me

Nomu S10 after a few months of regular use in pockets and bags
Nomu S10 after a few months of regular use in pockets and bags - photo by Peter Molnar
Camera
PENTAX K-5 II s
Aperture
f/8.0
Shutter speed
1/40 sec
Focal length (as set)
85.0 mm
Sensitivity
ISO 3200
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

About half a year ago we (my wife and me) decided we need to replace our phones - again. The Galaxy S4 was a sad choice after a Nexus 4 and a Galaxy Nexus, but those had to be replaced: they were literally falling apart and none of them supported European LTE bands - 3, 8, 20.

The Samsung Galaxy S4 is a weak, fragile creature, with weird bugs, a bloated, locked system, and with a slippery, rounded-everywhere shape that will certainly land on the tarmac a few times due to this. It has an impressive range of sensors, but that's it, and I wanted a phone which works reliably and which is a bit more rugged.

I don't believe in the current trend of bezel-free design and I'm very tired of all-glass, expensive, overheating, bloated, gargantuan "phones", so I started looking into less-known brands. Initially into names like Oneplus, Xiaomi and some similar, sort of established manufacturers, but soon I learnt getting the actually good models in Europe is quite hard and due to this, nearly none of them covers LTE band 20 - the one giffgaff1, my service provider has access to.

The danger of less known brands is that they can be cheap for very logical reason: many of them are built cheap. On the bright side, the usually bring a surprising but good feature: most of the time these phones feature a near-raw AOSP system, without any bloat, and only minor software modifications.

Not long into my search I found a gem: the Nomu S102 and the Nomu brand in general. Apart from the S10, there is now the S10 Pro3, the S204 and S305 model. All of them are IP68 - waterproof enough to survive being fully submerged under water for a little while, rugged, with huge batteries.

The S10 is a relatively basic phone. It lacks NFC, 5GHz WiFi, and any new, shiny tech - but it's loud, has a very capable 5000mAh battery, and covers more or less anything that falls under regular use. Yes, it's heavy. It also has a metal frame inside - some way it resembles a ThinkPad from the old days. It packs 2GB RAM, 16GB storage, and either expandable with microSD, or it accepts 2 SIMs. The screen is a HD IPS screen, which I prefer to the OLED screens personally. The 2.4GHz Wifi is fast and stable and has a better reception than my previous, high-end phones did.

It also comes with a protective foil by default - actually it comes with 2 layers; after removing the first, a second is still protecting the display. Also has some version of Qualcomm Quick Charge which is also supported by the European charger it came with, which is certainly needed for the monster battery it has.

Though the stock firmware accepted SD cards up to 32GB, for my surprise, one of the system updates made 64GB possible as well - I don't have any bigger ones, but I wouldn't be surprised if the upper limit was 256GB.

Overall it has all the features you'd want from a phone that was built to last and be a companion for rough weather or outdoor activities.

I got a very good phone for the price and it worked well enough that I even used it without rooting.

Same Nomu S10, again after a few months of use
Same Nomu S10, again after a few months of use - photo by Peter Molnar
Camera
PENTAX K-5 II s
Aperture
f/8.0
Shutter speed
1/40 sec
Focal length (as set)
85.0 mm
Sensitivity
ISO 3200
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

Troubles arise: the Triada malware fiasco

One day my wife came home, telling me that her phone started to act weird: it randomly shows full screen popups, installs random software, and generally acting weird.

Knowing the she is quite careful and not installing basically anything on her phone after a few initial, well established apps, it looked very much like either a hack through other networks she connects to or something completely out of our control.

Unfortunately it turned out to be the later: there has been reports that a good amount of similarly cheap Chinese phones got infected with malware - from the manufacturer6! Interestingly the malware did not manifest on my phone, but the possibility that it might got me itchy.

Having a little experience in flashing custom ROMs the solution was obvious: I need to find a malware-free ROM for the Nomu and re-flash them.

The last time I had to dwell deep into Android and flashing was with my beloved HTC Desire - in 2011 - apparently a lot has changed since, and the malware, Triada, managed to sit into a part of the system, called Zygote, which is deep enough that even with root privileges you couldn't get rid of it.

So here is how to get rid of the Triada malware on a Nomu S10 by flashing a malware free ROM.

The search for a malware-free ROM - the downside of niche phones

When you have a widely bought and known phone there are usually plenty of custom ROMs floating around for it. This used to be the case for the Nexus 4, the Galaxy S4, even for the HTC Desire.

This is not the case with the Nomu S10: it's rare, so nobody made a full-on custom ROM for it. I started looking into forums and threads on the topic and thankfully I came across a French forum7 mentioning something called Archos 50 Saphir ROM. I was hoping to find some Cyanogenmod LineageOS based ROM, but for my surprise, Archos is a retailer: they sell Archos branded phones. One of these is what they call 50 Saphir8 - which is a re-branded Nomu S10.

Apparently the S10 had been sold to various "brands" who buy cheap phones, put a logo and a ROM without malware on them, and sell them for double the price.

This is the Archos ROM I ended up using:

https://www.dropbox.com/s/yk2t2mlv1uch0y7/Archos-50-Saphir-14-OTA.zip?dl=0

You can also get it from needrom.com9. It's 1.1GB, so if you're tight on bandwidth, don't grab it immediately.

Another option is to get one of the newer, official Nomu ROMs, from their own site10 - the version 1.1.4, uploaded 2017-11-11 seems to be running fine and malware free for me for the past month:

https://drive.google.com/file/d/1ZXlnlic2z10CSFGoCf7nHygfTWsnFsAq/

I went with this solution, although it's reasonable to be cautious and stick to the Archos version.

OEM Unlocking - the upside of OEM phones

Note: OEM unlocking can trigger a complete factory reset, deleting anything on your phone, excluding the microSD card. I'd still suggest removing even the microSD card and saving everything before starting the process.

Since the S10 has been sold for other brands to be used with their labels, Nomu allowed these resellers to flash their own operating system. This is good news for us: it means we can do the same, and rather simply.

There is a process in newer Androids that protects your phone from being overwritten by a mere USB cable and a laptop which requires a working operating system. If you don't have one, you won't be able to unlock flashing on your phone, so first to a factory reset and then follow these steps.

In your running Android:

  1. enable the Developer mode:
  2. go to Settings
  3. enter About phone
  4. tap Build number approximately 6 times
  5. once you have Developer options under Settings, enter it
  6. enable OEM unlocking

For the next stage, you'll need a tool called fastboot. I'm using Debian, which is a linux distribution, and I have fastboot available from apt (the Debian app store, one could say) . I have no idea how to install it on Windows, but there are many tutorial available on the internet, and the commands should work once you have it. The commands below need to be executed as root user in linux.

Once you have fastboot:

  1. power off your phone
  2. press and hold volume up and the power buttons
  3. you'll be greeted with a little menu, looking like:
Select Boot Mode:
[VOLUME_UP to select, VOLUME_DOW is OK.]

[Recovery   Mode]       <<==
[Fastboot   Mode]
[Normal     Mode]

Select Fastboot mode by pressing volume up and press volume down. The screen will now print a new line:

=> FASTBOOT mode...

On your laptop, connect the USB cable and test if you see the device in your laptop:

fastboot devices
EEMRTK5PBEJ78DWS        fastboot

If you don't see the device, there is something wrong; try to repeat the process from the beginning of OEM unlocking.

Now enter:

fastboot flashing unlock

Note: it used to be fastboot oem unlock. Now it's fastboot flashing unlock. If it doesn't work, try the oem command version.

It will ask for confirmation and you'll have to press the corresponding volume button - read the instructions on the phone, but volume up should enable to unlock.

Congratulations! You can now install any ROM made for your phone.

Things that might go wrong with this:

  • to get out of fastboot press and hold the power button for a while
  • if neither fastboot oem unlock nor fastboot flashing unlock does nothing, do a factory reset, a cache clear, and re-enable OEM unlocking. For this, you need to boot the recovery option instead of fastboot, the select the options.
  • if this didn't work either... well, that's a problem, and I don't have a simple, fast, or working solution.

Flashing the stock ROM

Originally I tried to flash the stock ROM via the stock recovery, but every single time I got an error, telling me the zip file is corrupted. After a while I decided to take another approach.

Once you have the stock ZIP, extract it:

unzip NOMU_S10_COMMON_V1.1.4_2017_11_01_FQ5C62WTE1D.XWP.NOMU.M0.HB.H.SSXSSYDAKLS23.1101.V3.14.zip -d nomu_s10_1.1.4

It will create the following files in the nomu_s10_1.1.4 directory:

APDB_MT6735_S01_alps-mp-m0.mp1_W16.47
boot.img
boot-verified.img
BPLGUInfoCustomAppSrcP_MT6735_S00_MOLY_LR9_W1444_MD_LWTG_MP_V88_P92_1_lwg_n
cache.img
Checksum.ini
lk.bin
lk-verified.bin
logo.bin
logo-verified.bin
md1arm7.img
md1dsp.img
md1rom.img
md3rom.img
MT6737T_Android_scatter.txt
preloader_fq5c62wt_xwp_nomu.bin
preloader.img
recovery.img
recovery-verified.img
secro.img
secro-verified.img
system.img
trustzone.bin
userdata.img
V18S NOMU 软件配置说明.txt

The following will render your phone temporarily useless,; you will be without recovery, left only with a bootloader, for a short period. I seriously recommend only doing this with a fully charged phone and avoiding any accidental reboots during the process.

First, wipe the relevant partitions:

fastboot erase system
fastboot erase boot
fastboot erase recovery
fastboot erase cache

Now flash them:

cd nomu_s10_1.1.4
fastboot flash boot boot.img
fastboot flash recovery recovery.img
fastboot flash cache cache.img
fastboot flash system system.img

It will take a while, be patient.

Flashing the Archos ROM

Just use the same method as above, but instead of extracting the Nomu zip, extract the Archos zip.

However, unlike the stock ROM, the Archos ROM can be installed via adb sideload or simply selecting the zip in the dirty booted TWRP recovery described below. In case you're familiar with custom recovery zip installing then there's no need for the extract magic, but that method works just fine as well.

[Optional] How to root the Nomu S10 - dirty booting TWRP recovery

Dirty boot means we don't flash the recovery partition, only load it on the fly from the laptop and use it temporarily - think about it as a live linux distribution. I found a TWRP custom recovery which worked very well for me; unfortunately I don't remember the source, so I've uploaded it to Dropbox:

TWRP 3.0.2 recovery for Nomu S10:

https://www.dropbox.com/s/yebkei44w9oq6kh/nomu-s10_twrp-3.0.2.img?dl=0

Once you're in the recovery, you can install whatever you want, including SuperSu:

https://www.dropbox.com/s/semoj6evcio3nyw/SuperSU-v2.82-201705271822.zip?dl=0

SuperSu is a "systemless root" method, which means instead of touching the system partition, which triggers an alarm with many root detectors, it puts the su binary into /sbin - which is part of the boot image. It can be simply reverted, should that be needed.

I didn't have luck installing Magisk. While it worked flawlessly on a LineageOS install on a Nexus 10, no matter what I tried, it never worked here, but every single attempt triggered a factory reset. Just use SuperSu. If you do need features Magisk provides, maybe take a look at the Xposed framework instead, that worked well:

https://www.dropbox.com/s/xuqmah3vwh3nsre/xposed-v88.2-sdk23-arm.zip?dl=0

The simplest for these to install is to put them on the microSD card and select them as zip to be installed from TWRP. I will not cover the process this here, there are very good howto about TWRP and installing zips.

Waterproof - as long as you are careful

Not too long ago we spent a few days abroad, next to the sea, and found a tiny pool, left with water during low tide. The pool had a surprising amount if small fish and a small amount of coral in it, and, since the S10 is marketed as IP68, we decided to take a few underwater pictures with it. It's important to say that one of the first things we did when we got the phones was submerge it in a lake to test the waterproofness, and there were not issues, but it was just a few seconds.

It's a terrible underwater camera, so don't use it as one. The touchscreen gets mad and recognizes the water as constant touch, so pushing the camera shutter button is hard and tricky.

But after finishing taking the shoots it seemed like my wife's phone got a leak somewhere. It could have been a tiny bit of looseness in the rubber USB cover, or something completely unrelated, but after a few minutes out of the water it developed weird errors. The screen showed a lot of vertical lines, fading or altering the colour "behind" them, the resolution looked like it fell back to 320x240, and charging worked only sporadically. Apart from this, it was still working, receiving calls, responsive to touch.

After getting home I returned it to Amazon, who, due to the lack of replacement units with the original seller, refunded the value (the phone was less, the 6 months old). In my opinion, and according to the marketing, we did nothing out of ordinary use. We bought another one.

Touchscreen quality issues? (it may have been the only problematic one we got)

After buying the replacement for the defective one the next we received seemed fine - until you poked the touchscreen around the 'a' character on the keyboard. From that point, it started acting like the screen was being touched at multiple locations - would have made a nice stock video for ghost movies.

This was an obvious, immediate return, and we are now with yet another one without glitches - I hope it stays like that.

And you're still recommending it?!

Oh, yes.

It's cheap, with very nice features: small, compared to the battery it packs; runs for days on a single charge even with reasonable use; waterproof enough to survive being out in the rain for a long while, even submerged (when the flaps a strictly closed); the features it has all work; it's "hackable" by default (the good way, unlike the Samsung or HTC phones).

The malware-by-manufacturer is not a unique problem and not strictly a problem of no-name manufacturers. There were reports that similar issues hit well known brands, such as Samsung, LG, Asus - at a point, the problem even included some Nexus 5 phones by Google11.

My sole remaining moan with the phone is that it's glued. To take it apart, you'd need a special machine or a rather precise heat-gun and a steady hand. However, all the ones with screws, like the Blackview BV6000, are much larger and heavier.

So yes: regardless of the potential malware issue, I'm still recommending this phone, but do make sure you have a virus-free OS once before you start using it as your main phone.

  1. https://www.giffgaff.com/orders/affiliate/petermolnar2↩

  2. http://amzn.to/2AioYRp↩

  3. http://amzn.to/2CCxb9e↩

  4. http://amzn.to/2CNBf3a↩

  5. http://amzn.to/2E3E7IK↩

  6. https://news.drweb.com/news/?i=11390&lng=en↩

  7. https://forums.malwarebytes.com/topic/200072-trojantriada↩

  8. http://www.archos.com/gb-en/products/smartphones/saphir/archos_50saphir/index.html↩

  9. https://www.getdroidtips.com/stock-rom-archos-50-saphir/↩

  10. http://www.nomu.hk/s10-rom-download/↩

  11. https://arstechnica.com/information-technology/2017/03/preinstalled-malware-targets-android-users-of-two-companies/↩

]]> 2018-01-07T11:00:00+00:00 2018 Peter Molnar CC BY 4.0 https://petermolnar.net/puertito-de-los-molinos Puertito de los Molinos 2017-12-27T08:00:00+00:00 Puertito de los Molinos has huge, steep, beautiful walls of rock surrounding it; unfortunately it's more or less impossible to climb down on them. We planned to get here by sunset, but it was pitch black when we actually arrived, so instead before getting our flight home, we got up early and came back for sunrise. It was worth it: when sunrise comes with the high tide the waves are magnificent, even with a calm weather.

]]> 2017-12-27T08:00:00+00:00 2017 Peter Molnar CC BY-NC-ND 4.0 https://petermolnar.net/road-to-malpais-grande Road to Malpaís Grande 2017-12-26T08:00:00+00:00 The road to and from Pozo Negro is quiet, but that doesn't mean there is nothing to look at: it comes all the way along the black fields of lava which started at Malpaís Grande and ended up at the sea at Pozo Negro.

]]> 2017-12-26T08:00:00+00:00 2017 Peter Molnar CC BY-NC-ND 4.0 https://petermolnar.net/pozo-negro Pozo Negro 2017-12-25T08:00:00+00:00 Pozo Negro got about 2 sentences in the guide book for Fuerteventura, and most of those 2 lines was that it has 2 pescados, fish restaurants. Well, it did, in fact had nice fish, and if you're there by sunset, like we were, it also offers beautiful colours.

]]> 2017-12-25T08:00:00+00:00 2017 Peter Molnar CC BY-NC-ND 4.0