Please have a look at the individual commit messages for the details.

@sschuberth it sounds like the introduction of html5lib did modify the behavior of BeautifulSoup which uses it when/if available hence some completely unrelated markup cleanup tests are now failing ... I will look into this.

I will look into this.

Thanks!

@sschuberth how do you want me to proceed?
I can push your branch with extra commits to a scancode branch and submit a PR to you?

Sounds good, except that you don't have to file a PR with my fork. I'll take your changes directly from your branch.

@sschuberth I pushed https://github.com/nexB/scancode-toolkit/tree/338-spdx-tv and this has afew extra commits and all tests pass on Linux for now.

The main fix was to upgrade beautifulsoup4 and regenerate/review some expectations in tests.

The builds are running at https://ci.appveyor.com/project/nexB/scancode-toolkit/build/314 and at https://travis-ci.org/nexB/scancode-toolkit/builds/189249129

@sschuberth The addition of SPDX support to scancode is wonderful, and something we have been wanting for quite a while !!! Thanks, I intend to try it out as soon as possible.

Thanks @pombredanne, I've incorporated most of your changes, some of them squashed to make a more readable history.

@DennisClark Thanks for the praise! I expect there to be rough edges, but it's a start and should be easy to improve now.

Looking good and merging in.