What is torrent encryption and does it make my traffic anonymous?

Think of it like an underground fight club. Encrypting the traffic means nobody on the outside can see you enter or leave, but once you're inside, everybody there knows who you are and can monitor your participation.

This feature is really only useful if you have an ISP that blocks torrent traffic. Encrypting it means it doesn't appear to be torrent traffic, it's just an encrypted stream, but once you get past the ISP and connect to the swarm everybody else participating knows exactly who you are and what you're doing.

The "encryption" hides the content of the torrent data traffic from a casual observer and makes it harder to determine that the traffic is in fact torrent data traffic. It was designed to make it harder for ISPs to snoop on torrent traffic (and either block it, throttle it or send nastygrams). It can also be useful for evading the effects of buggy firmware in network devices*.

The cryptography used is relatively weak. The DH key exchange is only 768 bit with a fixed prime which is almost certainly crackable by a well-funded attacker. The actual encryption is rc4 which is known to have weaknesses though I don't know how relevant they are to this particular application. MITM attacks are possible if the attacker knows the "info hash" of the torrent in question.

Also it only protects data connections. It DOES NOT hide the fact you are present in the swarm from someone scanning the tracker or dht. It DOES NOT stop the bad guys from connecting to your client and downloading a copy of the file from you to demonstrate that you are illegally offering it for distribution.

* I have encountered cases where the same peice of a file was repeatly failling hash checks, turning on encryption fixed it. I suspect a buggy and/or overzealous NAT implementation was responsible.

what does this feature encrypt?

It encrypts the entire communication stream with other BT peers.

Name of the file, it's contents, size?

All of the above. Note that it is completely and utterly useless to encrypt these things as they are already PUBLIC.

Could it prevent DMCA notices?

It can prevent a residential ISP from casually snooping the traffic. In some countries this doesn't matter, as residential ISPs have no business snooping your traffic. In other countries, rightsholders have programs to cooperate with ISPs to detect piracy and serve out nastygrams.

If not, what does it actually do?

It is mainly security theatre. It will still be obvious to your ISP that you are using BT. It will still be easy for rightsholders to find out which IPs are sharing their content. So, the encryption doesn't solve any real problem. Someone added it to their BT client to draw dumb users who don't understand the security aspects at all. Then everyone else had to add it to their BT client too, even though it is mostly useless.

The DMCA notice was sent through Microsoft. Azure is not an anonymous service and makes no effort to conceal the fact that you're the owner of your IP. To protect your identity you need a VPN provicer which specializes in this, e.g., "PIA", "Hide my Ass", etc.

Bittorrent is not designed to anonymize. While it can be used through a proxy, it isn't 100% to protect you there either. See https://hal.inria.fr/inria-00471556 for information about anonymity leaks in bittorrent clients through the Tor network. Even if these holes are addressed, what motivation do the bittorrent client authors have to get security and privacy right?

Even if your client doesn't betray you and you find a VPN who's willing to hide your tracks, you have to question their motive. If they're served with legal papers, will they track you? why would they not operate in the legal framework of the country?

On encryption, Bitcomet has a good comment on this:

Please note that the encryption option is meant to hinder traffic shaping applications on the ISP side by obfuscating BitTorrent traffic between peers. However this doesn't anonymize you on the Internet as far as the other peers in the swarm are concerned, since your IP address will still be visible to each peer of that torrent swarm. If you aim for total anonymity you should look towards a VPN proxy solution which will masquerade your IP behind the VPN IP address, thus offering you a real degree of anonymity.

http://wiki.bitcomet.com/protocol_encryption

So, VPN is not an option.

This is where you're wrong.

VPN doesn't hide your identity. VPN merely shifts your identity to your VPN provider identity. VPN doesn't do anything for you directly, it merely allows you to hide behind someone's else skirt. Now, depending on what provider you've chosen, when authorities come asking who you are, it can either fight them for you or fight you for them.

Azure is not an identity-hiding service, so using it to conceal yourself cannot work. It doesn't mean that "VPN is not an option" it only means that "Azure VPN is not an option"

Question is: what does this feature encrypt?

It conceals the nature of the traffic from your service providers (your ISP, your VPN provider, and your VPN providers's ISP) to make traffic shaping (read: blocking torrents) more difficult.

It doesn't encrypt the sources unless they too are using encryption. One sided encrypition is like hiding in a safe from the world outside that knows your inside the safe.