What is the purely mathematical definition of encryption strength?

One of the definition is stated in the other answer (indistinguishably). Another way to assert the strength of an algorithm is to compute the complexity of an attack.

Let us assume $$E : \mathcal{K} \times \mathcal{M} \to \mathcal{C}$$ be your encryption function.
It takes as an input a key and a message (or plaintext) and return a ciphertext.

Given a list of pair of (plaintext,ciphertexts) encrypted with the same $key$, the strength of $E$ is given by the number of computations required to find the right $key$.

In other words, if your binary key as a length of $n$ bits. The size of the $\mathcal{K}$ is $2^n$. If you have a strong function, finding the right $k$ will require you on average $2^{n-1}$ computations.

Your function can be weakened by different kinds of attacks. e.g. the initial strength of DES is $\mathcal{O}(2^{55})$ as the size of the key is 56 bits. Matsui showed that with linear cryptanalysis, you can find the key with $2^{43}$ pairs of (plaintext,ciphertexts)[1], thus dicreasing the complexity to $\mathcal{O}(2^{43})$.

There isn't just one, because there are many different scenarios where you'd use such a function, where the attacker has lesser or greater powers, or variably stringent success goals—different attack models. For example:

• Does the attacker know any plaintext/ciphertext pairs encrypted with the same key? (Known plaintext attack)
• Is the attacker able to trick the defender into encrypting plaintexts of the attacker's choice, and observe the corresponding ciphertexts? (Chosen plaintext attack)
• In the latter case, is the attacker able to observe the ciphertexts in real time, and use this to make smarter choices of plaintext to submit? (Adaptive chosen plaintext attack)
• Is the attacker's goal to decrypt a message, or something weaker like telling true encrypted messages apart from random bits? (The latter is what's called a "distinguishing attack," and strengths are often expressed in terms of that.)

So it's common for algorithms to have different security levels in different scenarios. If you look for example at the "Known attacks" section in Wikipedia's entry on AES, you'll see a variety of figures, all qualified by the relevant attack model.

Another detail is that strengths are often expressed not as one number but as a function of the resources that the attacker expends in some model (e.g., in a chosen plaintext attack, the number queries, i.e. the number of plaintexts that they submit to the defender). Such functions generally give the probability that a randomized attacker will succeed if they expend that many resources.

And yet another detail is that often for some algorithms like block cipher modes of operation, which are parametrized by one or more primitives, the strength is often given as a function of the strengths of the primitives. If you're math-inclined or even just math-curious, two papers I found very useful to understand this are:

• Bellare, Kilian and Rogaway's "The Security of the Cipher Block Chaining Message Authentication Code"
• Bellare, Desai, Jokipii and Rogaway's "A Concrete Security Treatment of Symmetric Encryption"

These two papers are perhaps overwhelmingly detailed, but even just skimming the major points might be instructive.

One common definition is IND-CCA2 which, roughly, states that even given many pairs of plaintext and ciphertext an attacker cannot distinguish another ciphertext from randomness.

Note that some sort of randomness is commonly introduced to prevent simple attacks like frequency analysis which would work on bijections.

"Given any invertible function which are the quantified parameters which measure how effective the function is to encrypt a message?"

This answer addresses the units that are used to measure things related encryption. There are two quantifiable aspects of an algorithm: Security and Efficiency.

Many things influence security, as was thoroughly covered in the other answers. Context is important:

• things like key size and block size are measured in bits
• time-to-break can be measured in a model which replaces the unit of time to an invocation of the algorithm in question.

Efficiency is generally measured in Cycles per byte for symmetric algorithms. Asymmetric algorithms are often measured by the quantity of messages that can be enciphered/deciphered or signed/verified per second.

eBATS has even finer grained measurements of efficiency:

The eBATS (ECRYPT Benchmarking of Asymmetric Systems) project, part of eBACS, measures public-key systems according to the following criteria:

• Time to generate a key pair (a private key and a corresponding public key).
• Length of the private key.
• Length of the public key.
• Time to generate a secret shared with another user, given a private key and the other user's public key.
• Length of the shared secret.
• Time to encrypt a message using a public key.
• Length of the encrypted message.
• Time to decrypt an encrypted message using a private key.
• Time to sign a message using a private key.
• Length of the signed message.
• Time to verify a signed message using a public key.

I think encryption strength isn't a mathematical notion but a computational science. Though completely possible probably to state and characterize it in the language of sets and logic, I dontthink this approach is used ever. Instead a notion of deterministic algorithms or problem difficulty hierarchy is used, though often isomorphic to arithmetic hierarchy.