How many times may a character attempt a skill roll for the same test?

The DMG, at "Multiple Ability Checks," is your friend.

Read p.237. Some highlights include:

• No number of attempts make an impossible task possible
• Failed attempts may make possible tasks impossible (like the hairpin breaking off in the lock!)
• If repeated attempts are all it takes, dispatch with ability checks and figure out the time success should take

As general advice, you should only have the players roll if there are consequences for failure. The question you should ask is what happens if the rogue fails? Perhaps the tool breaks, or they jam the lock. Or perhaps they only have 5 minutes to catch a shift change, and a failure means it takes too long. In general, a player should never roll twice for the same thing because a failure should have changed the circumstances enough that just trying again until they succeed isn't possible, or will have consumed some finite resource such as HP or hit dice (such as when falling on a climb). If the player has an hour to pick the lock, and you don't think they could conceivably fail to pick it, then let them succeed and move on to the interesting bits with real consequences. For general advice on how to handle skill checks, check out this article. Section 3b is directly relevant to your situation, but the whole thing is a good read and will help manage the narrative flow of the challenges in your game.

Ability and Skill Checks tell you something about your world.

My policy on this issue is somewhat unusual. I (usually) use skill checks and ability checks to determine something about my world. For example, let us imagine that a character wants to climb a wall. I probably have a rough idea in my head of how difficult this particular wall is to climb, so I set a DC for it. If the player succeeds, their character climbs the wall, if they fail, the character starts, then falls (or some similar outcome). So far, pretty standard.

However, that check has now told me something interesting - that wall is too difficult for that character to climb in those circumstances. This means that, no matter how many more times they attempt it, they will always fail - it is just too difficult for them. Now, if circumstances change (a friend gives them a leg up, they acquire pitons, etc.), they are welcome to make another check. If they fail even that one, that is usually a good sign that the particular challenge is just too difficult no matter what the circumstances. Clearly, exceptional circumstances (like acquiring a ladder), could make the challenge possible once again - common sense must be excercised.

What I find to be the benefit of this system, is that it gets around the knowledge check problem. Knowledge checks are annoying because there isn't a clear challenge that the character is facing. They exist to work out whether a character knows something or not. Now, in (gasp) real life, although we might suddenly remember something later on, trying to remember something doesn't usually work. We either know things, or we don't. We can't just 'try again' to know it. Thus, if a player fails a knowledge check in my game, that means that their character does not know that piece of information. No matter what they roll from now on, their character just does not know.

Of course, alternative circumstances could arise to change this. They might, for example, find themselves in a library, in which case an investigation check might discover the required information. If they fail that, perhaps it means that the library doesn't have the required book?

Taking it Further

If you want to take this further, you could do something like this:

Player: I want to pick that lock.

DM: Roll a dexterity check.

Rolls 2

DM: You inspect the lock closely. It's a well-made dwarven tumbler design, and you realise immediately that such a lock is quite beyond your ability.

As you can see, the result of failure is not that the character tries and fails, it is that the character realises that they cannot succeed. This is particularly useful for potentially dangerous tasks. Characters should know roughly what their ability is. Looking up at a cliff, they might realise that this particular cliff is just too difficult, and attempting it is courting injury.

One thing I have done with some success is to adjust the DC of a repeated check based on the results of previous attempts -- basically, a progressive skill check, where your rogue figures out one thing on try 1, another thing on try 2, and finally gets it on try 3 so to speak, learning from previous failed checks in the case of Open Lock instead of simply trying the same thing over and over until it finally works. (Of course, you can grant advantage on a subsequent check or checks in 5e, or adjust the modifier to their roll, for that matter.)

Example: breaking down a DC 20 door. First check rolls an 18 against the DC 20 -- door is damaged but holds. Second check rolls a 5 on a DC 17 -- no change, or a penalty/negative change if you wish for an extreme failure (such as increasing the DC as depicted here, putting a malus on the roll, or attracting unwanted attention). Third check rolls a 16 against DC 18 -- almost there! Fourth check rolls a 13 against a DC 11 -- the door opens with a CRASH.

If he's just going to roll over and over again until he succeeds, why are you having him make a check at all? Checks are tools, not arbitrary stumbling blocks. You need to use them to do something. The only reason you should ever be forced into calling for a check, is to impartially determine the results of an uncertain objective. As such, one has to be clear on just what the objective is here: is he trying to open a lock or is he trying to escape? If he's trying to open a lock, then the result is certain, in the absence of any outside interference, he will puzzle at it until it is solved. If he is trying to escape however, you're now talking about a much larger, more complicated objective, covering more time, more space, and more activities than just this one lock. In that case, the outcome is uncertain, either he will escape or he will not. Once the check has been made, even if he remains trapped, there is no reason to repeat it, because the issue has already been resolved. That's why it's called a resolution mechanic.

As a general rule, I hate "I take 20". D&D is just like life, in that on any given day, your success at the same task may provide vastly different results.

If I'm helping my son look for a toy car in the garage, we could most definitely take our time and search until we find it. However, one day I might have more patience, or have gotten more sleep so my recollection is sharper, or my eyes might not be as tired, etc.

There are many different circumstances that may seem innocuous, but more closely represent life.

I believe with your thief in a cage scenario, the more times he tried and failed, the better he would get because he's learning the lock and gaining modifiers. Just as in life, practice makes perfect.

In this scenario, once he finally freed himself, I would also give him a fair permanent modifier to his open lock skill for free, because just as in life, we learn and get better from our experiences.