What's the maximum number of dice that could be rolled at any one time?

There is no limit on the number of dice which can be rolled at once. For example, you can stack any arbitrary number of explosive runes into a space, and if you fail a dispel check, they will all go off at once. That's 5d6 x however many times someone thought it would be funny to stack them.

As a practical matter, more than 20 dice at once is relatively rare.

Separately, it sounds like you should investigate a different approach to your dice-roller. Maximum number or type of dice shouldn't be an issue.

Not only is there no hard limit on the number of dice of any given type rolled, one may also use several different die types in a single roll. So, while 1000d6 gives the damage dealt by a small Kansas farmhouse when dropped on a Witch from a height of 10 feet, 9d6+2d10+(1d12+13)X3 would give the total result of a critical hit from a +2 Shocking Burst Greataxe of Axiomatic Power when wielded by a character with 32 Strength against a chaotic goblin. A die in D&D 3.5 can be a d2, d3, d4, d6, d8, d10, d12, d20, or d100, but supporting arbitrary die types and amounts is typically preferred when it comes to die rollers. In addition to die types, it is sometimes important or helpful to know what each part of a roll is coming from. For example, in the greataxe example above the '2d10' is electricity damage. If the goblin had some sort of eletrical resistance we would want to know how much of the total came from that 2d10 so as to avoid subtracting out more damage than can actually be prevented by a given defense. We can do this by rolling multiple times if source-tracking isn't supported by your die roller, but it is more convenient to have that done for us. In the greataxe example, a helpful roller might output "101 (59 physical, 11 electricity, 31 lawful)", having been prompted by user-given tags.

There are several existing rpg-focused die-rollers. You may want to take a look at AnyDice, a free online probability calculator intended for use with RPGs (it includes a die roller feature). The damage calculator in ToME4 (which is open source) does a really good job of implementing user-given tags on die rolls, but I'm not good enough to sort out what parts of this massive open-source repository that's contained in. The core 'letting you give you tags' part is here, though, and maybe if you know a lot of Lua that could be helpful.

As was mentioned in comments, you may get better advice on the programming parts of your project by asking related questions on Stack Overflow.

I'll provide a detailed answer about dice usage in D&D 3.5 specifically, matching the question's tag.

In D&D 3.5, rather large numbers of dice can be rolled at once. The core spell Disintegrate, for instance, deals "2d6 points of damage per caster level (to a maximum of 40d6)". This type of spell is generally considered a "save-or-die" scenario, where the exact total of damage is unlikely to matter; but it does exist, and could come up.

Now, for some specific trends related to dice:

The dice types used in D&D are d2, d3, d4, d6, d8, d10, d12, d20, and d100 (often written as "d%"). d1000 is also occasionally used in some supplements, but I don't recall any use in the core rules. I'll note that the d2, d3, and d100 are not physically present in most physical dice sets (in a non-digital setting, they're usually emulated using other dice), but are used throughout the rules nonetheless.

In terms of large quantities of a single kind of die (possibly mixed with others), I'd say that d6s are the most common, but d4, d10, and d8 are all seen as well. Spellcasters and the Rogue's Sneak Attack ability regularly use large numbers of d6s for a single damage type, scaling at a rate of 1d6 per 1 or 2 levels. The spell Magic Missile is a case where many players will end up rolling 5d4+5 at once, although strictly speaking that's a case of 5 separate 1d4+1 rolls.

You'll also encounter large quantities of mixed dice, including cases where it's important to distinguish different sets of the same type of die. This is more common with martial fighters, and again also some rogues - anyone who uses weapons, really, due to the various magical properties that add extra damage dice. For instance, a critical hit from a +1 Shocking Burst Thundering Dragonbane longbow, as wielded by a 14th level human rogue, would involve rolling dice for 3d8 piercing + 1d6 electric + 2d10 electric + 2d8 sonic + 2d6 + 7d6 damage (plus a constant amount added on top).

If you want to support epic-level characters (which is a dramatic uptick in power), then be aware that there are published spells dealing 305d6 damage on a failed save.

Other answers have touched on the programming aspect, and I agree that "many dice in assorted groups" is the preferred standard for a general-purpose dice roller. How much is "many"? Fairly large, it seems.