Would it be rude to ask a famous professor who doesn't know me personally for a recommendation letter if I have published in a high ranking journal?

The benefit of recommendation letters is that they give insight into abilities that are not reflected in the rest of your application. Your grades and your publication are already in your application. A professor who doesn't know anything else about you has nothing to write that will help you.

To directly answer your question: it comes across more naïve/inexperienced than rude, but either way I do not see how it will help your case.

P.S. This answer assumes that the professor is not already familiar with your work before your email. To be more specific, I'm imagining you sending unsolicited emails to a bunch of the biggest names in your field (selected purely because they are "famous"), who have not previously heard of you or your work (as far as you know), asking them to write a recommendation letter for you entirely on the basis of your having published a paper in a journal.

If the professor is already familiar with your work (e.g. has cited it, or has had an email correspondence with you about it), then I agree with this answer that his/her letter can potentially be helpful. Similarly, I think Ben Webster offers some useful advice about attempting this in a more nuanced - and potentially more effective - way.

While I don't really like adding answers to questions that have too many already, I also think it's a little hard to cut through the arguing any other way.

1. It's not rude to ask for a letter of recommendation. Ever. (I mean, if you do it rudely, then yes, but the bare fact of asking isn't). It might be unlikely to succeed, or unrealistic in some cases, but it's not rude.

2. You should try to ascertain whether this is a normal thing to do in your field. In my field (mathematics), it borders on inconceivable that an undergrad would carry out and publish truly impressive work without an established researcher in the field supervising them and able to write a detailed letter about it. Maybe there are some other fields where that can be the case, though I can't say I know what they would be (perhaps some of the other answerers are). Similarly, JeffE's advice to try to create a dossier of the level that might be appropriate for a faculty position sounds completely preposterous to me, but I suppose it must not be in computer science.

3. Whenever you ask someone for a letter of recommendation, think about what you are hoping the person will say, and whether they are placed well to say it. So, I would only consider asking for a person to write a letter based purely on having read my research if I knew they were familiar with it, and could place it in a context that is not obvious to the people reading the letter. It can be very valuable to have a letter saying essentially "this paper is actually really important. here is why." I've read such letters, I've written such letters. But the paper has to be really important (in the view of the author) for such a letter to work. I suspect a lot of the argument here is based around whether the answerer really thinks this is an impressive publication or not (which we are all guessing about).

So, if I were you, I would ask some of your professors whether they really think this potential letter writer will be impressed by the publication. If they think maybe yes, then you can email her/him, and say:

Dear Prof. X,

I'm a student at the University of Y and am applying to graduate schools in country Z (or maybe be more specific) in underwater basket weaving, with a focus on the use of hemp. Prof. W thought you might be interested in this publication of mine, since it relates to your work on macramae at the bottom of the Mariana Trench. I was wondering if you thought you could write a strong recommendation for me on the basis of this publication. Thank you very much for your time.

Sincerely,

Joe Schmoe

Asking someone if they think they are able to write a strong letter for you is good way of leaving them an easy out, and not getting a terrible letter since someone finds it easier to write a terrible letter than to say no (it happens). I would almost certainly reject such a request since I really wouldn't feel like I could write a strong letter. But if the publication really is that good, maybe someone will feel like they can.

Let me disagree completely with all the other answers here, that I think have missed a crucial point of your question. In short: I believe it is a reasonable plan to ask for the letter.

Here's why: The OP has published a paper in a high ranking venue. Now, if this venue is really a top place, and the OP thinks that the professor knows about the result, which is supposedly a great result, then the fact that the professor doesn't know personally the OP is a huge advantage. This is the best recommendation that one can get: someone that doesn't know him/her but have heard of his/her result, based solely on its scientific merit!

P.S. do not add your grades. Ask for a reference based solely on the result, and explain your contribution if you're a coauthor.

Adding on to @ff524's answer, most professors tend to only write recommendations if they know the candidate for some time or worked/researched with them for a certain minimum period of time. There are also very few institutions who may require LoR's from people whom you are known to for at least X amount of time. Even if not, an LoR which is just a repeat of the résumé may not so serve your purpose.

Such a letter wouldn't help you.

Since the professor doesn't know you, the best possible outcome would be for them to give very generic praise of you. Chances are the admissions committee will swiftly disregard this letter.

If you look into what makes a good recommendation letter, you will find that it's not so much the prestige of the professor, but how closely they worked with you, and whether they give specific examples of your good qualities. Occasionally you might get situations where a famous professor is so famous, that merely their good word is enough to get you in - but this seems to be the exception rather than the rule. It is also very awkward to write a letter for someone you don't know well.

I would think about finding a more suitable recommender. Granted, if you have not only published, but the publication has attracted correspondence from major researchers, this sounds like it would give you a huge advantage over other applicants - and in principle one could write "I don't know this person but I've read their paper, which is very good - here are all the things this person did right in this paper". It still sounds like a stretch to me. Perhaps talk about the paper a lot in your statement of purpose, and hopefully it comes up during the interview.

No

If I was part of the admissions board, and received a letter of recommendation saying essentially "I don't know them personally, but this paper they wrote is great", this is what I would be thinking:

You are applying to a place where you believe no-one is capable of evaluating your paper on its own merits, so you need to get someone else to look at it and explain its quality to them. In that case, why are you even applying? It's clear that there is nothing you can learn from them. After all, if they are not even competent enough to evaluate your previous paper, how are they going to be able to evaluate any future work?

I wouldn't dismiss you out of hand for it, but it would definitely count strongly against you - someone who thinks they already know more than the professor is not someone I would want to be teaching, and is almost certainly going to be causing problems throughout their course - potentially even legal trouble, if they start making accusations of discrimination when the professors give them anything less than full marks.

If you think your paper is very good, and should be considered, then include it with your application. Otherwise the best case you could hope for is being damned by faint praise; the worst is something like I describe above.

The point to a letter of recommendation is that it is a recommendation. If they don't know you, then what is their basis for recommending you? If they're straightforward and honest, they can't. Therefore, the request is basically asking someone to lie. Yes, that is rude.

Hence, I favor the first sentence of Ébe Isaac's answer.