Konrad Rudolph  👨‍🔬 💻 🧬

uniq (1)

Don't you like "uniq"?

I do but `uniq` needs sorted input.

You are right it seems I never had the unsorted requirement so far. Good job then ;-)

One other way the awk method is useful is you don't need to compare the whole line. For instance, in a web server log you may only care to see when it's an new unseen remote client, so use $1 instead of $0.

Efficient in what sense? Doesn't this store 'seen' in memory? `sort` uses local disk space as necessary. I use `awk '! seen[$0]++' <( cat /dev/urandom | strings | head -n 1000000000 ) > /dev/null` and I quickly saturate memory. What am I missing?

Less memory efficient, but better asymptotic runtime if not memory bound. But my use case was indeed not due to preformance but requiring to preserve the original order.

Oh yeah I am _totally_ using this script from now on. Preserving original order is much more comprehensible in many contexts. I was just bristling at shoving every unique line in a hash (unbounded memory growth).

How did it save your day? Did you specifically need unsorted output? Was there some resource consumption trade-off that worked in your favor? Is your sort implementation wildly inefficient? Inquiring minds want to know.

Yes, the original order of the lines matters in my case.

Coolio! If I did $1 instead of $0 would it return unique values from column 1? Im Still an AWK white belt :)

useless use of awk award; man uniq

Not at all: `uniq` requires pre-sorted files. I need the original order. The awk solution is also more efficient than `sort | uniq` or `sort -u`.