What is exactly meant by a “data set”?

I think that Wikipedia does a decent job at defining it:

Most commonly a data set corresponds to the contents of a single database table, or a single statistical data matrix, where every column of the table represents a particular variable, and each row corresponds to a given member of the data set in question. The data set lists values for each of the variables, such as height and weight of an object, for each member of the data set. Each value is known as a datum. The data set may comprise data for one or more members, corresponding to the number of rows.

The term data set may also be used more loosely, to refer to the data in a collection of closely related tables, corresponding to a particular experiment or event. An example of this type is the data sets collected by space agencies performing experiments with instruments aboard space probes.

In the open data discipline, dataset is the unit to measure the information released in a public open data repository. The European Open Data portal aggregates more than half a million datasets. In this field other definitions have been proposed but currently there is not an official one. Some other issues (real-time data sources, non-relational datasets, etc.) increases the difficulty to reach a consensus about it.

As you can see, the term is somewhat vague.

In my experience, "dataset" (or "data set") is an informal term that refers to a collection of data. Generally a dataset contains more than one variable and concerns a single topic; it's likely to concern a single sample.

A mistake I often see writers of Cross Validated questions make is using "dataset" as a synonym for "variable" or "vector".

I think you might need to define data point before you can define data set: why is one primitive and not needing definition, but not vice versa?

At least two definitions make sense to me:

1. One or more observations (cases, records, rows) for one or more variables (fields. columns).

2. Whatever is stored as data within a file readable by a program of choice.

Tabular layout is common but I don't think it's part of any definition; how the data are stored can be practically important, naturally.

P.S. The word "format" is so overloaded that to me it's best avoided unless specified unambiguously. I've seen it used for

1. General or specific text or binary file format

2. Data structure, e.g. tabular or other

3. Data storage or variable types, e.g. bit, integer, real, character

4. Display format controlling presentation, e.g. details on number of decimal places; decimal, hexadecimal or binary display.

There are already some good answers here and I don't think I can penetrate any deeper than Nick Cox or Franck Dernoncourt the issue of whether "dataset" refers to the conceptual collection of related data, or to the particular arrangement of those data e.g. into a table/matrix or a computer-readable file. Franck's extract mentions edge cases like continuously-collected data, or data spread across several tables, which are worth bearing in mind if you assumed there was going to be a simple definition. (Not all statistics software can handle it, but it is very easy to imagine a case where data is stored in a relational database with multiple tables. Is the entire database a single "dataset"?)

One thing I will add though is that datasets aren't generally sets, in the mathematical sense! Sensu stricto either a set contains an object or it doesn't, but can't contain more than one copy of that object. If I roll a die eight times and score 1, 4, 3, 5, 5, 4, 6, 4 then the set of scores rolled is just {1, 3, 4, 5, 6}. Note that the elements could be in any order, I've just written them ascending in value but the set {5, 4, 1, 6, 3} is mathematically equal to it, for instance. This isn't what we usually mean by a dataset though!

A multiset (or bag) allows entries to be repeated, e.g. {1, 4, 3, 5, 5, 4, 6, 4} though note this still doesn't include a sense of order, so is equal to {1, 3, 4, 4, 4, 5, 5, 6}. So the "set" in "dataset" really means "multiset". Moreover, if you want order to be preserved, you might instead use a vector: (1, 4, 3, 5, 5, 4, 6, 4) is not the same as (1, 3, 4, 4, 4, 5, 5, 6}. But this is only for recording one variable - for several, it may be more convenient to use a matrix to tabulate with order preserved. For more sophisticated situations such as measuring a property of a 3d grid of voxels over time, you might even move up to arranging the data in a tensor.

But note that conceptually a multiset may suffice in most simple situations, even if it's inconvenient for practical purposes. If I tossed a coin simultaneously with rolling the die, and wanted to record the two results together, then I could use a multiset like {(1, H), (3, T), (4, H), (4, H), (4, T), (5, H), (5, T), (6, T)} instead of a matrix. An ordinary set will not suffice, as it wouldn't count the multiplicity of the (4, H), for instance.