Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a program that needs to generate temporary files. It is written for cluster machines.

If I saved those files to a system-wide temporary directory (eg: /tmp), some users complained the program failed because they didn't have proper access to /tmp. But if I saved those files to the working directory, those users also complained they didn't want to see those mysterious files.

Which one is a better practice? Should I insist that saving to /tmp is the right approach and defend any failure as "working as intended" (ie. ask your admin for proper permission/access)?

share|improve this question
3  
check if the program has access and if not find another temp dir – ratchet freak yesterday
19  
If your admin screwed up the access rights, he should definitely fix it. What would you do if your admin forgot to add execute rights to your program? – Doc Brown yesterday
6  
You will not find /tmp on most windows systems, but there is a OS call that will tell you where to put temp files. – Ian yesterday
22  
If some people didn't have access to /tmp on a Unix-like system, it's misconfigured. The superuser should do something like chmod 1777 /tmp. – musiphil yesterday
9  
Beware that $TMPDIR might point to a different path than /tmp/, which you should use instead. See some of the answers ;) – marcelm 23 hours ago

Temporary files have to be stored into the operating system temporary directory for several reasons:

  • The operating system makes it very easy to create those files while ensuring that their names would be unique.

  • Most backup software knows what are the directories containing temporary files, and skips them. If you use the current directory, it could have an important effect on the size of incremental backups if backups are done frequently.

  • The temporary directory may be on a different disk, or in RAM, making the read-write access much, much faster.

  • Temporary files are often deleted during the reboot (if they are in a ramdisk, they are simply lost). This reduces the risk of infinite growth if your app is not always removing the temp files correctly (for instance after a crash).

    Cleaning temp files from the working directory could easily become messy if the files are stored together with application and user files. You can mitigate this problem by creating a separate directory within the current directory, but this could lead to another problem:

  • The path length could be too long on some platforms. For instance, on Windows, path limits for some APIs, frameworks and applications are terrible, which means that you can easily hit such limit if the current directory is already deep in the tree hierarchy and the names of your temporary files are too long.

  • On servers, monitoring the growth of the temporary directory is often done straight away. If you use a different directory, it may not be monitored, and monitoring the whole disk won't help to easily figure out that it's the temp files which take more and more place.

As for the access denied errors, make sure you let the operating system create a temporary file for you. The operating system may, for instance, know that for a given user, a directory other than /tmp or C:\Windows\temp should be used; thus, by accessing those directories directly, you may indeed encounter an access denied error.

If you get an access denied even when using the operating system call, well, it simply means that the machine was badly configured; this was already explained by Blrfl. It's up to the system administrator to configure the machine; you don't have to change your application.

Creating temporary files is straightforward in many languages. A few examples:

  • Bash:

    # The next line will create a temporary file and return its path.
    path="$(mktemp)"
    echo "Hello, World!" > "$path"
    
  • Python:

    import tempfile
    
    # Creates a file and returns a tuple containing both the handle and the path.
    handle, path = tempfile.mkstemp()
    with open(handle, "w") as f:
        f.write("Hello, World!");
    
  • C#:

    // Creates a file and returns the path.
    var path = Path.GetTempFileName();
    File.WriteAllText(path, "Hello, World!");
    
  • PHP:

    # Creates a file and returns the handle.
    $temp = tmpfile();
    fwrite($temp, "Hello, World!");
    fclose($temp);
    
  • Ruby:

    require "tempfile"
    
    # Creates a file and returns the file object.
    file = Tempfile.new ""
    file << "Hello, World!"
    file.close
    

Note that in some cases, such as in PHP and Ruby, the file is removed when the handle is closed. That's an additional benefit of using the libraries bundled with the language/framework.

share|improve this answer
2  
What do you mean by "make sure you let the operating system create a temporary file for you". So instead of e.g. fopen("/tmp/mytmpfile", "w"); I should make some system call to handle temporary files? – gurka yesterday
26  
@gurka: You should be calling tmpfile(3) to generate your temporary files, or at least calling mktemp(3) to create the file names. – TMN yesterday
3  
@TMN: They are just library functions that run in the user space, and they don't have any magic to bypass the permission error given by the operating system. – musiphil yesterday
20  
@musiphil Both tmpfile and mktemp uses external variables to determine the path for temporary files. These may have been set up to point to another directory than /tmp/, perhaps a per-user directory. Trying to create a filename manually in /tmp/ may fail, while tmpfile and mktemp would return valid paths. – pipe yesterday
2  
@musiphil: I never said they'd fix the permission problem, I was responding to his question about using system calls to create the files. – TMN yesterday

Should I insist saving to /tmp is the right approach and defend for any failure as "working as intended" (ie. ask your admin for proper permission access)?

There are standards for this, and the best thing you can do is conform to them.

POSIX, which is followed by pretty much every non-mainframe OS of any significance that you're likely to run into, has provisions for creating uniquely-named temporary files in a directory using default values that can be reconfigured by the environment:

  • The C stdio.h header may optionally include a P_tmpdir macro that names the system's temporary directory.
  • TMPDIR is the canonical environment variable for changing the location of temporary files. Prior to POSIX, there were other variables used, so I tend to go with the first of that or TMP, TEMPDIR and TEMP that has a value, punting and using the system default if none of those exist.
  • The mkstemp() and tempfile() functions will generate unique temporary files.

If your users are being denied the ability to create temporary files, the system is either misconfigured or the administrators aren't making clear what their policy is on such things. In those cases, you'd be on very firm ground in saying that your program conforms to a well-established portability standard and that its behavior can be changed using the environment variables the standard specifies.

share|improve this answer
    
P_tmpdir is not a part of stdio.h as defined by the C language specification. It might be defined by POSIX or SVID. – musiphil yesterday
1  
@musiphil: As implied by the (now-clarified) answer, it's part of POSIX. (Technically, it's an X/Open System Extension that POSIX incorporated. See pubs.opengroup.org/onlinepubs/009695399/basedefs/stdio.h.html.) – Blrfl yesterday
    
Fully agree with all the above. A good example is Linux systems with pam_tmpdir - this sets TMPDIR and TMP to be different for each user, for robustness and privacy. It's also useful to be able to set TMPDIR for a single command - if you have your usual temporary directory in a RAM filesystem for speed, you may need to do this for commands that generate huge temporary files (such as a giant sort, for example). Don't ignore the standards/conventions that your users expect! – Toby Speight 23 hours ago
    
Definitely check the environment for the location of temporary files and never hard-code /tmp. Because a shared tmp has security issues, one mitigation I have often seen is to create per-user /tmp directories with no read-write permission for anyone else. It removes possible race conditions and symlink attacks. – Zan Lynx 15 hours ago

The temp-file-directory is highly operating system/environment dependant. For example a web-servers-temp dir is seperate from the os-temp-dir for security reasons.

Under ms-windows every user has its own temp-dir.

you should use the createTempFile() for this if such a function is available.

share|improve this answer
1  
Just be mindful of hidden OS limitations in Windows. We discovered the hard way that the maximum number of files in a folder was limited to 65,565. Sure, that's a lot of files, and sure, you should never conceivably have that many laying around. But are you sure that every app cleans up after itself in a timely and well-behaved manner? – Mike Hofer 12 hours ago
    
Ah, I've seen your comment too late. I just wrote the same above. BTW the limit is primarily due to the mechanics of the GetTimeFileName() function, not NTFS. That folder limit you mentioned applies only to FAT32. – JensG 12 hours ago

The previous answers, although correct, aren't valid for most large scale computer clusters.

Computer clusters not always follow the standard conventions for machines, usually for good reasons, and there is no point in discussing it with the sysadmins.

Your current directory is referring to the central file system, which is accessed through the network. This is not only slow, but also puts loads on the system for the rest of the users, so you shouldn't use it unless you aren't writing much and you can recover from it if the job crashes.

The computing nodes have their own hard drive, that is the fastest file system available, and what you should be using. The cluster documentation should tell you what it is, typically /scratch, /tmp/[jobid], or some non standard enviroment variable ($SNIC_TMP in one of the ones I use).

So, what I recommend is making it user-configurable. The defaults can be the first one you have write access to:

  • $TMPDIR
  • tmpfile
  • /tmp
  • .

But expect a low success rate with this approach, and make sure to emit a big fat warning.

share|improve this answer
    
+1 for putting in config – jk. 20 hours ago
    
Which ones exactly are the previous answers? – user61852 13 hours ago
1  
So what you're saying here is that because some clusters that don't take the trivial step of adhering to a well-established standard for telling programs where to write their temporary files, that's one additional cluster-specific customization required per program. Pretty weak tea if you ask me. – Blrfl 11 hours ago

For many applications, you should consider putting temporary files in $XDG_RUNTIME_DIR or $XDG_CACHE_HOME (the other XDG dirs are for nontemporary files). For instructions on calculating them if they are not explicitly passed in the environment, see the XDG basedir spec or find a library that already implements that part.

Note, however, that $XDG_RUNTIME_DIR is a new addition and there is no standard fallback for older systems due to security concerns.

If neither of those is suitable, then /tmp is the correct place. You should never assume the current directory is writable.

share|improve this answer

This is more like an alternative, but you might unlink() the file immidiately after fopen() . It depends of usage pattern of cource.

Unlinking the files, if it can be done, helps for several ways:

  • file is not seen - user not see it.
  • file is not seen from other processes - there is not chance other process to modify the file by mistake.
  • easy cleanup if program crash.

Files must be created in /tmp. If user have not rights to create file there, this means system is missconfigured.

Files can not be created in users home directory. Lots of users, such "nobody", " www-data" and many others, does not have rights to write in their home directories, or they are even chroot()-ed. Note that even in chroot environment /tmp still exists.

share|improve this answer
    
While this might be a good idea in general, it doesn't help the users who are lacking write permissions on the directory the file is to be created in. – 5gon12eder 14 hours ago
3  
It also doesn't answer the question, which is where to put temporary files. – Blrfl 11 hours ago
    
I believe my answer is somehow important. I did edit, probably is more clear this way. – Nick 1 hour ago

protected by gnat 20 hours ago

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.