How to Have Shared Log Files Under Windows

How do you have shared log files under Windows?

It is possible to have multiple batch processes safely write to a single log file. I know nothing about Python, but I imagine the concepts in this answer could be integrated with Python.

Windows allows at most one process to have a specific file open for write access at any point in time. This can be used to implement a file based lock mechanism that guarantees events are serialized across multiple processes. See https://stackoverflow.com/a/9048097/1012053 and http://www.dostips.com/forum/viewtopic.php?p=12454 for some examples.

Since all you are trying to do is write to a log, you can use the log file itself as the lock. The log operation is encapsulated in a subroutine that tries to open the log file in append mode. If the open fails, the routine loops back and tries again. Once the open is successful the log is written and then closed, and the routine returns to the caller. The routine executes whatever command is passed to it, and anything written to stdout within the routine is redirected to the log.

Here is a test batch script that creates 5 child processes that each write to the log file 20 times. The writes are safely interleaved.

@echo off
setlocal
if "%~1" neq "" goto :test

:: Initialize
set log="myLog.log"
2>nul del %log%
2>nul del "test*.marker"
set procCount=5
set testCount=10

:: Launch %procCount% processes that write to the same log
for /l %%n in (1 1 %procCount%) do start "" /b "%~f0" %%n

:wait for child processes to finish
2>nul dir /b "test*.marker" | find /c "test" | >nul findstr /x "%procCount%" || goto :wait

:: Verify log results
for /l %%n in (1 1 %procCount%) do (
  <nul set /p "=Proc %%n log count = "
  find /c "Proc %%n: " <%log%
)

:: Cleanup
del "test*.marker"
exit /b

==============================================================================
:: code below is the process that writes to the log file

:test
set instance=%1
for /l %%n in (1 1 %testCount%) do (
  call :log echo Proc %instance% says hello!
  call :log dir "%~f0"
)
echo done >"test%1.marker"
exit

:log command args...
2>nul (
  >>%log% (
    echo ***********************************************************
    echo Proc %instance%: %date% %time%
    %*
    (call ) %= This odd syntax guarantees the inner block ends with success  =%
            %= We only want to loop back and try again if redirection failed =%
  )
) || goto :log
exit /b

Here is the output that demonstrates that all 20 writes were successful for each process

Proc 1 log count = 20
Proc 2 log count = 20
Proc 3 log count = 20
Proc 4 log count = 20
Proc 5 log count = 20

You can open the resulting "myLog.log" file to see how the writes have been safely interleaved. But the output is too large to post here.

It is easy to demonstrate that simultaneous writes from multiple processes can fail by modifying the :log routine so that it does not retry upon failure.

:log command args...
>>%log% (
  echo ***********************************************************
  echo Proc %instance%: %date% %time%
  %*
)
exit /b

Here are some sample results after "breaking" the :log routine

The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
Proc 1 log count = 12
Proc 2 log count = 16
Proc 3 log count = 13
Proc 4 log count = 18
Proc 5 log count = 14

Where to store an application log file on Windows

The Application Data directory would seem to be the perfect place, but it's an area that is nearly invisible. You need to give your users an easy way to get to it.

Have your installation script create a Log folder in the Application Data area for your program, and include a link to the folder in your Start menu.

Best place for log files in an in-house IT environment

You can use MSMQ. Write your logs to a MSMQ queue, and then have a service that picks these logs up and puts them in a database or out to a file in a central location if you want. It wouldn't be instantaneous, but you could tell it to run whenever you want to get new log entries. Plus it would be reliable since it uses MSMQ.

Best location for exception log files (Windows)

With the newer OS vista and win 2008 writing to the program files directory may need ACL or be virtualized to the user's folder. I would suggest setting up a directory under %AllUsersProfile% this way you have one common location for all user's log files and your applications does not need to run as admin to write them there.

How to create specific user log files in PowerShell?

I believe you need something like this, if not please provide more code.

$creator = (get-aduser $env:username | Select Name).Name
$date = get-date -UFormat "%d/%m/%Y"

I believe you have the $user in your script.

$log = $creator + "created on " + $date + " the user account " + $user.Name
Out-File $log "C:\temp\log.csv"

Best place to store configuration files and log files on Windows for my program?

If you're not using ConfigurationManager to manage your application and user settings, you should be. The configuration toolkit in the .NET Framework is remarkably well thought out, and the Visual Studio tools that interoperate with it are too.

The default behavior of ConfigurationManager puts both invariant (application) and modifiable (user) settings in the right places: the application settings go in the application folder, and the user settings go in System.Environment.SpecialFolder.LocalApplicationData. It works properly under all versions of Windows that support .NET.

As for log files, System.Environment.SpecialFolder.LocalApplicationData is generally the place that you want to put them, because it's guaranteed to be user-writeable.

There are certainly cases where you wouldn't - for instance, if you want to write files to a network share so that you easily can access them remotely. There's a pretty wide range of ways to implement that, but most of them start with creating an application setting that contains the path to the shared folder. All of them involve administration.

I have a couple of complaints about ConfigurationManager and the VS tools: there needs to be better high-level documentation than there is, and better documentation of the VS-generated Settings class. The mechanism by which the app.config file turns into the application configuration file in the target build directory is opaque (and the source of one of the most frequently asked questions of all: "what happened to my connection string?"). And if there's a way of creating settings that don't have default values, I haven't found it.

UAC and log files management

I'd strongly suggest not writing the log files to the same location as you install your application, but instead to one of the standard public locations, which you can access by environment variables.

See this link for more details on how to set this in Log4Net : How to specify common application data folder for log4net?

The two common locations to log to which avoid UAC restrictions are:
CommonApplicationData (https://msdn.microsoft.com/en-us/library/windows/desktop/aa367992(v=vs.85).aspx) which is a location where all users can write to, so you might want to use this if you want a common logging location regardless of who is logged on to Windows and running your application.

LocalAppData (https://msdn.microsoft.com/en-us/library/windows/desktop/aa369768(v=vs.85).aspx) which is a location specified to your currently logged on user. This would allow you to keep your log files from different Windows users separate from each other.

I'm not sure off the top of my head whether you'd have the same issue with writes to the SQL Server CE database. The pattern I've followed in the past to work with UAC is to install all static files under Program Files, then all data under one of the above 2 mentioned folders depending on whether the application data and logging was per-user or per-installation.