Ruby: FormatMessageW trouble

What's a correct way to call FormatMessageW in Ruby?

   require 'win32api'

   FormatMessage = Win32API.new 'kernel32', 'FormatMessageW', 'IPIIPII', 'I'
   msg = '\0' * 255
   FormatMessage.call 0x00001000 | 0x00000100, nil, 6, 1024, msg, 0, 0

FormatMessage returns not null result but msg contains not readable message. What's wrong?

I believe this is the code you are looking for:

require 'win32api'

FORMAT_MESSAGE_FROM_SYSTEM = 0x1000
FormatMessage = Win32API.new 'kernel32', 'FormatMessageW', 'IPIIPIP', 'I'
msgw = ("\x00" * 256).force_encoding("UTF-16LE")
count = FormatMessage.call FORMAT_MESSAGE_FROM_SYSTEM, nil, 6, 1033, msgw, msgw.size, nil
msgw = msgw[0, count]
msg = msgw.encode("UTF-8")
puts msg

When I run this with Ruby, the output is "The handle is invalid.", which is the correct Windows error message for error code 6.

There were some problems with your original code.

1. The last argument to FormatMessageW is a pointer, so you should use P instead of I, especially if you want it to work on 64-bit Windows.
2. In Ruby '\0' is actually a two-byte ASCII string, not a single-byte null character. You can confirm this by running p '\0'.bytes.to_a. It looks like you tried to allocate 255 bytes, but you actually allocated 510 bytes. You should allocate an even number of bytes because wide characters in Windows take 2 bytes.
3. As @theB pointed out, your first argument to FormatMessageW was wrong, since you specified that FormatMessageW should allocate its own buffer.
4. You specified language code 1024. I can't find a definition for that. Maybe you meant 1033, which is "English - United States". Specifying 1024 doesn't seem to actually cause problems though.
5. You should use force_encoding to set the encoding of your string to UTF-16LE, because that is the encoding used for wide strings in Windows (or if it's not exactly the same, at least it is compatible most of the time).
6. The 6th argument to FormatMessageW should be the number of characters in your buffer (which is the number of bytes divided by 2, by the way). Your code just passed 0 for that argument.
7. Strings in Ruby can contain any arbitrary bytes, including null characters, but it's not necessarily a good idea to let that happen because things like String#size will return surprising results. FormatMessageW returns the number of characters in the formatted message, so we can use that to trim off the null characters at the end. (Conveniently, FormatMessageW returns 0 if there is an error, so our trimming would result in an empty string.)
8. You should use String#encode to convert your string from UTF-16LE to UTF-8 because UTF-8 strings are much easier to operate on and print in Ruby.

If you don't care about internationalization and unicode, you could have just used FormatMessageA instead. Here is some code that will work for that:

require 'win32api'

FORMAT_MESSAGE_FROM_SYSTEM = 0x1000
FormatMessage = Win32API.new 'kernel32', 'FormatMessageA', 'IPIIPIP', 'I'
msg = ("\x00" * 256).force_encoding("ASCII-8BIT")
count = FormatMessage.call FORMAT_MESSAGE_FROM_SYSTEM, nil, 6, 1033, msg, msg.size, nil
msg = msg[0, count]
puts msg

P.S. DWORD is an unsigned integer type. I am not sure what the right letter for that is in Ruby's Win32API class; it might be that I represents a signed integer, and should be replaced by something else.