Non ascii characters in Balloon WinApi popup

I try to show a balloon popup with a Unicode text, but I always see a ??? characters inside. When the title (szInfoTitle) doesn't contain non-ascii characters everything works just fine.

And here is the fully working F# example:

  type NotifyIconData =
    struct
      val mutable cbSize: System.Int32  // DWORD
      val mutable hWnd: System.IntPtr // HWND
      val mutable uID: System.Int32 // UINT
      val mutable uFlags: System.Int32  // UINT
      val mutable uCallbackMessage: System.Int32  // UINT
      val mutable hIcon: System.IntPtr  // HICON
      [<MarshalAs(UnmanagedType.ByValTStr, SizeConst=128)>]
      val mutable szTip: System.String  // char[128]
      val mutable dwState: System.Int32  // DWORD
      val mutable dwStateMask: System.Int32  // DWORD
      [<MarshalAs(UnmanagedType.ByValTStr, SizeConst=256)>]
      val mutable szInfo: System.String // char[256]
      val mutable uTimeoutOrVersion: System.Int32  // UINT
      [<MarshalAs(UnmanagedType.ByValTStr, SizeConst=64)>]
      val mutable szInfoTitle: System.String // char[64]
      val mutable dwInfoFlags: System.Int32  // DWORD
      val mutable guidItem: Guid
      val mutable hBalloonIcon: IntPtr //HIcon
    end

  [<DllImport("shell32.dll", SetLastError = true)>]
  extern bool Shell_NotifyIcon(uint dwMessage, NotifyIconData& pnid)

  [<DllImport("kernel32.dll")>]
  extern IntPtr GetConsoleWindow();

  let createNotify hwnd =
    let mutable data = NotifyIconData()
    data.cbSize <- Marshal.SizeOf(typedefof<NotifyIconData>)
    data.hWnd <- hwnd
    data.uID <- 0
    data.uFlags <- 0x00000001 ||| 0x00000002 ||| 0x00000004 ||| 0x00000008 ||| 0x00000020 ||| 0x00000080
    data.szTip <- "Tooltip"
    data.dwState <- 0
    data.dwStateMask <- 0x00000001 ||| 0x00000002
    data.guidItem <- Guid.NewGuid ()
    data.hIcon <- PInvoke.User32.LoadIcon(IntPtr.Zero, IntPtr(32512))
    let result = Shell_NotifyIcon(uint NotifyCommand.Add, &(data))

    data.uTimeoutOrVersion <- 4

    let versionResult = Shell_NotifyIcon(uint NotifyCommand.SetVersion, &(data))
    data

  let createBalloon message (d: NotifyIconData) =
    let mutable data = d
    
    data.uFlags <-  0x00000010 ||| 0x00000020
    data.dwInfoFlags <- 4
    data.hBalloonIcon <- PInvoke.User32.LoadIcon(IntPtr.Zero, IntPtr(32512))
    data.szInfo <- message
    data.szInfoTitle <- "Balloon два dva dede ąęśćóŌ"
    data.uTimeoutOrVersion <- 30000
    let result = Shell_NotifyIcon(uint NotifyCommand.Modify, &(data))

    result

  let showBallon hwnd message =
    GetConsoleWindow ()
    |> createNotify 
    |> createBalloon message

I try to use Shell.NotifyIconW but it also doesn't work.

In addition to Shell_NotifyIconW, you also need to check whether your NOTIFYICONDATA is a wide byte version to support Unicode. Check the cbSize you get:

• x86 ansi : cbSize = 508
• x86 unicode: cbSize = 956
• x86 ansi : cbSize = 528
• x86 ansi : cbSize = 976

In my test, using your sample without any charset setting, the default cbSize = 528 (ansi version), you need to explicitly specify the charset of both your struct and function as unicode:

[<StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)>]
type NotifyIconData =
  struct
    val mutable cbSize: System.Int32  // DWORD
    val mutable hWnd: System.IntPtr // HWND
    val mutable uID: System.Int32 // UINT
    val mutable uFlags: System.Int32  // UINT
    val mutable uCallbackMessage: System.Int32  // UINT
    val mutable hIcon: System.IntPtr  // HICON
    [<MarshalAs(UnmanagedType.ByValTStr, SizeConst=128)>]
    val mutable szTip: System.String  // char[128]
    val mutable dwState: System.Int32  // DWORD
    val mutable dwStateMask: System.Int32  // DWORD
    [<MarshalAs(UnmanagedType.ByValTStr, SizeConst=256)>]
    val mutable szInfo: System.String // char[256]
    val mutable uTimeoutOrVersion: System.Int32  // UINT
    [<MarshalAs(UnmanagedType.ByValTStr, SizeConst=64)>]
    val mutable szInfoTitle: System.String // char[64]
    val mutable dwInfoFlags: System.Int32  // DWORD
    val mutable guidItem: Guid
    val mutable hBalloonIcon: IntPtr //HIcon
  end

[<DllImport("shell32.dll", CharSet = CharSet.Unicode)>]
extern bool Shell_NotifyIcon(UInt32 dwMessage, NotifyIconData& pnid)

In addition, there is no indication in the Shell_NotifyIcon document that the error code can be used.