C# BitMap is unable to locate pixels

1

I've been working on a small application that basically runs off bitmap screenshot images. The bitmap is then detecting a certain pixel from a Resources.GrabPixel .bmp I loaded in. This method doesn't seem reliable in certain ways when many similar pixels are involved.

If I was to use this in a BlueStacks or Andy Emulator the return would be likely false. I've had instances were it would detect a game icon but for some odd reason a few days later it would randomly stop detecting that same icon.

So is there a better method and approach I should be learning more towards? I'm not an expert in C# and any sample examples I could try and build off of would be great. I will list my code below if anyone thinks it can be improved by any means please share.

Goal: Detect certain pixel, while discarding unmatched pixels. A more accurate pixel reader.

The code listed below contains the bitmap and mouse_events.

public partial class frmMain : Form
{
    private const UInt32 MOUSEEVENTF_LEFTDOWN = 0x0002;
    private const UInt32 MOUSEEVENTF_LEFTUP = 0x0004;

    [DllImport("user32.dll", EntryPoint = "SetCursorPos")]
    [return: MarshalAs(UnmanagedType.Bool)]
    public static extern bool SetCursorPos([In] int X, [In] int Y);
    [DllImport("user32.dll", CharSet = CharSet.Auto, CallingConvention = CallingConvention.StdCall)]
    static extern void mouse_event(uint dwFlags, uint dx, uint dy, uint dwData,
    int dwExtraInfo);

    public enum MouseEventFlags : uint
    {
        LEFTDOWN = 0x00000002,
        LEFTUP = 0x00000004,
        MIDDLEDOWN = 0x00000020,
        MIDDLEUP = 0x00000040,
        MOVE = 0x00000001,
        ABSOLUTE = 0x00008000,
        RIGHTDOWN = 0x00000008,
        RIGHTUP = 0x00000010,
        WHEEL = 0x00000800,
        XDOWN = 0x00000080,
        XUP = 0x00000100
    }
    public enum MouseEvents
    {
        MOUSEEVENTF_LEFTDOWN = 0x02,
        MOUSEEVENTF_LEFTUP = 0x04,
        MOUSEEVENTF_RIGHTDOWN = 0x08,
        MOUSEEVENTF_RIGHTUP = 0x10,
        MOUSEEVENTF_WHEEL = 0x0800,
    }

    public frmMain()
    {
        InitializeComponent();
    }

    // Loads the exe program
    private void btLogin_Click(object sender, EventArgs e)
    {
        // takes a snapshot of the screen
        Bitmap bmpScreenshot = Screenshot();

        // makes the background of the form a screenshot of the screen
        ////this.BackgroundImage = bmpScreenshot;

        // find the Icon  and check if it exists
        Point location;
        bool success = FindBitmap(Properties.Resources.Programloader, bmpScreenshot, out location);

        // check if it found the bitmap
        if (success == false)
        {
            MessageBox.Show("Couldn't locate Andy.exe!");
            return;
        }

        // move the mouse to Icon 
        Cursor.Position = location;

        // click
        MouseClick();
        MouseClick();
        if (success == true)
        {
            System.Threading.Thread.Sleep(5000);
        }
    }
    /// <summary>
    /// Simulates a mouse click
    /// </summary>
    private void MouseClick()
    {
        mouse_event((uint)MouseEventFlags.LEFTDOWN, 0, 0, 0, 0);
        Thread.Sleep((new Random()).Next(20, 30));
        mouse_event((uint)MouseEventFlags.LEFTUP, 0, 0, 0, 0);
    }

    /// <summary>
    /// Takes a snapshot of the screen
    /// </summary>
    ///  <returns> A snapshot of the screen</return>
    private Bitmap Screenshot()
    {
        // this is where we will store a snapshot of the screen
        Bitmap bmpScreenshot = new Bitmap(Screen.PrimaryScreen.Bounds.Width, Screen.PrimaryScreen.Bounds.Height);

        // creates a graphics object so we can draw the screen in the bitmap (bmpScreenshot)
        Graphics g = Graphics.FromImage(bmpScreenshot);

        // copy from screen into the bitmap we created
        g.CopyFromScreen(0, 0, 0, 0, Screen.PrimaryScreen.Bounds.Size);

        // return the screenshot
        return bmpScreenshot;
    }

    /// <summary>
    /// Finds the location of a bit map within another bitmap and  returns if it was successfully found
    /// </summary>
    /// <param name="bmpNeedle"> The image we want to find</param>
    /// <param name="bmpHaystack"> Where we want to search for the image</param>
    /// <param name="location"> Where we found the image</param>
    /// <returns> If the bmpNeedle was found successfully </returns>
    private bool FindBitmap(Bitmap bmpNeedle, Bitmap bmpHaystack, out Point location)
    {
        for (int outerX = 0; outerX < bmpHaystack.Width - bmpNeedle.Width; outerX++)
        {
            for (int outerY = 0; outerY < bmpHaystack.Height - bmpNeedle.Height; outerY++)
            {
                for (int innerX = 0; innerX < bmpNeedle.Width; innerX++)
                {
                    for (int innerY = 0; innerY < bmpNeedle.Height; innerY++)
                    {
                        Color cNeedle = bmpNeedle.GetPixel(innerX, innerY);
                        Color cHaystack = bmpHaystack.GetPixel(innerX + outerX, innerY + outerY);

                        if (cNeedle.R != cHaystack.R || cNeedle.G != cHaystack.G || cNeedle.B != cHaystack.B)
                        {
                            goto notFound;
                        }
                    }
                }
                location = new Point(outerX, outerY);
                return true;
            notFound:
                continue;
            }
        }
        location = Point.Empty;
        return false;
    }

    private void frmMain_Load(object sender, EventArgs e)
    {
        timer1.Start();
        timer1.Interval = 1;
    }

    private void timer1_Tick(object sender, EventArgs e)
    {
        label1.Text = Cursor.Position.X.ToString() + ":" + Cursor.Position.Y.ToString();
    }

    private void button1_Click_3(object sender, EventArgs e)
    {
        // takes a snapshot of the screen
        Bitmap bmpScreenshot = Screenshot();

        // makes the background of the form a screenshot of the screen
        //this.BackgroundImage = bmpScreenshot;

        // find the login button and check if it exists
        Point location;
        bool success = FindBitmap(Properties.Resources.Map, bmpScreenshot, out location);

        // check if it found the bitmap
        if (success == false)
        {
            MessageBox.Show("Test Failed Badly.");
            return;
        }

        // move the mouse to login button
        Cursor.Position = location;

        // click
        MouseClick();
        MouseClick();
    }

    // Uses a Mouse_event to scroll (Right)
    private void button2_Click_2(object sender, EventArgs e)
    {
        uint X = (775);//set x position 
        uint Y = (340);//set y position 

        // Thread.Sleep(10000);
        mouse_event((uint)MouseEvents.MOUSEEVENTF_LEFTDOWN, X, Y, 0, 0);
        // Thread.Sleep(2000);
        SetCursorPos((int)X + 10, (int)Y + 10);
        // Thread.Sleep(2000);
        mouse_event((uint)MouseEvents.MOUSEEVENTF_LEFTUP, X + 15, Y + 15, 0, 0);

        uint A = (775);//set x position 
        uint B = (340);//set y position 
        mouse_event((uint)MouseEvents.MOUSEEVENTF_LEFTDOWN, A, B, 50, 50);

        SetCursorPos((int)A + 50, (int)B + 50);

        mouse_event((uint)MouseEvents.MOUSEEVENTF_LEFTDOWN, A, B, 50, 50);

        Cursor.Position = new Point(715, 218);
        Thread.Sleep(2000);
        Cursor.Position = new Point(400, 218);
        Thread.Sleep(1000);
        mouse_event((uint)MouseEvents.MOUSEEVENTF_LEFTUP, X + 15, Y + 15, 0, 0);
    }

    // Scans the sceen for a particular build(Farm,Sawmill,ect.)
    private void button6_Click_1(object sender, EventArgs e)
    {
        // takes a snapshot of the screen
        Bitmap bmpScreenshot = Screenshot();

        // makes the background of the form a screenshot of the screen
        //this.BackgroundImage = bmpScreenshot;

        // find the login button and check if it exists
        Point location;
        bool success = FindBitmap(Properties.Resources.HallofWar, bmpScreenshot, out location);

        // check if it found the bitmap
        if (success == false)
        {
            // MessageBox.Show("Test Failed Badly.");
            return;
        }

        // move the mouse to login button
        Cursor.Position = location;

        // click
        MouseClick();
    }
}
c#
bitmap
asked on Stack Overflow Jun 30, 2015 by Jordan Miller • edited Jul 1, 2015 by vesan

1 Answer

1

If I understand it right, you are trying to find out if a large bitmap contains a smaller bitmap. First off, let me say that Bitmap.GetPixel is very slow, so if you want to go this route, you'll be much better off if you copy the bitmap into an array first. You do that by calling Bitmap.LockBits and then you use Marshal.Copy to copy the raw data into an array. This MSDN page has an example.

If you don't want to reinvent the wheel, I would suggest using a library for this, for example OpenCV (and its .NET port , Emgu CV). You can look at this question, which is about something very similar to what you are doing.

Finally, a word of warning: if I undestand it right, you are trying to find a program icon in a screenshot. However, depending on your DPI settings, resolution, transparency of the icon and its size, the icon you have may not be 100% pixel-by-pixel identical to the icon rendered on the screen.

answered on Stack Overflow Jun 30, 2015 by vesan • edited May 23, 2017 by Community

User contributions licensed under CC BY-SA 3.0