CR on game result screen failing to get text information - how can I pre process the images to be successful?

0

When I first ran into problems I read this article: Tesseract Doc on Improving Quality. I tried various transformations on my image and did manage to get some significant improvement. However it is still a long way from reasonable.

Here is what a game result screen looks like straight out of the game: Straight out of the game Here is what Tess4J gave me trying to OCR the unprocessed image:

AFTER ACTION REPORT
Ax Amarah
umermmax Su‘rends‘
5km: Irun
UNCONVENTIONAL MINOR VICTORY
us Army Uncnnvermuna‘
AocsPTAaLE . 6mm x FAILED
sscunsn V Targets x FAILED
FAILED x Parameters . ACCEPTABLE
6650 mes 9104
55 Mm 0K 0 Mm 0K
21 Mm mm 77 Mm mm
39 Mm Warned 53 Mm Warned
u Mm Mwssmg 2 Mm Mwssmg
u Talks Lust 0 Talks Lust
3 Armured Vmwc‘es Lust 0 Armured Vauc‘es Lust
a 0m: Vauc‘es Lust a 0m: Vauc‘es Lust
0 Amman Lust 0 Amman Lust
Flewew Map Em

Which is pretty bad. Here is what I was expecting:

AFTER ACTION REPORT
Al Amarah
Unconventional Surrender
Skill: Iron
UNCONVENTIONAL MINOR VICTORY
U.S. Army Unconventional
ACCEPTABLE Ground FAILED
SECURED Targets FAILED
FAILED Parameters ACCEPTABLE
6650 Points 9104
56 Men OK 0 Men OK
21 Men Killed 77 Men Killed
39 Men Wounded 53 Men Wounded
0 Men Missing 2 Men Missing
0 Tanks Lost 0 Tanks Lost
3 Armored Vehicles Lost 0 Armored Vehicles Lost
0 Other Vehicles Lost 0 Vehicles Lost
0 Aircraft Lost 0 Aircraft Lost
Point mouse at evaluations for details
Review Map End

I did some image processing and experimening - you can see some of that in the code below. The best results I found were from converting to grey scale, inverting, increasing contrast and sharpening: enter image description here

Which got me:

AFTER ACTION REPORT
Ax Amarah
Unmrwenmnna‘ Surrender
5km: [run
UNCONVENTIONAL MINOR VICTORY
us, Army Unmnvenmona‘
ACCEPTABLE . Ground £3 FAILED
SECURED M Targzcs 23 FAILED
FAILED 53 Parameters . ACCEPTABLE
5550 F'mncs 9104
55 Men UK 0 Men OK
21 Men mm 77 Men mm
39 Men Wounded 52 Men Wuundad
0 Men Mwssmg 2 Men Mwssmg
u Tanks ms: 0 Tanks ms:
3 Armured Vehwc‘es ms: 0 Armored Vamcxes ms:
0 Omar Vamcxes ms: 0 Omar Vamcxes ms:
0 Amman ms: 0 Amman ms:
mm meuxe u Hummus m deullx
‘ Flewew Map E ‘ End E

I also tried switching the engine mode to TessOcrEngineMode.OEM_TESSERACT_CUBE_COMBINED but that didn't really help much:

AFTER ACTION REPORT
Al Amarah
UttaottgettCiottal Surrender
Skill: [run
UNCONVENTIONAL MINOR VICTORY
us, Army UttuoptgeptCiomtl
ACCEPTABLE . Ground £3 FAILED
SECURED M Targzcs 23 FAILED
FAILED 53 Parameters . ACCEPTABLE
5550 F'mncs 9104
55 Men UK 0 Men OK
21 Men mm 77 Men mm
39 Men Wounded 53 Men Wuundad
0 Men Missing 2 Men Missing
u Tanks ms: 0 Tanks ms:
3 Armured Vehirtles ms: 0 Armored Vehirtles ms:
0 Omar Vehirtles ms: 0 Omar Vehirtles ms:
0 Amman ms: 0 Amman ms:
mm meuxe u Hummus m details
l Review Map E l End E

What transformations or changes to the configuration of Tess4J should I be making to make a sucessful OCR run of the game end screen?

Here is my code (note this is just a test class but I implemented a bunch of basic image processing methods to preform B&W conversion, negative, dilation, erosion, zoom, sharpen etc.:

package com.lesliesoftware.tesstest;

import java.awt.Color;
import java.awt.Graphics2D;
import java.awt.image.BufferedImage;
import java.awt.image.BufferedImageOp;
import java.awt.image.ConvolveOp;
import java.awt.image.Kernel;
import java.awt.image.RescaleOp;
import java.io.File;
import java.io.IOException;

import javax.imageio.ImageIO;

import com.lesliesoftware.lcommon.util.StringUtil;
import com.lesliesoftware.lcommon.util.file.FileHelper;

import net.sourceforge.tess4j.Tesseract;
import net.sourceforge.tess4j.TesseractException;
import net.sourceforge.tess4j.ITessAPI.TessOcrEngineMode;

public class CMEndScreenProcessImage {

    public static void main (String[] args) {
        File endScreenImageFile = new File ("P:\\TessTest\\test\\CombatMissionEndScreen2019.07.03.png");

        Tesseract tesseract = new Tesseract ();
        try {
            //  Process the image
            //  Read the raw image
            BufferedImage imageData = ImageIO.read (endScreenImageFile);
            
            boolean preProcessImage = true;
            if (preProcessImage)  {
                File intermediateFile = endScreenImageFile;
                
                imageData = convertImageToGreyScale (imageData);
                intermediateFile = writeNewImageFile (intermediateFile, "_GreyScale", imageData);
                
                imageData = convertImageToNegative (imageData);
                intermediateFile = writeNewImageFile (intermediateFile, "_Negative", imageData);
                
                imageData = increaseContrast (imageData, 1.2f, 0.8f);
                intermediateFile = writeNewImageFile (intermediateFile, "_Contrast", imageData);
            
//            imageData = zoomImageBy (imageData, 2.0f, 2.0f);
//            intermediateFile = writeNewImageFile (intermediateFile, "_Zoom", imageData);
            
//            imageData = dilateGrayscaleImage (imageData);
//            intermediateFile = writeNewImageFile (intermediateFile, "_Dilate", imageData);
            
                imageData = sharpenImage (imageData);
                intermediateFile = writeNewImageFile (intermediateFile, "_Sharpen", imageData);
            
//            imageData = erodeGrayscaleImage (imageData);
//            intermediateFile = writeNewImageFile (intermediateFile, "_Erode", imageData);
//            
//            imageData = erodeGrayscaleImage (imageData);
//            intermediateFile = writeNewImageFile (intermediateFile, "_Erode", imageData);
            
                endScreenImageFile = writeNewImageFile (endScreenImageFile, "_Processed", imageData);
                System.out.println ("Final image can be found at: " + endScreenImageFile.getAbsolutePath ());
            }

            //  Setup the tesseract object
            tesseract.setDatapath ("P:/Tess4J/tessdata");
            boolean useCubeCombined = false;
            if (useCubeCombined)  {
                tesseract.setOcrEngineMode (TessOcrEngineMode.OEM_TESSERACT_CUBE_COMBINED);
                tesseract.setLanguage ("eng");
            }

            //  Run the character recognition
            String text = tesseract.doOCR (imageData);

            //  Output the results
            System.out.print (text);
        } catch (IOException exception) {
            System.out.println ("File IO Error!:");
            exception.printStackTrace();
        } catch (TesseractException exception) {
            System.out.println ("OCR Error!:");
            exception.printStackTrace();
        }
    }
    
    
    private static BufferedImage sharpenImage (BufferedImage imageData) {
        // A 3x3 kernel that sharpens an image
        Kernel kernel = new Kernel (3, 3, new float[] {
                -1, -1, -1, 
                -1, 9, -1, 
                -1, -1, -1
        });
 
        BufferedImageOp op = new ConvolveOp(kernel);
 
        imageData = op.filter(imageData, null);
        
        return imageData;
    }


    /**
     * This method will perform erosion operation on the grayscale image img.
     * 
     * Code taken from https://github.com/yusufshakeel/Java-Image-Processing-Project/blob/master/DYimageFX-project/src/dyimagefx/morph/Erosion.java
     * Except the erode code is really a dilation and the dilation is really an erode so method contents have been swapped here
     * 
     * @param img The image on which erosion operation is performed
     */
    public static BufferedImage erodeGrayscaleImage(BufferedImage imageData){
        /**
         * Dimension of the image img.
         */
        int width = imageData.getWidth();
        int height = imageData.getHeight();
        
        //buff
        int buff[];
        
        //output of dilation
        int output[] = new int[width*height];
        
        //perform dilation
        for(int y = 0; y < height; y++){
            for(int x = 0; x < width; x++){
                buff = new int[9];
                int i = 0;
                for(int ty = y - 1; ty <= y + 1; ty++){
                   for(int tx = x - 1; tx <= x + 1; tx++){
                       if(ty >= 0 && ty < height && tx >= 0 && tx < width){
                           //pixel under the mask
                           int rgb = imageData.getRGB(tx, ty);
                           buff[i] = (rgb >> 16) & 0x000000FF;;
//                           buff[i] = imageData.getRed(tx, ty);
                           i++;
                       }
                   }
                }
                
                //sort buff
                java.util.Arrays.sort(buff);
                
                //save highest value
                output[x+y*width] = buff[8];
            }
        }
        
        /**
         * Save the dilation value in image img.
         */
        for(int y = 0; y < height; y++){
            for(int x = 0; x < width; x++){
                int v = output[x+y*width];
                Color colour = new Color (v, v, v);
                imageData.setRGB (x, y, colour.getRGB ());
//                imageData.setPixel(x, y, 255, v, v, v);
            }
        }
        
        return imageData;
    }

    
    /**
     * This method will perform dilation operation on the grayscale image img.
     * 
     * Code taken from https://github.com/yusufshakeel/Java-Image-Processing-Project/blob/master/DYimageFX-project/src/dyimagefx/morph/Dilation.java
     * Except the erode code is really a dilation and the dilation is really an erode so method contents have been swapped here
     * 
     * @param img The image on which dilation operation is performed
     */
    public static BufferedImage dilateGrayscaleImage(BufferedImage imageData){
        /**
         * Dimension of the image img.
         */
        int width = imageData.getWidth();
        int height = imageData.getHeight();
        
        //buff
        int buff[];
        
        //output of erosion
        int output[] = new int[width*height];
        
        //perform erosion
        for(int y = 0; y < height; y++){
            for(int x = 0; x < width; x++){
                buff = new int[9];
                int i = 0;
                for(int ty = y - 1; ty <= y + 1; ty++){
                   for(int tx = x - 1; tx <= x + 1; tx++){
                       /**
                        * 3x3 mask [kernel or structuring element]
                        * [1, 1, 1
                        *  1, 1, 1
                        *  1, 1, 1]
                        */
                       if(ty >= 0 && ty < height && tx >= 0 && tx < width){
                           //pixel under the mask
                           int rgb = imageData.getRGB(tx, ty);
                           buff[i] = (rgb >> 16) & 0x000000FF;;
//                           buff[i] = imageData.getRed(tx, ty);
                           i++;
                       }
                   }
                }
                
                //sort buff
                java.util.Arrays.sort(buff);
                
                //save lowest value
                output[x+y*width] = buff[9-i];
            }
        }
        
        /**
         * Save the erosion value in image img.
         */
        for(int y = 0; y < height; y++){
            for(int x = 0; x < width; x++){
                int v = output[x+y*width];
                Color colour = new Color (v, v, v);
                imageData.setRGB (x, y, colour.getRGB ());
//                imageData.setPixel(x, y, 255, v, v, v);
            }
        }
        
        return imageData;
    }

    
    private static BufferedImage increaseContrast (BufferedImage imageData, float brighten, float offset) throws IOException {
        //  Create the RescaleOP object
        RescaleOp rescale = new RescaleOp (brighten, offset, null);

        //  Perform the scaling operation - increase brightness
        BufferedImage contrastImageData = rescale.filter (imageData, null);
        
        return contrastImageData;
    }


    private static BufferedImage zoomImageBy (BufferedImage imageData, float xScale, float yScale) throws IOException {
        //  Make an empty image buffer to store image later
        int zoomWidth = (int)(imageData.getWidth () * xScale);
        int zoomHight = (int)(imageData.getHeight () * yScale);
        BufferedImage zoomImageData = new BufferedImage (zoomWidth, zoomHight, imageData.getType ());

        //  Creating a 2D platform on the buffer image for drawing the new image
        Graphics2D graphic = zoomImageData.createGraphics ();

        //  Draw new image starting from 0 0 to the zoomed image size
        graphic.drawImage (imageData, 0, 0, zoomWidth, zoomHight, null);
        graphic.dispose ();
        
        return zoomImageData;
    }


    public static File writeNewImageFile (File inputImageFile, String additionToName, BufferedImage zoomImageData) throws IOException {
        //  Append additionToName to the original file name
        String rootFileNameStr = FileHelper.getFileName (inputImageFile);
        String rootFileNameExt = FileHelper.getFileExtension (inputImageFile);
        File outputImageFile = new File (inputImageFile.getParentFile (), rootFileNameStr + additionToName + StringUtil.DOT + rootFileNameExt);
        
        //  Write the modified image
        ImageIO.write(zoomImageData, rootFileNameExt, outputImageFile);
        return outputImageFile;
    }


    private static BufferedImage convertImageToNegative (BufferedImage imageData) throws IOException {
        //  Loop for each pixel to convert to negative using the algorithm from https://www.geeksforgeeks.org/image-processing-java-set-4-colored-image-negative-image-conversion/?ref=lbp 
        int width = imageData.getWidth(); 
        int height = imageData.getHeight(); 
        for (int y = 0; y < height; y++) 
        { 
            for (int x = 0; x < width; x++) 
            { 
                int p = imageData.getRGB(x,y); 
                int a = (p>>24)&0xff; 
                int r = (p>>16)&0xff; 
                int g = (p>>8)&0xff; 
                int b = p&0xff; 
  
                //subtract RGB from 255 
                r = 255 - r; 
                g = 255 - g; 
                b = 255 - b; 
  
                //set new RGB value 
                p = (a<<24) | (r<<16) | (g<<8) | b; 
                imageData.setRGB(x, y, p); 
            } 
        } 
        
        return imageData;
    }
    

    private static BufferedImage convertImageToGreyScale (BufferedImage imageData) throws IOException {
        //  Loop for each pixel to convert to greyscale - using algorithm from https://www.geeksforgeeks.org/image-processing-in-java-set-3-colored-image-to-greyscale-image-conversion/?ref=lbp  
        int width = imageData.getWidth(); 
        int height = imageData.getHeight(); 
        for (int y = 0; y < height; y++)  { 
            for (int x = 0; x < width; x++)   { 
                //  Here (x,y) denotes the coordinate of image for modifying the pixel value. 
                int p = imageData.getRGB(x,y); 
  
                int a = (p>>24)&0xff; 
                int r = (p>>16)&0xff; 
                int g = (p>>8)&0xff; 
                int b = p&0xff; 
  
                //  calculate average 
                int avg = (r+g+b)/3; 
  
                //  replace RGB value with avg 
                p = (a<<24) | (avg<<16) | (avg<<8) | avg; 
  
                imageData.setRGB(x, y, p); 
            } 
        } 
        
        return imageData;
    }

}


java
ocr
tesseract
tess4j
asked on Stack Overflow Oct 5, 2020 by Ian Leslie

0 Answers

Nobody has answered this question yet.


User contributions licensed under CC BY-SA 3.0