Gesture Detector doesn't want to size itself to a child container (unless you give it a color)

So this problem is driving me mad so any help would be greatly appreciated :)

I currently have a custom Gesture Detector which I want to size to the width and height parameters I've given to the child container. Though it will only do this if the child container is given a colour (transparent in this case). I'll post some code and screenshots down below to explain a bit further.

Things that I've tried include using SizedBox instead of a container but that didn't work. I also tried changing the values of behaviour but that didn't seem to change anything either.

//The params width and height are given values up here
...
return Container(
        color: Color(0XFF0000FF), //Blue container which shows the actual size I want the GD to be.
        child: ColorFiltered( //Red shows how big the GD actually is.
          colorFilter: ColorFilter.mode(Color(0XFFFF0000), BlendMode.hue),
            child: RawGestureDetector(
            key: gestureKey,
            behavior: HitTestBehavior.translucent,
            gestures: <Type, GestureRecognizerFactory>{
              _SingleDeviceGestureDetector: GestureRecognizerFactoryWithHandlers<
                  _SingleDeviceGestureDetector>(
                () => _SingleDeviceGestureDetector(
                  //When the pointer touches down on the screen
                  onHorizontalDragDown: (details) { ... }
                  onHorizontalDragUpdate: (details) { ... }
                  onHorizontalDragUp: (details) { ...}
                ),
                (_SingleDeviceGestureDetector instance) {},
              ),
            },
            child: Container(  //The child container with a transparent colour.          
              //color: Color(0X00000000),
              height: height,
              width: width,
              child: ...
            ),
          ),
        ),
      );
    });
  }

1 Shows the widget when color is uncommented

2 Shows the widget when color is commented

When you uncomment the color field in the child container (1) the gesture detector will be the same size as the first parent container (which is what I want), and when you comment out the color field (2) the gesture detector is smaller.

Even though I have the fix, it feels very 'hacky' and it would still be good to figure out why it does this.

Cheers