Making video using Opencv library but not sure how to specify or convert it to h264 codec format

So I am using AWS lambda to build some videos using Opencv library. However when I store the video in S3 and try to display it within S3 without downloading it, it does not work so I believe that's because of the codec format, has to be h264, because when I use this website, https://www.ofoct.com/video-converter/convert-to-h-264-video.html, to convert the video and re-upload it to S3 again, the video works fine.

This is how I am making the video

fourcc = cv2.VideoWriter_fourcc(*'DIVX')   
out = cv2.VideoWriter(output_file, fourcc, 25.0, (width, height))

I tried to change the fourcc to mp4v and I got same issue. I tried to use 0x7634706d and 0x00000021 instead of fourcc and same thing.

I have been looking to find something or some way in Python that converts the video to h264 and that I can use in Lambda but could not find nothing so far. it's been 3 days of searching.