Storing nested structures and avoiding memory leaks

I'm making a small game in C++, it's working but I'm having trouble clearing allocated memory when removing objects.

The game consists of 'shapes' which in turn are made up several 'blocks' (think tetris). Here's a minimal version of my setup for storing game objects:

#include <vector>
#include <set>
#include <algorithm>
using namespace std;

struct Block{
    int x;
};

struct Shape{
    set<Block*> blocks;
};

vector<Block*> all_blocks; //Pointers to all blocks in the game
vector<Shape*> all_shapes; //        -       shapes     -
    
int main() {

    //At start of level, create blocks and shapes objects on heap and
    //store pointers to them in the two vectors above
    Block* b1 = new Block{2};
    Block* b2 = new Block{3};
    Block* b3 = new Block{5};

    Shape* s1 = new Shape{ set<Block*>{b1,b2} };
    Shape* s2 = new Shape{ set<Block*>{b3} };

    all_blocks.push_back(b1);
    all_blocks.push_back(b2);
    all_blocks.push_back(b3);

    all_shapes.push_back(s1);
    all_shapes.push_back(s2);

    //...working game...//

    //Clearing memory
    for(Block* b : all_blocks){
        delete b;
    }
    for(Shape* s : all_shapes){
        delete s;
    }
    all_blocks.clear();
    all_shapes.clear();
}

Everything seems to be working so far. However now I want to be able to delete individual blocks during runtime. I came up with the following complicated function

void delete_block(Block* b){
    for (Shape* s: all_shapes) { //Delete the block from every shape that contains it
        (s->blocks).erase(b);
    }
    //Now some shape(s) may contain no blocks, remove all empty shapes,
    auto it = begin(all_shapes);
    while (it != end(all_shapes)){
        if ((**it).blocks.size() == 0) {
            it = all_shapes.erase(it);  //Remove shape from all_shapes vector
            delete (*it);               //Delete the shape object from heap
        }else{
            it++;
        }
    }
    //Finally delete block from vector ´all_blocks´ and delete it from heap
    all_blocks.erase(remove(all_blocks.begin(), all_blocks.end(), b), all_blocks.end());
    delete b;
    }

This works some of the time, but after several delete_block(all_blocks[0]) it always crashed with a memory error, so something is wrong with this function, or possibly my whole setup, but I can't seem to find the problem. My console says it returned 0xC0000374. I will probably redo it with smart pointers later, but for now I would like to understand what's wrong first.

How can I remove a single block/shape from the game without memory leaks and errors? What did I do wrong?