Allow modification of list during loop

In most cases, the user is modifying the array directly within the original loop. In my case, the array is being modified by a method being called inside of the loop, outside of the loop's control. I need a way for an outsider to be able to add a value or remove a value in response to an event.

I am currently working on an event system for things such as input. The event system is mostly working, however there are times where in response to an event, a listener will actually wish to stop listening for events (For example, if a user clicks the mouse button to leave one menu, the listener for that menu will remove itself from the input listeners).

The issue with this though is that when a listener removes itself in an event listener method, it modifies the array being looped through in order to call said event methods. This results in a ConcurrentModificationException. One solution I've come up with is cloning the list so that it is not the one being modified during the loop. However, I would assume this could be resource intensive and reduce performance by a lot.

Another functioning system I've come up with is having two lists (one for adding, and for removing) when looping through the listeners. If the add or remove methods are called during a loop, these lists will have those values added to them. At the end of the loop, the values are added and removed accordingly. The issue with this is that requires state management and can look quite messy. Is there a better way to get around this issue than the two things I just described?

/**
 * The base listener.
 */
public interface Listener {

    /**
     * Called when an event occurs.
     *
     * @param id
     *      the ID of the event.
     */
    public void event(int id);

}

/**
 * The test implementation of the base listener.
 */
public class TestListener implements Listener {

    @Override
    public void event(int id) {
        if(id == 0xDEADBEEF) {
            /*
             * This is where the error occurs.
             */
            TestSystem.removeListener(this);
        }
    }

}

/*
 * The revolutionary system that does foo and bar!
 */
class TestSystem {

    private static final ArrayList<Listener> LISTENERS = new ArrayList<Listener>();

    /**
     * Adds a listener.
     *
     * @param listener
     *      the listener to add.
     */
    private static void addListener(Listener listener) {
        LISTENERS.add(listener);
    }

    /**
     * Removes a listener.
     *
     * @param listener
     *      the listener to remove.
     */
    private static void removeListener(Listener listener) {
        LISTENERS.remove(listener);
    }

    /**
     * Calls an event.
     *
     * @param event
     *      the event to call.
     */
    private static void callEvent(Consumer<? super Listener> event) {
        for(Listener listener : LISTENERS) {
            event.accept(listener);
        }
    }

    public static void main(String[] args) {
        addListener(new TestListener()); // Add listener
        callEvent(listener -> listener.event(0xDEADBEEF)); // Call event
    }

}

If you need anymore information, please let me know in the comments. The stack trace being produced from this code is:

Exception in thread "main" java.util.ConcurrentModificationException
    at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
    at java.util.ArrayList$Itr.next(ArrayList.java:859)
    at org.ardenus.engine.list.ConcurrentLoop.callEvent(ConcurrentLoop.java:72)
    at org.ardenus.engine.list.ConcurrentLoop.main(ConcurrentLoop.java:81)