Large Training Set For NER


I have a project that involves taking property descriptions and labeling key data elements. I decided to use spaCy in an effort to train my own NER pipe since these descriptions are not written like conventional sentences. However, when I go to train it gets to about 20% and then crashes and i am unable to find an explanation.

How it's set up

Below is a sample of my JSON. The full JSON is 2.6 MB and contains >1000 descriptions ranging from 40 - ~500 tokens. The file contains ~54000 tokens in total. (Be advised the below data has been changed to not reflect the actual property)

    "id": 1, "paragraphs": 
    "raw": "Lots 1 and 2 of Block 1, in the City of Santa Clarita, County of Los Angeles, State of California, as per Map recorded in Book 1, Page 1 of Miscellaneous Maps, in the Office of the County Recorder of said County "
        , "sentences": 
                        {"id": 1, "orth": "Lots", "ner": "B-LOT"}
                        , {"id": 2, "orth": "1", "ner": "I-LOT"}
                        , {"id": 3, "orth": "and", "ner": "I-LOT"}
                        , {"id": 4, "orth": "2", "ner": "L-LOT"}
                        , {"id": 5, "orth": "of", "ner": "O"}
                        , {"id": 6, "orth": "Block", "ner": "B-BLOCK"}
                        , {"id": 7, "orth": "1,", "ner": "L-BLOCK"}
                        , {"id": 8, "orth": "in", "ner": "O"}
                        , {"id": 9, "orth": "the", "ner": "O"}
                        , {"id": 10, "orth": "City", "ner": "O"}
                        , {"id": 11, "orth": "of", "ner": "O"}
                        , {"id": 12, "orth": "Santa", "ner": "O"}
                        , {"id": 13, "orth": "Clarita,", "ner": "O"}
                        , {"id": 14, "orth": "County", "ner": "O"}
                        , {"id": 15, "orth": "of", "ner": "O"}
                        , {"id": 16, "orth": "Los", "ner": "O"}
                        , {"id": 17, "orth": "Angeles,", "ner": "O"}
                        , {"id": 18, "orth": "State", "ner": "O"}
                        , {"id": 19, "orth": "of", "ner": "O"}
                        , {"id": 20, "orth": "California,", "ner": "O"}
                        , {"id": 21, "orth": "as", "ner": "O"}
                        , {"id": 22, "orth": "per", "ner": "O"}
                        , {"id": 23, "orth": "Map", "ner": "O"}
                        , {"id": 24, "orth": "recorded", "ner": "O"}
                        , {"id": 25, "orth": "in", "ner": "O"}
                        , {"id": 26, "orth": "Book", "ner": "B-BOOK"}
                        , {"id": 27, "orth": "1,", "ner": "L-BOOK"}
                        , {"id": 28, "orth": "Page", "ner": "B-PAGE"}
                        , {"id": 29, "orth": "1", "ner": "L-PAGE"}
                        , {"id": 30, "orth": "of", "ner": "O"}
                        , {"id": 31, "orth": "Miscellaneous", "ner": "B-MAPTYPE"}
                        , {"id": 32, "orth": "Maps,", "ner": "L-MAPTYPE"}
                        , {"id": 33, "orth": "in", "ner": "O"}
                        , {"id": 34, "orth": "the", "ner": "O"}
                        , {"id": 35, "orth": "Office", "ner": "O"}
                        , {"id": 36, "orth": "of", "ner": "O"}
                        , {"id": 37, "orth": "the", "ner": "O"}
                        , {"id": 38, "orth": "County", "ner": "O"}
                        , {"id": 39, "orth": "Recorder", "ner": "O"}
                        , {"id": 40, "orth": "of", "ner": "O"}
                        , {"id": 41, "orth": "said", "ner": "O"}
                        , {"id": 42, "orth": "County", "ner": "O"}

I took the file that comes with spaCy in the cli folder and created my own version for this process. I left the core functionality of the file in tact and just added a few things such as some new labels for my data set and a custom tokenizer that works with white space instead of the conventional tokenizer. The function is below:

def NERTrain(lang
          , output_dir
          , train_data
          , dev_data
          , n_iter=30
          , n_sents=0
          , parser_multitasks=''
          , entity_multitasks=''
          , use_gpu=-1
          , vectors=None
          , gold_preproc=False
          , version="0.0.0"
          , meta_path=None
          , verbose=False
          , newLabels = None):
    Train a model. Expects data in spaCy's JSON format.
    n_sents = n_sents or None
    output_path = util.ensure_path(output_dir)
    train_path = util.ensure_path(train_data)
    dev_path = util.ensure_path(dev_data)
    meta_path = util.ensure_path(meta_path)
    if not output_path.exists():
    if not train_path.exists():
        prints(train_path, title=Messages.M050, exits=1)
    if dev_path and not dev_path.exists():
        prints(dev_path, title=Messages.M051, exits=1)
    if meta_path is not None and not meta_path.exists():
        prints(meta_path, title=Messages.M020, exits=1)
    meta = util.read_json(meta_path) if meta_path else {}
    if not isinstance(meta, dict):
               title=Messages.M052, exits=1)
    meta.setdefault('lang', lang)
    meta.setdefault('name', 'unnamed')

    pipeline = ['ner']

    # Take dropout and batch size as generators of values -- dropout
    # starts high and decays sharply, to force the optimizer to explore.
    # Batch size starts at 1 and grows, so that we make updates quickly
    # at the beginning of training.
    dropout_rates = util.decaying(util.env_opt('dropout_from', 0.2),
                                  util.env_opt('dropout_to', 0.2),
                                  util.env_opt('dropout_decay', 0.0))
    batch_sizes = util.compounding(util.env_opt('batch_from', 1),
                                   util.env_opt('batch_to', 16),
                                   util.env_opt('batch_compound', 1.001))
    max_doc_len = util.env_opt('max_doc_len', 5000)
    corpus = GoldCorpus(train_path, dev_path, limit=n_sents)
    n_train_words = corpus.count_train()

    lang_class = util.get_lang_class(lang)
    nlp = lang_class()

    if "ner" in nlp.pipe_names:

    ner = nlp.create_pipe("ner")
    nlp.add_pipe(ner, first=True)

    meta['pipeline'] = pipeline
    if vectors:
        print("Load vectors model", vectors)
        util.load_model(vectors, vocab=nlp.vocab)
        for lex in nlp.vocab:
            values = {}
            for attr, func in nlp.vocab.lex_attr_getters.items():
                # These attrs are expected to be set by data. Others should
                # be set by calling the language functions.
                if attr not in (CLUSTER, PROB, IS_OOV, LANG):
                    values[lex.vocab.strings[attr]] = func(lex.orth_)
            lex.is_oov = False
#    for name in pipeline:
#        nlp.add_pipe(nlp.create_pipe(name), name=name)
    if parser_multitasks:
        for objective in parser_multitasks.split(','):
    if entity_multitasks:
        for objective in entity_multitasks.split(','):
    optimizer = nlp.begin_training(lambda: corpus.train_tuples, device=use_gpu)
    nlp._optimizer = None
    other_pipes = [pipe for pipe in nlp.pipe_names if pipe != "ner"]

    if(newLabels != None):
        for l in newLabels:

    print("Itn.  Dep Loss  NER Loss  UAS     NER P.  NER R.  NER F.  Tag %   Token %  CPU WPS  GPU WPS")
        train_docs = corpus.train_docs(nlp, projectivize=True, noise_level=0.0,
                                       gold_preproc=gold_preproc, max_length=0)
        train_docs = list(train_docs)
        with nlp.disable_pipes(*other_pipes):
            for i in range(n_iter):
                with tqdm.tqdm(total=n_train_words, leave=False) as pbar:
                    losses = {}
                    for batch in minibatch(train_docs, size=batch_sizes):
                        batch = [(d, g) for (d, g) in batch if len(d) < max_doc_len]
                        if not batch:
                        docs, golds = zip(*batch)
                        nlp.update(docs, golds, sgd=optimizer,
                                   drop=next(dropout_rates), losses=losses)
                        pbar.update(sum(len(doc) for doc in docs))

                with nlp.use_params(optimizer.averages):
                    epoch_model_path = output_path / ('model%d' % i)
                    nlp_loaded = util.load_model_from_path(epoch_model_path)
                    dev_docs = list(corpus.dev_docs(
                    nwords = sum(len(doc_gold[0]) for doc_gold in dev_docs)
                    start_time = timer()
                    scorer = nlp_loaded.evaluate(dev_docs, verbose)
                    end_time = timer()
                    if use_gpu < 0:
                        gpu_wps = None
                        cpu_wps = nwords/(end_time-start_time)
                        gpu_wps = nwords/(end_time-start_time)
                        with Model.use_device('cpu'):
                            nlp_loaded = util.load_model_from_path(epoch_model_path)
                            dev_docs = list(corpus.dev_docs(
                                            nlp_loaded, gold_preproc=gold_preproc))
                            start_time = timer()
                            scorer = nlp_loaded.evaluate(dev_docs)
                            end_time = timer()
                            cpu_wps = nwords/(end_time-start_time)
                    acc_loc = (output_path / ('model%d' % i) / 'accuracy.json')
                    with'w') as file_:
                    meta_loc = output_path / ('model%d' % i) / 'meta.json'
                    meta['accuracy'] = scorer.scores
                    meta['speed'] = {'nwords': nwords, 'cpu': cpu_wps,
                                     'gpu': gpu_wps}
                    meta['vectors'] = {'width': nlp.vocab.vectors_length,
                                       'vectors': len(nlp.vocab.vectors),
                                       'keys': nlp.vocab.vectors.n_keys}
                    meta['lang'] = nlp.lang
                    meta['pipeline'] = pipeline
                    meta['spacy_version'] = '>=%s' % about.__version__
                    meta.setdefault('name', 'model%d' % i)
                    meta.setdefault('version', version)

                    with'w') as file_:
                print_progress(i, losses, scorer.scores, cpu_wps=cpu_wps,
        print("Saving model...")
        with nlp.use_params(optimizer.averages):
            final_model_path = output_path / 'model-final'

What I've Done

When the attempting to run the full JSON file failed, I attempted the same with a smaller sample of 100. The process was able to run all the way through with no issues. Now before I go chopping up my dataset into bite sized chunks of 100 (which i really don't want / shouldn't have to do) i wanted to see if anyone could take a look and see if this is possibly some sort of 1. limit in spacy i have somehow hit, 2. memory issue, 3. or some sort of code issue i overlooked.

Please be advised that this process is being run on my local machine which is speced as follows:

PC specs

Windows 10
Intel(R) Core(TM) i7-6600U CPU @ 2.6GHz 2.81 GHz
16.0 GB Ram
Python 3.7.4
spaCy 2.0.16

Any help would be greatly appreciated thank you


After i asked this question, i figured in the mean time i would attempt to process my files in small batches of 100. Interestingly enough, one of the files caused the process to crash. Immediately i thought it was a data issue, so i added a "print" to the training function so i can see which text was causing it. But after i added the "print" the file completed without error. I am not sure what to make of this but just some added information.


I was finally able to get an error message related to the crash. Unhandled exception at 0x00007FF8EB9E2BE2 (ner.cp37-win_amd64.pyd) in python.exe: 0xC0000005: Access violation reading location 0x000001C4213D1FE4. occurred The error is marked on invoke_main() within exe_common.inl I've attempted to find more information about this error and only found very little. It appears to be some kind of Windows error? Any help is appreciated.

asked on Stack Overflow Oct 15, 2019 by Brendt McKnight • edited Oct 23, 2019 by Brendt McKnight

1 Answer


In the end this turned out to be version incompatibilities between some of spaCy's dependents. This appears to have been caused by several uninstalls and re-installs of older and newer versions of spaCy. I got a fresh environment made and installed only the most current version of spaCy and everything works great. If you are using Anaconda Navigator i would not trust the package installer from the UI. It appears to be linked to older versions and you are much better off using PIP from the terminal.

answered on Stack Overflow Oct 24, 2019 by Brendt McKnight

User contributions licensed under CC BY-SA 3.0