{"id":7416,"date":"2023-12-22T11:15:25","date_gmt":"2023-12-22T10:15:25","guid":{"rendered":"https:\/\/eurocc.nscc.sk\/?p=7416"},"modified":"2024-02-15T13:40:40","modified_gmt":"2024-02-15T12:40:40","slug":"identifikacia-entit-pre-extrakciu-adries-z-transkriptovanych-rozhovorov-s-vyuzitim-syntetickych-dat","status":"publish","type":"post","link":"https:\/\/eurocc.nscc.sk\/en\/identifikacia-entit-pre-extrakciu-adries-z-transkriptovanych-rozhovorov-s-vyuzitim-syntetickych-dat\/","title":{"rendered":"Named Entity Recognition for Address Extraction in Speech-to-Text Transcriptions Using Synthetic Data"},"content":{"rendered":"
\n
\n
\n
\n

Named Entity Recognition for Address Extraction in Speech-to-Text Transcriptions Using Synthetic Data<\/strong><\/p>\n\n\n\n

<\/p>\n<\/div><\/div>\n\n\n\n

Many businesses spend large amounts of resources for communicating with clients. Usually, the goal is\nto provide clients with information, but sometimes there is also a need to request specific information\nfrom them.\nIn addressing this need, there has been a significant effort put into the development of chatbots\nand voicebots, which on one hand serve the purpose of providing information to clients, but they can\nalso be utilized to contact a client with a request to provide some information.\nA specific real-world example is to contact a client, via text or via phone, to update their postal address. The address may have possibly changed over time, so a business needs to update this information\nin its internal client database. <\/p>\n\n\n\n

<\/p>\n<\/div>\n\n\n\n

\n
\"\"<\/a>
illustrative image<\/figcaption><\/figure>\n<\/div>\n<\/div>\n\n\n\n

Nonetheless, when requesting such information through novel channels|like chatbots or voicebots|\nit is important to verify the validity and format of the address. In such cases, an address information\nusually comes by a free-form text input or as a speech-to-text transcription. Such inputs may contain\nsubstantial noise or variations in the address format. To this end it is necessary to lter out the noise\nand extract corresponding entities, which constitute the actual address. This process of extracting\nentities from an input text is known as Named Entity Recognition (NER). In our particular case we\ndeal with the following entities: municipality name, street name, house number, and postal code. This\ntechnical report describes the development and evaluation of a NER system for extraction of such\ninformation.<\/p>\n\n\n\n

Problem Description and Our Approach<\/strong><\/p>\n\n\n\n

This work is a joint effort of Slovak National Competence Center for High-Performance Computing\nand nettle, s.r.o., which is a Slovak-based start-up focusing on natural language processing, chatbots,\nand voicebots. Our goal is to develop highly accurate and reliable NER model for address parsing. The\nmodel accepts both free text as well as speech-to-text transcribed text. Our NER model constitutes\nan important building block in real-world customer care systems, which can be employed in various\nscenarios where address extraction is relevant.

The challenging aspect of this task was to handle data which was present exclusively in Slovak\nlanguage. This makes our choice of a baseline model very limited.\nCurrently, there are several publicly available NER models for the Slovak language. These models\nare based on the general purpose pre-trained model SlovakBERT [1]. Unfortunately, all these models\nsupport only a few entity types, while the support for entities relevant to address extraction is missing.\nA straightforward utilization of popular Large Language Models (LLMs) like GPT is not an option\nin our use cases because of data privacy concerns and time delays caused by calls to these rather\ntime-consuming LLM APIs.<\/p>\n\n\n\n

We propose a fine-tuning of SlovakBERT for NER. The NER task in our case is actually a classification task at the token level. We aim at achieving proficiency at address entities recognition with a\ntiny number of real-world examples available. In Section 2.1 we describe our dataset as well as a data\ncreation process. The significant lack of available real-world data prompts us to generate synthetic\ndata to cope with data scarcity. In Section 2.2 we propose SlovakBERT modifications in order to train\nit for our task. In Section 2.3 we explore iterative improvements in our data generation approach.\nFinally, we present model performance results in Section 3.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Data<\/h2>\n\n\n\n

The aim of the task is to recognize street names, house numbers, municipality names, and postal codes\nfrom the spoken sentences transcribed via speech-to-text. Only 69 instances of real-world collected\ndata were available. Furthermore, all of those instances were highly affected by noise, e.g., natural\nspeech hesitations and speech transcription glitches. Therefore, we use this data exclusively for testing.\nTable 1 shows two examples from the collected dataset.<\/p>\n\n\n\n

<\/p>\n\n\n

\n
\"\"<\/a>
Table 1: Two example instances from our collected real-world dataset. The Sentence column show-\ncases the original address text. The Tokenized text column contains tokenized sentence representation,\nand the Tags column contains tags for the corresponding tokens. Note here that not every instance\nnecessarily contains all considered entity types. Some instances contain noise, while others have gram-\nmar\/spelling mistakes: The token \\ Dalsie\" is not a part of an address and the street name \\bauerova\"\nis not capitalized.<\/figcaption><\/figure><\/div>\n\n\n

Artificial generation of training dataset occurred as the only, but still viable option to tackle the\nproblem of data shortage. Inspired by the 69 real instances, we programmatically conducted numerous\nexternal API calls to OpenAI to generate similar realistic-looking examples. BIO annotation scheme [2]\nwas used to label the dataset. This scheme is a method used in NLP to annotate tokens in a sequence\nas the beginning (B), inside (I), or outside (O) of entities. We are using 9 annotations: O, B-Street,\nI-Street, B-Housenumber, I-Housenumber, B-Municipality, I-Municipality, B-Postcode, I-Postcode.

We generated data in multiple iterations as described below in Section 2.3. Our final training\ndataset consisted of more than 104\nsentences\/address examples. For data generation we used GPT3.5-turbo API along with some prompt engineering. Since the data generation through this API is\nlimited by the number of tokens \u2014 both generated as well as prompt tokens\u2014we could not pass the\nlist of all possible Slovak street names and municipality names within the prompt. Hence, data was\ngenerated with placeholders streetname and municipalityname only to be subsequently replaced\nby randomly chosen street and municipality names from the list of street and municipality names,\nrespectively. A complete list of Slovak street and municipality names was obtained from the web pages\nof the Ministry of Interior of the Slovak republic [3].

With the use of OpenAI API generative algorithm we were able to achieve organic sentences without\nthe need to manually generate the data, which sped up the process significantly. However, employing\nthis approach did not come without downsides. Many mistakes were present in the generated dataset,\nmainly wrong annotations occurred and those had to be corrected manually. The generated dataset was split, so that 80% was used for model\u2019s training, 15% for validation and 5% as synthetic test data,\nso that we could compare the performance of the model on real test data as well as on artificial test\ndata.<\/p>\n\n\n\n

Model Development and Training<\/strong> <\/p>\n\n\n\n

Two general-purpose pre-trained models were utilized and compared: SlovakBERT [1] and a distilled\nversion of this model [4]. Herein we refer to the distilled version as DistilSlovakBERT. SlovakBERT\nis an open-source pretrained model on Slovak language using a Masked Language Modeling (MLM)\nobjective. It was trained with a general Slovak web-based corpus, but it can be easily adapted to new\ndomains to solve new tasks [1]. DistilSlovakBERT is a pre-trained model obtained from SlovakBERT\nmodel by a method called knowledge distillation, which significantly reduces the size of the model\nwhile retaining 97% of its language understanding capabilities.

We modified both models by adding a token classification layer, obtaining in both cases models\nsuitable for NER tasks. The last classification layer consists of 9 neurons corresponding to 9 entity\nannotations: We have 4 address parts and each is represented by two annotations \u2013 beginning and\ninside of each entity, and one for the absence of any entity. The number of parameters for each model\nand its components are summarized in Table 2.<\/p>\n\n\n

\n
\"\"<\/a>
Table 2: The number of parameters in our two NER models and their respective counts for the base\nmodel and the classication head.<\/figcaption><\/figure><\/div>\n\n\n

Models\u2019 training was highly susceptible to overfitting. To tackle this and further enhance the\ntraining process we used linear learning rate scheduler, weight decay strategies, and some other hyperparameter tuning strategies.

Computing resources of the HPC system Devana, operated by the Computing Centre, Centre of\noperations of the Slovak Academy of Sciences were leveraged for model training, specifically utilizing\na GPU node with 1 NVidia A100 GPU. For a more convenient data analysis and debugging, an\ninteractive environment using OpenOnDemand was employed, which allows researches remote web\naccess to supercomputers.

The training process required only 10-20 epochs to converge for both models. Using the described\nHPC setting, one epoch\u2019s training time was on average 20 seconds for 9492 samples in the training\ndataset for SlovakBERT and 12 seconds for DistilSlovakBERT. Inference on 69 samples takes 0.64\nseconds for SlovakBERT and 0.37 seconds for DistilSlovakBERT, which demonstrates model\u2019s efficiency\nin real-time NLP pipelines.<\/p>\n\n\n\n

Iterative Improvements<\/strong><\/p>\n\n\n\n

Although only 69 instances of real data were present, the complexity of it was quite challenging to\nimitate in generated data. The generated dataset was created using several different prompts, resulting\nin 11,306 sentences that resembled human-generated content. The work consisted of a number of\niterations. Each iteration can be split into the following steps: generate data, train a model, visualize\nobtained prediction errors on real and artificial test datasets, and analyze. This way we identified\npatterns that the model failed to recognize. Based on these insights we generated new data that\nfollowed these newly identified patterns. The patterns we devised in various iterations are presented\nin Table 3. With each newly expanded dataset both of our models were trained, with SlovakBERT\u2019s\naccuracy always exceeding the one of DistilSlovakBERT\u2019s. Therefore, we have decided to further utilize\nonly SlovakBERT as a base model.<\/p>\n\n\n\n

Results<\/strong><\/p>\n\n\n\n

The confusion matrix corresponding to the results obtained using model trained in Iteration 1 (see\nTable 3)\u2014is displayed in Table 4. This model was able to correctly recognize only 67.51% of entities in test dataset. Granular examination of errors revealed that training dataset does not represent the\nreal-world sentences well enough and there is high need to generate more and better representative\ndata. In Table 4 it is evident, that the most common error was identification of a municipality as a\nstreet. We noticed that this occurred when municipality name appeared before the street name in the\naddress. As a result, this led to data generation with Iteration 2 and Iteration 3.<\/p>\n\n\n

\n
\"\"<\/a>
Table 3: The iterative improvements of data generation. Each prompt was used twice: First with and\nthen without noise, i.e., natural human speech hesitations. Sometimes, if mentioned, prompt allowed\nto shue or omit some address parts.<\/figcaption><\/figure><\/div>\n\n\n

This process of detailed analysis of prediction errors and subsequent data generation accounts for\nmost of the improvements in the accuracy of our model. The goal was to achieve more than 90%\naccuracy on test data. Model\u2019s predictive accuracy kept increasing with systematic data generation.\nEventually, the whole dataset was duplicated, with the duplicities being in uppercase\/lowercase. (The\nutilized pre-trained model is case sensitive and some test instances contained street and municipality\nnames in lowercase.) This made the model more robust to the form in which it receives input and led\nto final accuracy of 93.06%. Confusion matrix of the final model can be seen in Table 5.<\/p>\n\n\n

\n
\"\"<\/a>
Table 4: Confusion matrix of model trained on dataset from the rst iteration, reaching model's\npredictive accuracy of 67.51%.<\/figcaption><\/figure><\/div>\n\n
\n
\"\"<\/a>
Table 5: Confusion matrix of the nal model with the predictive accuracy of 93.06%. Comparing the\nresults to the results in Table 4, we can see that the accuracy increased by 25.55%.<\/figcaption><\/figure><\/div>\n\n\n

There are still some errors; notably, tokens that should have been tagged as outside were occasionally misclassified as municipality. We have opted not to tackle this issue further, as it happens\non words that may resemble subparts of our entity names, but, in reality, do not represent entities\nthemselves. See an example below in Table 6.<\/p>\n\n\n

\n
\"\"<\/a>
Table 6: Examples of the nal model's predictions for two test sentences. The rst sentence contains\none incorrectly classied token: the third token \\Kal\" with ground truth label O was predicted as\nB-Municipality. The misclassication of \\Kal\" as a municipality occurred due to its similarity to\nsubwords found in \\Kalsa\", but ground truth labeling was based on context and authors' judgment.\nThe second sentence has all its tokens classied correctly.<\/figcaption><\/figure><\/div>\n\n\n

Conclusions<\/strong><\/p>\n\n\n\n

In this technical report we trained a NER model built upon SlovakBERT pre-trained LLM model as\nthe base. The model was trained and validated exclusively on artificially generated dataset. This well\nrepresentative and high quality synthetic data was iteratively expanded. Together with hyperparameter fine-tuning this iterative approach allowed us to reach predictive accuracy on real dataset exceeding\n90%. Since the real dataset contained a mere 69 instances, we decided to use it only for testing.\nDespite the limited amount of real data, our model exhibits promising performance. This approach\nemphasizes the potential of using exclusively synthetic dataset, especially in cases where the amount\nof real data is not sufficient for training.

This model can be utilized in real-world applications within NLP pipelines to extract and verify the\ncorrectness of addresses transcribed by speech-to-text mechanisms. In case a larger real-world dataset\nis available, we recommend to retrain the model and possibly also expand the synthetic dataset with\nmore generated data, as the existing dataset might not represent potentially new occurring data\npatterns. This model can be utilized in real-world applications within NLP pipelines to extract and verify the\ncorrectness of addresses transcribed by speech-to-text mechanisms. In case a larger real-world dataset\nis available, we recommend to retrain the model and possibly also expand the synthetic dataset with\nmore generated data, as the existing dataset might not represent potentially new occurring data\npatterns.
The model is available on https:\/\/huggingface.co\/nettle-ai\/slovakbert-address-ner<\/a><\/p>\n\n\n\n

Acknowledgement<\/strong><\/p>\n\n\n\n

The research results were obtained with the support of the Slovak National competence centre for\nHPC, the EuroCC 2 project and Slovak National Supercomputing Centre under grant agreement\n101101903-EuroCC 2-DIGITAL-EUROHPC-JU-2022-NCC-01.<\/p>\n\n\n\n

AUTHORS <\/strong><\/p>\n\n\n\n

Bibi\u00e1na Laj\u010dinov\u00e1 \u2013 Slovak National Supercomputing Centre<\/p>\n\n\n\n

Patrik Val\u00e1bek \u2013 Slovak National Supercomputing Centre, ) Institute of Information Engineering, Automation, and Mathematics, Slovak University of Technology in Bratislava <\/p>\n\n\n\n

Michal Spi\u0161iak – nettle, s. r. o. <\/p>\n<\/div><\/div>\n\n\n\n

\n

Full version of the article SK<\/a>
Full version of the article EN<\/a><\/p>\n\n\n\n

References:<\/strong>:<\/p>\n\n\n\n

[1] Matu\u0301s Pikuliak, Stefan Grivalsky, Martin Konopka, Miroslav Blsta\u0301k, Martin Tamajka, Viktor Bachraty\u0301, Maria\u0301n Simko, Pavol Bala\u0301zik, Michal Trnka, and Filip Uhla\u0301rik. Slovakbert: Slovak masked language model. CoRR, abs\/2109.15254, 2021.

[2] Lance Ramshaw and Mitch Marcus. Text chunking using transformation-based learning. In Third Workshop on Very Large Corpora, 1995.
<\/p>\n\n\n\n

[3] Ministerstvo vnu\u0301tra Slovenskej republiky. Register adries. https:\/\/data.gov.sk\/dataset\/register-adries-register-ulic. Accessed: August 21, 2023.

[4] Ivan Agarsky\u0301. Hugging face model hub. https:\/\/huggingface.co\/crabz\/distil-slovakbert, 2022. Accessed: September 15, 2023.<\/p>\n\n\n\n

<\/p>\n\n\n\n


<\/p>\n\n\n\n

\n
Success-Stories<\/a><\/div>\n<\/div>\n\n\n
\"\"<\/a> Artificial Intelligence and a Supercomputer as a New Weapon Against Environmental Disasters<\/strong><\/a> 26 Mar<\/span> -<\/span> Scientists from Nitra, Slovakia are teaching machines to predict industrial failures before they can cause damage. Thanks to collaboration with the European supercomputer LUMI, they have developed a digital \u201cguardian\u201d capable of detecting pipeline leaks or manufacturing faults with high accuracy\u2014helping protect both the environment and companies\u2019 budgets.<\/span><\/div>
\"\"<\/a> The Slovak Recipe for Fair Play and Happier Players<\/strong><\/a> 25 Mar<\/span> -<\/span> Do you play games on your phone and sometimes feel like the game just doesn\u2019t understand you? Experts from Nitra, Slovakia, have used one of Europe\u2019s most powerful supercomputers to change that. Thanks to the Italian giant named Leonardo, they discovered how to read between the lines of player behavior and make the gaming experience more personal and fair.<\/span><\/div>
\"\"<\/a> Urban buildings awaken: Slovak AI gives a second chance to underused spaces<\/strong><\/a> 4 Mar<\/span> -<\/span> Mest\u00e1 s\u00fa \u017eiv\u00e9 organizmy, ktor\u00e9 sa neust\u00e1le menia. Mnoh\u00ed z n\u00e1s v\u0161ak v susedstve denne m\u00ed\u0148aj\u00fa tich\u00e9 svedectv\u00e1 minulosti \u2013 pr\u00e1zdne \u0161koly, nevyu\u017e\u00edvan\u00e9 \u00farady \u010di ch\u00e1traj\u00face verejn\u00e9 budovy. \u010casto si kladieme ot\u00e1zky: \u201ePre\u010do je to zatvoren\u00e9?\u201c \u201eNemohol by tu by\u0165 rad\u0161ej denn\u00fd stacion\u00e1r, \u0161k\u00f4lka alebo kult\u00farne centrum?\u201c<\/span><\/div><\/div>\n<\/div><\/div>","protected":false},"excerpt":{"rendered":"

Many businesses spend large amounts of resources for communicating with clients. Usually, the goal is\nto provide clients with information, but sometimes there is also a need to request specific information\nfrom them.\nIn addressing this need, there has been a significant effort put into the development of chatbots\nand voicebots, which on one hand serve the purpose of providing information to clients, but they can\nalso be utilized to contact a client with a request to provide some information.\nA specific real-world example is to contact a client, via text or via phone, to update their postal address. The address may have possibly changed over time, so a business needs to update this information\nin its internal client database. <\/p>","protected":false},"author":2,"featured_media":7435,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"templates\/template-full-width.php","format":"standard","meta":[],"categories":[9],"tags":[],"_links":{"self":[{"href":"https:\/\/eurocc.nscc.sk\/en\/wp-json\/wp\/v2\/posts\/7416"}],"collection":[{"href":"https:\/\/eurocc.nscc.sk\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/eurocc.nscc.sk\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/eurocc.nscc.sk\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/eurocc.nscc.sk\/en\/wp-json\/wp\/v2\/comments?post=7416"}],"version-history":[{"count":82,"href":"https:\/\/eurocc.nscc.sk\/en\/wp-json\/wp\/v2\/posts\/7416\/revisions"}],"predecessor-version":[{"id":8478,"href":"https:\/\/eurocc.nscc.sk\/en\/wp-json\/wp\/v2\/posts\/7416\/revisions\/8478"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/eurocc.nscc.sk\/en\/wp-json\/wp\/v2\/media\/7435"}],"wp:attachment":[{"href":"https:\/\/eurocc.nscc.sk\/en\/wp-json\/wp\/v2\/media?parent=7416"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/eurocc.nscc.sk\/en\/wp-json\/wp\/v2\/categories?post=7416"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/eurocc.nscc.sk\/en\/wp-json\/wp\/v2\/tags?post=7416"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}