Natural Language Processing: Top Key Challenges & Applications

Such a collection needs to be versioned, to enable updates beyond the cycle of academic review and to enable replicability and comparison to prior approaches. In order to better understand the strengths and weaknesses of our models, we furthermore require more fine-grained evaluation across a single metric, highlighting on what types of examples models excel and fail at. ExplainaBoard (Liu et al., 2021) implements such a fine-grained breakdown of model performance across different tasks, which can be seen below. Another way to obtain a more fine-grained estimate of model performance is to create test cases for specific phenomena and model behaviour, for instance using the CheckList framework (Ribeiro et al., 2020). Many recent influential benchmarks such as ImageNet, SQuAD, or SNLI are large in scale, consisting of hundreds of thousands of examples and were developed by academic groups at well-funded universities.

The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. Participatory events such as workshops and hackathons are one practical solution to encourage cross-functional synergies and attract mixed groups of contributors from the humanitarian sector, academia, and beyond. In highly multidisciplinary sectors of science, regular hackathons have been extremely successful in fostering innovation (Craddock et al., 2016). Major NLP conferences also support workshops on emerging areas of basic and applied NLP research. Formulating a comprehensive definition of humanitarian action is far from straightforward.

Concept Challenges of natural language processing (NLP)#

Applying normalization to our example allowed us to eliminate two columns–the duplicate versions of “north” and “but”–without losing any valuable information. Combining the title case and lowercase variants also has the effect of reducing sparsity, since these features are now found across more sentences. Conversational AI can extrapolate which of the important words in any given sentence are most relevant to a user’s query and deliver the desired outcome with minimal confusion.

The recent GEM benchmark, for instance, explicitly includes metrics as a component that should be improved over time, as can be seen below. It is also worth acknowledging that there is extensive literature on the use of openness to counteract these restrictions. 5 Masakhane is a grassroots organization whose mission is to strengthen and spur NLP research in African languages for Africans by Africans.

Machine Translation is generally translating phrases from one language to another with the help of a statistical engine like Google Translate. The challenge with machine translation technologies is not directly translating words but keeping the meaning of sentences intact along with grammar and tenses. In recent years, various methods have been proposed to automatically evaluate machine translation quality by comparing hypothesis translations with reference translations. The ability to de-bias data (i.e. by providing the ability to inspect, explain and ethically adjust data) represents another major consideration for the training and use of NLP models in public health settings. Failing to account for biases in the development (e.g. data annotation), deployment (e.g. use of pre-trained platforms) and evaluation of NLP models could compromise the model outputs and reinforce existing health inequity (74).

Examples of Natural Language Processing in Action

Initiatives like Masakhane, which has amassed a network of more than 2,000 African researchers actively engaged in publishing research, and the KenCorpus project unite researchers to elevate local languages. Chat GPT has created tremendous speculation among stakeholders in academia, not the least of whom are researchers and teaching staff (Biswas, 2023). Chat GPT is a Natural Language Processing (NLP) model developed by OpenAI that uses a large dataset to generate text responses to student queries, feedback, and prompts (Gilson et al., 2023). It can simulate conversations with students to provide feedback, answer questions, and provide support (OpenAI, 2023).

Therefore, you should be aware of the potential risks and implications of your NLP work, such as bias, discrimination, privacy, security, misinformation, and manipulation. You should also follow the best practices and guidelines for ethical and responsible NLP, such as transparency, accountability, fairness, inclusivity, and sustainability. Since the number of labels in most classification problems is fixed, it is easy to determine the score for each class and, as a result, the loss from the ground truth. But in NLP, though output format is predetermined in the case of NLP, dimensions cannot be specified. It is because a single statement can be expressed in multiple ways without changing the intent and meaning of that statement. Evaluation metrics are important to evaluate the model’s performance if we were trying to solve two problems with one model.

Yet, organizations often issue written reports that contain this information, which could be converted into structured datasets using NLP technology. Chatbots have previously been used to provide individuals with health-related assistance in multiple contexts20, and the Covid-19 pandemic has further accelerated the development of digital tools that can be deployed in the context of health emergencies. The use of language technology to deliver personalized support is, however, still rather sparse and Chat GPT unsystematic, and it is hard to assess the impact and scalability of existing applications. More recently, machine translation was also attempted to adapt and evaluate cTAKES concept extraction to German [80], with very moderate success. Making use of multilingual resources for analysing a specific language seems to be a more fruitful approach [152, 153, 164]. Machine translation is used for cross-lingual Information Retrieval to improve access to clinical data for non-native English speakers.

In addition, as one of the main bottlenecks is the lack of data and standards for this domain, we present recent initiatives (the DEEP and HumSet) which are directly aimed at addressing these gaps. You can foun additiona information about ai customer service and artificial intelligence and NLP. With this work, we hope to motivate https://chat.openai.com/ humanitarians and NLP experts to create long-term impact-driven synergies and to co-develop an ambitious roadmap for the field. There are a number of additional resources that are relevant to this class of applications.

Intelligent document processing

However, manually-labelled gold standard annotations remain a prerequisite and though ML models are increasingly capable of automated labelling, human annotation becomes essential in cases where data cannot be auto-labelled with high confidence. The first category is linguistic and refers to the challenges of decoding the inherent complexity of human language and communication. Openness must be practiced in a manner that considers the communities directly or indirectly providing the data used in commercial and noncommercial settings for AI development. The interests of these communities may, depending on the use case, involve financial benefits, social benefits, or (mere) attribution or acknowledgment. The MakerereNLP project involved the delivery of open, accessible, and high-quality text and speech datasets for East African languages from Uganda, Tanzania, and Kenya.

Based on this assumption, words can be represented as vectors of numbers that quantify (more or less explicitly) how often they tend to co-occur with each other word in the vocabulary (i.e., being present in the same sentence, or in a window of given length). These vectors can be interpreted as coordinates on a high-dimensional semantic space where words with similar meanings (“cat” and “dog”) will be closer than words whose meaning is very different (“cat” and “teaspoon”, see Figure 1). This simple intuition makes it possible to represent the meaning of text in a quantitative form that can be operated upon algorithmically or used as input to predictive models. We refer to Boleda (2020) for a deeper explanation of this topic, and also to specific realizations of this idea under the word2vec (Mikolov et al., 2013), GloVe (Bojanowski et al., 2016), and fastText (Pennington et al., 2014) algorithms. In summary, there are still a number of open challenges with regard to deep learning for natural language processing. Deep learning, when combined with other technologies (reinforcement learning, inference, knowledge), may further push the frontier of the field.

NLP, a branch of Artificial Intelligence (AI), intelligently understands and derives meaning from human language. It aids developers in organizing and structuring data to perform tasks, such as automatic summarization, translation, named entity recognition, relationship extraction, sentiment analysis, speech recognition, and topic segmentation. The use of social media data during the 2010 Haiti earthquake is an example of how social media data can be leveraged to map disaster-struck regions and support relief operations during a sudden-onset crisis (Meier, 2015). On January 12th, 2010, a catastrophic earthquake struck Haiti, causing widespread devastation and damage, and leading to the death of several hundred thousand people. In the immediate aftermath of the earthquake, a group of volunteers based in the United States started developing a “crisis map” for Haiti, i.e., an online digital map pinpointing areas hardest hit by the disaster, and flagging individual calls for help.

The more features you have, the more possible combinations between features you will have, and the more data you’ll need to train a model that has an efficient learning process. That is why we often look to apply techniques that will reduce the dimensionality of the training data. It can identify that a customer is making a request for a weather forecast, but the location (i.e. entity) is misspelled in this example. By using spell correction on the sentence, and approaching entity extraction with machine learning, it’s still able to understand the request and provide correct service.

Various researchers (Sha and Pereira, 2003; McDonald et al., 2005; Sun et al., 2008) [83, 122, 130] used CoNLL test data for chunking and used features composed of words, POS tags, and tags. Improving models’ grasp of context involves using advanced algorithms and expanding training datasets to be more diverse and encompassing. Enhanced Machine Learning ModelsRecent advancements have introduced models like BERT and GPT-3, which better understand context and ambiguity. These models are trained on extensive datasets, enabling a deeper grasp of language nuances.

• By providing the ability to rapidly analyze large amounts of unstructured or semistructured text, NLP has opened up immense opportunities for text-based research and evidence-informed decision making (29–34).
• However, there is a lot of focus on graph-inspired RNNs as it emerges that a graph structure may serve as the best representation of NLP data.
• One example is Gamayun (Öktem et al., 2020), a project aimed at crowdsourcing data from underrepresented languages.
• Under this license, the dataset can be used for any purpose, including commercial purposes, and adaptations or derivative data outputs must be shared under identical terms.
• Effective change management practices are crucial to facilitate the adoption of new technologies and minimize disruption.
• The authors would like to thank Galja Angelova and Svetla Boycheva for their knowledgeable insight on clinical NLP work on Bulgarian.

By engaging technologists, members of the scientific and medical community and the public in creating tools with open data repositories, funders can exponentially increase utility and value of those data to help solve pressing national health issues. This challenge is part of a broader conceptual initiative at NCATS to change the “currency” of biomedical research. NCATS held a Stakeholder Feedback Workshop in June 2021 to solicit feedback on this concept and its implications for researchers, publishers and the broader scientific community.

While not specific to the clinical domain, this work may create useful resources for clinical NLP. Medical ethics, translated into privacy rules and regulations, restrict the access to and sharing of clinical corpora. Some datasets of biomedical documents annotated with entities of clinical interest may be useful for clinical NLP [59]. Finally, we identify major NLP challenges and opportunities with impact on clinical practice and public health studies accounting for language diversity. Natural Language Processing excels at understanding syntax, but semiotics and pragmatism are still challenging to say the least.

Ethical Concerns and Biases in NLP Models

Since razor-sharp delivery of results and refining of the same becomes crucial for businesses, there is also a crunch in terms of training data required to improve algorithms and models. A conversational AI (often called a chatbot) is an application that understands natural language input, either spoken or written, and performs a specified action. A conversational interface can be used for customer service, sales, or entertainment purposes. Another main use case of NLP is through sentiment analysis, where machines comprehend data and text to determine the mood of the sentiment expressed.

7 The Kenya Language Corpus was founded by Maseno University, the University of Nairobi, and Africa Nazarene University early in 2021. These universities have been jointly creating a language corpus, and while using machine learning and NLP, are creating tomorrow’s African language chatbot. In the Igbo and Setswana languages, these sayings include expressions that speak to how discussions about taking (or bringing) often revolve around other people’s property.

Most crises require coordinating response activities across multiple sectors and clusters, and there is increasing emphasis on devising mechanisms that support effective inter-sectoral coordination. Secondly, pretrained NLP models often absorb and reproduce biases (e.g., gender and racial biases) present in the training data (Shah et al., 2019; Blodgett et al., 2020). This is also a known issue within the NLP community, and there is increasing nlp challenges focus on developing strategies aimed at preventing and testing for such biases. Vector representations of sample text excerpts in three languages created by the USE model, a multilingual transformer model, (Yang et al., 2020) and projected into two dimensions using TSNE (van der Maaten and Hinton, 2008). Text excerpts are extracted from a recent humanitarian response dataset (HUMSET, Fekih et al., 2022; see Section 5 for details).

Give this NLP sentiment analyzer a spin to see how NLP automatically understands and analyzes sentiments in text (Positive, Neutral, Negative). Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., et al. (2020). “Language models are few-shot learners,” in Advances in Neural Information Processing Systems 33 (NeurIPS 2020), (Online). All authors sought relevant references to be added and each contributed to the creation of Table 2. All authors contributed to the writing process and approved the final version of the manuscript.

Within Canada, health data are generally controlled regionally and, due to security and confidentiality concerns, there is reluctance to provide unhindered access to these systems and their integration with other datasets (e.g. data linkage). A recent survey of social media users found that the majority considered analysis of their social media data to identify mental health issues “intrusive and exposing” and they would not consent to this (84). The objective of this manuscript is to provide a framework for considering natural language processing (NLP) approaches to public health based on historical applications. This overview includes a brief introduction to AI and NLP, suggests opportunities where NLP can be applied to public health problems and describes the challenges of applying NLP in a public health context.

• Modeling tools similar to those deployed for social and news media analysis can be used to extract bottom-up insights from interviews with people at risk, delivered either face-to-face or via SMS and app-based chatbots.
• Conversational AI can extrapolate which of the important words in any given sentence are most relevant to a user’s query and deliver the desired outcome with minimal confusion.
• When students are provided with content relevant to their interests and abilities, they are more likely to engage with the material and develop a deeper understanding of the subject matter.
• The UMLS (Unified Medical Language System [137]) aggregates more than 100 biomedical terminologies and ontologies.
• For example, in image retrieval, it becomes feasible to match the query (text) against images and find the most relevant images, because all of them are represented as vectors.

Additionally, the model’s accuracy might be impacted by the quality of the input data provided by students. If students do not provide clear, concise, and relevant input, the system might struggle to generate an accurate response. This is particularly challenging in cases in which students are not sure what information they need or cannot articulate their queries in a way that the system easily understands.

With the emergence of the COVID-19, NLP has taken a prominent role in the outbreak response efforts (88,89). NLP has been rapidly employed to analyze the vast quantity of textual information that has been made available through unrestricted access to peer-review journals, preprints and digital media (90). NLP has been widely used to support the medical and scientific communities in finding answers to key research questions, summarization of evidence, question answering, tracking misinformation and monitoring of population sentiment (91–97).

Creating large-scale resources and data standards that can scaffold the development of domain-specific NLP models is essential to make many of these goals realistic and possible to achieve. In other domains, general-purpose resources such as web archives, patents, and news data, can be used to train and test NLP tools. There is increasing emphasis on developing models that can dynamically predict fluctuations in humanitarian needs, and simulate the impact of potential interventions. This, in turn, requires epidemiological data and data on previous interventions which is often hard to find in a structured, centralized form.

When designing a benchmark, collecting—at a minimum—test data in other languages may help to highlight new challenges and promote language inclusion. Similarly, when evaluating models, leveraging the increasing number of non-English language datasets in tasks such as question answering and summarisation (Hasan et al., 2021) can provide additional evidence of a model’s versatility. Finally, as with any new technology, consideration must be given to assessment and evaluation of NLP models to ensure that they are working as intended and keeping in pace with society’s changing ethical views. These NLP technologies need to be assessed to ensure they are functioning as expected and account for bias (87). Although today many approaches are posting equivalent or better-than-human scores on textual analysis tasks, it is important not to equate high scores with true language understanding.

This is another major obstacle to technical progress in the field, as open sourcing would allow a broader community of humanitarians and NLP experts to work on developing tools for humanitarian NLP. The development of efficient solutions for text anonymization is an active area of research that humanitarian NLP can greatly benefit from, and contribute to. Social media posts and news media articles may convey information which is relevant to understanding, anticipating, or responding to both sudden-onset and slow-onset crises. Research on the use of NLP for targeted information extraction from, and document classification of, EHR text shows that some degree of success can be achieved with basic text processing techniques. It can be argued that a very shallow method such as lexicon matching/regular expressions to a customized lexicon/terminology is sufficient for some applications [128]. For tasks where a clean separation of the language-dependent features is possible, porting systems from English to structurally close languages can be fairly straightforward.

The very fact of using or reusing these datasets means consenting to the proprietary nature of the data and other terms upon which the data is made available. Some of these terms may mean that, while the proprietary nature of the data is acknowledged, such proprietary rights are given up in their entirety. This requires collective efforts, considering the broader impacts on society and on the individuals who contribute data.

Recently many model agnostic tools have been developed to assess and correct unfairness in machine learning models in accordance with the efforts by the government and academic communities to define unacceptable AI development (76–81). The humanitarian sector—that is, the ecosystem of organizations and activities aimed at providing assistance in the context of crises and disasters—could greatly benefit from tools that make it possible to draw operational insights from large volumes of text data. Secondary sources such as news media articles, social media posts, or surveys and interviews with affected individuals also contain important information that can be used to monitor, prepare for, and efficiently respond to humanitarian crises.

This is especially problematic in contexts where guaranteeing accountability is central, and where the human cost of incorrect predictions is high. Finally, we analyze and discuss the main technical bottlenecks to large-scale adoption of NLP in the humanitarian sector, and we outline possible solutions (Section 6). We conclude by highlighting how progress and positive impact in the humanitarian NLP space rely on the creation of a functionally and culturally diverse community, and of spaces and resources for experimentation (Section 7). In summary, there is a sharp difference in the availability of language resources for English on one hand, and other languages on the other hand.

The technique is highly used in NLP challenges — one of them being to understand the context of words. This challenge will spur innovation in NLP to advance the field and allow the generation of more accurate and useful data from biomedical publications, which will enhance the ability for data scientists to create tools to foster discovery and generate new hypotheses. This promotes the development of resources for basic science research, as well as developing partnerships with software designers in the NLP space.

Without cross-disciplinary and cross-domain collaboration (between the areas of linguistics, journalism, and AI), communities may lose their ability to guide how their languages progress amid the AI revolution. More open communication channels between communities, researchers, private actors, and government actors are imperative for articulating societal needs and priorities in an evolving technological landscape. Hidden Markov Models are extensively used for speech recognition, where the output sequence is matched to the sequence of individual phonemes. HMM is not restricted to this application; it has several others such as bioinformatics problems, for example, multiple sequence alignment [128].

NLP labeling is an iterative process, for both entity recognition and named entity recognition. Organizations face considerable challenges in managing large volumes of documents, and the use of Named Entity Recognition (NER) can help overcome these challenges by automatically extracting information from text, audio and video documents. RAVN Systems, a leading expert in Artificial Intelligence (AI), Search and Knowledge Management Solutions, announced the launch of a RAVN (“Applied Cognitive Engine”) i.e. powered software Robot to help and facilitate the GDPR (“General Data Protection Regulation”) compliance.

Particular articles were chosen to emphasize the breadth of potential applications for NLP in public health as well as the not inconsiderable challenges and risks inherent in incorporating AI/NLP in public health analysis and decision support. Pragmatic level focuses on the knowledge or content that comes from the outside the content of the document. When a sentence is not specific and the context does not provide any specific information about that sentence, Pragmatic ambiguity arises (Walton, 1996) [143]. Pragmatic ambiguity occurs when different persons derive different interpretations of the text, depending on the context of the text.

A final challenge of NLP for AI is the ethical and social issues that arise from its use and impact. NLP can have positive or negative effects on human communication, interaction, and decision-making, depending on how it is designed, implemented, and regulated. For example, NLP can enhance customer service, education, and accessibility, but it can also pose risks of privacy, security, bias, discrimination, and misinformation. To overcome this challenge, businesses need to follow ethical principles and best practices for NLP development and deployment.

Srihari [129] explains the different generative models as one with a resemblance that is used to spot an unknown speaker’s language and would bid the deep knowledge of numerous languages to perform the match. Discriminative methods rely on a less knowledge-intensive approach and using distinction between languages. Whereas generative models can become troublesome when many features are used and discriminative models allow use of more features [38]. Few of the examples of discriminative methods are Logistic regression and conditional random fields (CRFs), generative methods are Naive Bayes classifiers and hidden Markov models (HMMs).

Refine algorithms for greater processing efficiency, thus reducing the need for extensive hardware resources. Many NLP tools are developed with a focus on English, leaving speakers of other languages disadvantaged. Applying stemming to our four sentences reduces the plural “kings” to its singular form “king”. We can apply another pre-processing technique called stemming to reduce words to their “word stem”. For example, words like “assignee”, “assignment”, and “assigning” all share the same word stem– “assign”. By reducing words to their word stem, we can collect more information in a single feature.

Faster and more powerful computers have led to a revolution of Natural Language Processing algorithms, but NLP is only one tool in a bigger box. Data scientists have to rely on data gathering, sociological understanding, and just a bit of intuition to make the best out of this technology. Facebook vs. Power Ventures Inc is one of the most well-known examples of big-tech trying to push against the practice. In this case, Power Ventures created an aggregate site that allowed users to aggregate data about themselves from different services, including LinkedIn, Twitter, Myspace, and AOL. The other issue, and the one most relevant to us, is the limited ability of humans to consume data since most adult humans can only read about 200 to 250 words per minute – college graduates average at around 300 words. You can extract all the data into a structured, machine-readable JSON format with parsed tasks, descriptions and SOTA tables.