Evaluation of Large Language Models on Summarizing, Classifying and Information Retrieval for Technological Patents

Thumbnail Image

URL

Journal Title

Journal ISSN

Volume Title

Perustieteiden korkeakoulu | Master's thesis

Authors

Date

2024-08-19

Department

Major/Subject

Machine Learning, Data Science and Artificial Intelligence

Mcode

SCI3044

Degree programme

Master’s Programme in Computer, Communication and Information Sciences

Language

en

Pages

41

Series

Abstract

Millions of patents are filed every year to protect intellectual property. The patents are typically lengthy and complex, making it necessary to have automated ways to classify, summarize and extract information from them. Recently, large language models have emerged as an option to tackle these tasks in natural language processing. In this thesis, a dataset is developed for the evaluation of large language models on technological patents. The dataset is composed of summarization, classification and information retrieval tasks. Four 7-billion parameter general purpose language models are evaluated on the dataset. The models are used within a one-shot prompting context. The models achieve good results compared to traditional machine learning models, proving the potential usefulness of large language models in a setting with a limited amount of labelled training data.

Description

Supervisor

Kurimo, Mikko

Thesis advisor

Wang, Liang

Keywords

patenting, large language models, natural language processing, generative AI

Other note

Citation

Permanent link to this item

https://urn.fi/URN:NBN:fi:aalto-202408255802

Collections

[dipl] Perustieteiden korkeakoulu / SCI