View source for Research:Wikipedia type Articles Generated by LLM (Not for Publication on Wikipedia)

{{Research_project
<!-- When was the project page created? Do not edit this section, it will be autogenerated-->
 | created = 09:26, 21 November 2023 (UTC)
 
<!-- Who should be contacted about this project? e.g. {{Investigator|Aaron Halfaker|Wikimedia Foundation}} You can link to your username if you have one:  {{Investigator|[[User:EpochFail|Aaron Halfaker]]|Wikimedia Foundation}}-->
 | contact = {{Investigator|[[User:Terribilis11|Theodore Kanell]]|Stanford University}}, {{Investigator|Yijia Shao| Stanford University}}, {{Investigator|Peter Xu|Stanford University}}, {{Investigator|YuCheng Jiang|Stanford University}}
 
<!-- Who else is working on the project? You can add as many {{Investigator}} entries as needed -->
 | collaborators = 
 
<!-- What year/month did work start? ... or when will it start? -->
 | start_year = 
 | start_month = 
 
<!-- What year/month did work end? ... or when will it end? -->
 | end_year = 
 | end_month = 
 
<!-- Project status: draft -> proposed -> planned -> active -> completed -->
 | status = active
 
<!-- Tags represent metadata about the project.  This section provides templates for all available tags.  -->
 | tags = 
 
<!-- These tags are only used for projects formally supported by the Wikimedia Foundation under our Open Access policy. Your contact at WMF will take care of filling this out.
  {{Research_project/Tags/WMF supported
   | contact = 
   | api = 
   | data_collection = 
   | endorsement = 
   | funding = 
   | private_data = 
   | subject_recruitment =
  }}
-->

<!-- This project's manuscript is published as open-access 
  {{Research_project/Tags/Open access
   | url = 
   | label = 
   | ref = 
  }}
-->
 
<!-- This project's code is open-source
  {{Research_project/Tags/Open source
   | url = 
   | label = 
  }}
-->
 
<!-- This project's data is available for download and reuse.
  {{Research_project/Tags/Open data
   | url = 
   | label = 
  }}
-->

}} <!-- End of Research project template -->

''We are students working on a research project in conjunction with the Stanford Open Virtual Assistant Lab
[https://oval.cs.stanford.edu/ (OVAL)] 

Our research team is building a LLM-based system which can generate a full-length (capped at 2.5k tokens around 1.25k words) Wikipedia page for a given topic without the need for supplemental information (e.g., human written outlines, curated references, etc.). Besides automatic evaluation, we would like to have frequent wikipedia editors collaborate with scoring the articles and providing feedback. 

Our goal is only for educational research, and we are not intending to try to publish these LLM generated articles on Wikipedia. Our LLM will ideally generate Wikipedia style articles with citations, and different sub-points. We will be scoring the essay based on 1. Well Written, 2. Verifiable with no original research, 3. Broad in its coverage, and 4. Qualitative comments (The first three metrics for a Good Article + Qualitative comments). We would take a subset of our articles produced and score them by actual Wikipedia editors as a way to verify our scoring is within reason.

The Stanford Open Virtual Assistant Lab is a supporter of Wikipedia and its editors. We want to do all we can to guarantee our work is not detrimental to Wikipedia. 
''

== Methods ==
''
Please fill out this form if interested which includes a consent form in compliance with IRB standards. 
Link[https://docs.google.com/forms/d/e/1FAIpQLSfaivclenvs9pdnW7cFcsTyvYy-wSCR_Vr_oYzJx_2bm-ZAqA/viewform?usp=sf_link]

We are hoping to gather a group of 10-15 frequent Wikipedia editors to score a sample size of around 30 articles. We know that this can be time-consuming especially concerning verifiability and hallucination of an LLM. To help ease this we will be using a UI system that will show the given articles while also having the general sections of citations immediately available to streamline the process. 
''

=== Timeline ===
''We are hoping to have human evaluators by the end of November or beginning of December.''

=== Policy, Ethics and Human Subjects Research ===
''
IRB approval August 23, 2021. 
In this work, we study the automatic Wikipedia generation problem as a way to push the frontier of automatic expository writing and automatic knowledge curation. All the studies and the evaluation in this work are designed to prevent the dissemination of misinformation by not publishing generated content online and implementing strict accuracy checks. We avoid any disruption to Wikipedia or related communities, as our system does not interact with live pages. Also, although we try to generate grounded articles, we believe there is no privacy issue related to this work as we only use information publicly available on the Internet.

The primary risk of our work is that the Wikipedia articles written by our system are grounded on information on the Internet which may contain some biased or discriminative contents. Currently, our system relies on the search engine to retrieve high-quality information but does not include any post-processing module. We believe improving the retrieval module to have good coverage of different viewpoints and adding a content sifting module to the current system will be a critical next step to achieve better neutrality and balance in the generated articles. In our experiment, we manually go through all the topics in the test set to ensure the topics themselves are not biased or discriminative.

Another limitation we see from an ethical point of view is that we only consider writing English Wikipedia articles in this work. Extending the current system to a multilingual setup is a meaningful direction for future work as there are more interesting topics that do not have their Wikipedia pages in non-English languages. 
''

== Results==
''Results pending''

== Resources ==
''''

== References ==
<references />