Arxiv.org: The role of grey literature in competitive intelligence

The term grey literature is used to describe a wide range of different information that is produced outside of traditional publishing and distribution channels, and which is often not well represented in indexing databases. (Snook, 2023)

There are more definitions for grey literature such as the Institute of Scientific and Technical Information of China (ISTIC) Definition, National Library of Medicine (NLM) Definition, and the Prague Definition

A widely accepted definition in the scholarly community for grey literature is:

“Information produced on all levels of government, academia, business and industry in electronic and print formats not controlled by commercial publishing” ie. where publishing is not the primary activity of the producing body.”

(Third International Conference on Grey Literature in 1997 (ICGL Luxembourg definition, 1997  – Expanded in New York, 2004))

Introduction

Value of grey literature

Here is how grey literature can be an important asset for your research (Snook, 2023):

• It can capture research findings that yield zero or contradictory outcomes as well as discoveries in specialized or developing fields of study. Commercial publishers, who might have a more commonplace, profit-driven publication strategy, might not address these.
• It might be more up-to-date than research material that has been formally published, which can take time to go through a sometimes drawn-out peer review and publishing procedure.
• You can get content from a wider variety of writers and organizations thanks to it. Not everyone has the opportunity to publish through commercial channels.

Grey literature encompasses a broad and expanding spectrum of content. Not all researchers will find relevance in all of these examples. For instance, data on clinical trials is largely relevant to health and medical research. Information from corporate and market research will be very helpful to business researchers.

Grey literature typology

Types of content that we can describe as grey literature is (Snook, 2023):

• Blogs
• Clinical trials
• Company Information
• Conference papers/proceedings
• Datasets
• Discussion Forums
• Dissertations and theses
• Email discussion lists
• Government documents and reports
• Interviews
• Market reports
• Newsletters
• Pamphlets
• Patents
• Policy statements
• Pre-print articles
• Press releases
• Research reports
• Statistical Reports
• Survey results
• Tweets
• Wikis
• Working papers

arXiv.org

The easiest way of accessing a bigger volume of grey literature is through many dedicated databases such as arXiv.org, worldcat.org, opengrey.eu, nusl.cz, and more.

To take a closer look at accessing grey literature, we will examine arXiv.org.

ArXiv is a free distribution service and an open-access archive for 2,250,224 scholarly articles in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics. Materials on this site are not peer-reviewed by arXiv. (arXiv.org, 2023)

ArXiv offers researchers a broad range of services: article submission, compilation, production, retrieval, search and discovery, web distribution for human readers, and API access for machines, together with content curation and preservation. Our emphasis on openness, collaboration, and scholarship provides the strong foundation on which arXiv thrives. (arXiv.org, 2023)

ArXiv currently serves the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics. arXiv was founded by Paul Ginsparg in 1991 and is now maintained and operated by Cornell Tech. (arXiv.org, 2023)

Operations are maintained by the arXiv Leadership team and arXiv staff at Cornell. Governance of arXiv is led by the Leadership Team with guidance from the arXiv Scientific Advisory Board and the arXiv Member Advisory Board. arXiv is a community-supported resource funded by Cornell University, the Simons foundation, member institutions, and donors. (arXiv.org, 2023)

Registered users may submit articles to be announced by arXiv. There are no fees or costs for article submission. Submissions to arXiv are subject to a moderation process that classifies a material as topical to the subject area and checks for scholarly value. Material is not peer-reviewed by arXiv – the contents of arXiv submissions are wholly the responsibility of the submitter and are presented “as is” without any warranty or guarantee. By hosting works and other materials on this site, arXiv, Cornell University, and their agents do not in any way convey implied approval of the assumptions, methods, results, or conclusions of the work. (info.arxiv.org, 2023)

Search options

Simple Search: The Simple Search box on the ArXiv homepage allows users to enter keywords or phrases related to their research interests. It performs a basic search across the entire ArXiv dataset and displays results based on relevance. Users can choose between “Show abstracts” and “Hide abstracts” before querying in the search.

Advanced Search: The Advanced Search option provides more specific search capabilities. It allows users to refine their search by specifying fields such as author, title, abstract, categories, comments, journal reference, ACM classification, MSC classification, DOI, and more. Users are provided with a date option to choose a timeframe in which the papers have been released. Advanced Search supports Boolean operators, wildcard characters, and proximity searches to further customize search queries.

Subject Categories: ArXiv organizes its papers into subject categories to facilitate browsing and searching within specific research areas. Users can explore papers in various categories such as physics, computer science, mathematics, and more. Each category has subcategories, allowing users to narrow down their focus.

ArXiv API: ArXiv offers an Application Programming Interface (API) that allows developers to access ArXiv data programmatically. The API provides more advanced search capabilities, enabling users to retrieve and analyze specific subsets of the dataset based on their requirements.

Syntax rules and operators

Arxiv uses the following syntax rules and operators (arXiv.org, 2023) :

• Wildcards:
  • Use ? to replace a single character or * to replace any number of characters. Can be used in any field, but not in the first character position.
• Boolean operators:
  • There are three possible Boolean operators: “AND”, “OR” and “AND NOT” (“ANDNOT” when using the API) (arXiv.org, 2023)
• Expressions:
  • TeX expressions can be searched, enclosed in single $ characters.
• Phrases:
  • Enclose phrases in double quotes for exact matches in the title, abstract, and comments.
• Dates:
  • Sorting by announcement date will use the year and month the original version (v1) of the paper was announced. Sorting by submission date will use the year, month, and day the latest version of the paper was submitted.
• Multiple author names: 
  • Separate individuals with a ; (semicolon). Example: Jin, D S; Ye, J
• Journal References:
  • If a journal reference search contains a wildcard, matches will be made using wildcard matching as expected. For example, math* will match math, maths, mathematics.
  • If a journal reference search does not contain a wildcard, only the exact phrases entered will be matched. For example, math would match math or math and science but not maths or mathematics.
  • All journal reference searches that do not contain a wildcard are literal searches: a search for Physica A will match all papers with journal references containing Physica A, but a search for Physica A, 245 (1997) 181 will only return the paper with journal reference Physica A, 245 (1997) 181.

Structure of the system

The extensive library of preprints on ArXiv.org can be grouped into categories and navigated using a hierarchical structure. Here is an in-depth breakdown of ArXiv.org’s structure:

Main Categories: There are several major subject categories in ArXiv.org, each of which represents a broad academic discipline. Physics, mathematics, computer science, quantitative biology, quantitative finance, and statistics make up the major categories. These categories act as broad divisions to classify papers according to their main areas of study.

Subcategories: There are additional subcategories that further categorize papers within each primary category. For instance, there are numerous subcategories under the physics category, including astrophysics, high-energy physics, condensed matter, quantum physics, and others. Each subcategory corresponds to a certain region or subfield within the overall topic.

Sections: Subcategories are further organized into sections. Sections offer a more detailed categorization of articles based on particular subjects or fields of study. You might find parts like astrophysics of galaxies, cosmology and N-body simulations, solar and stellar astrophysics, etc. within the astrophysics subcategory, for instance. Within a subcategory, sections aid users in navigating to more specialized study areas.

Papers: The main component of ArXiv.org is the papers themselves. Each paper is a preprint that a particular author or group of writers has uploaded. The title, authors, abstract, and primary body of the research is often included in PDF-format papers. They might also contain addenda, figures, and references.

Versioning: Authors can upload several copies of a paper to ArXiv in order to update or revise their work. Each version is given a special identification number in the format “arXiv:YYMM.XXXX” (where YYMM stands for the submission year and month). Versioning makes it easier for the scientific community to discuss and provide input on papers as they change over time.

Search and discovery: The search feature on ArXiv.org enables users to look for articles using a variety of parameters, including keywords, authors, titles, and other search terms. Users can also look through articles in particular sections, subsections, or categories to find research on topics that interest them.

Data and information typology

1. Preprint Data: Preprints are copies of scientific articles that authors have submitted to ArXiv.org before they have undergone formal peer review. They are early-stage research discoveries.
2. Metadata: Every preprint in the repository of ArXiv.org has its own set of metadata. Title, author(s), abstract, publication date, subject categories, version history, and unique identifiers are just a few of the details that make up metadata. This metadata offers descriptive details about the papers and makes it easier to search for, find, and organize the content.
3. Full-Text Content: The preprints that are hosted on ArXiv.org are fully searchable. Each preprint’s whole material, which normally includes the primary study section, figures, tables, and references, is accessible to users. Users may read and thoroughly analyze the research findings thanks to the full-text material.
4. Author Information: ArXiv.org also includes information about the authors of the preprints. Their names, affiliations, contact details, and links to their profiles or other websites may all be included in this. Author information validates the work’s reputation and skill while also helping to provide context.
5. Categorization and Classification: As already mentioned above, preprints on ArXiv.org are divided into subject categories, subcategories, and sections using a hierarchical categorization scheme. Users can browse and search for publications within particular fields or study areas using this categorization. The category helps with organizing the content and offers a methodical way to browse the repository.
6. Versioning Data: ArXiv.org maintains versioning data for each preprint that has undergone updates or revisions. Versioning data includes information about the different versions of a paper, such as submission dates, changes made between versions, and unique identifiers for each version. This allows users to track the evolution of a research paper over time.
7. Usage Statistics: ArXiv.org collects and maintains usage statistics that provide insights into the usage patterns, popularity, and impact of the preprints hosted on the platform. Usage statistics include the number of downloads, views, citations, and other metrics that reflect the engagement and dissemination of the research.

Data availability and licensing

As an academic content repository, arXiv maintains a permanent record of each article and version published. Anyone can see and download any article on arXiv.org for free.

ArXiv requires submission authors’ consent before posting and disseminating their work in order to preserve this scientific record. The submitter must attest that they have the authority to grant the license they have chosen as the means of granting permission.

Below is a list of the various licensing types that are offered. With the exception of CC Zero, the original copyright owner continues to keep ownership of all of the licenses listed here after posting on arXiv.

Available Licences (info.arxiv.org, 2023):

• CC BY: Creative Commons Attribution
  • This license allows reusers to distribute, remix, adapt, and build upon the material in any medium or format, so long as attribution is given to the creator. The license allows for commercial use.
• CC BY-SA: Creative Commons Attribution-ShareAlike
  • This license allows reusers to distribute, remix, adapt, and build upon the material in any medium or format, so long as attribution is given to the creator. The license allows for commercial use. If you remix, adapt, or build upon the material, you must license the modified material under identical terms.
• CC BY-NC-SA: Creative Commons Attribution-Noncommercial-ShareAlike
  • This license allows reusers to distribute, remix, adapt, and build upon the material in any medium or format for noncommercial purposes only, and only so long as attribution is given to the creator. If you remix, adapt, or build upon the material, you must license the modified material under identical terms.
• CC BY-NC-ND: Creative Commons Attribution-Noncommercial-NoDerivatives
  • This license allows reusers to copy and distribute the material in any medium or format in unadapted form only, for noncommercial purposes only, and only so long as attribution is given to the creator.
• arXiv.org perpetual, non-exclusive license
  • This license gives limited rights to arXiv to distribute the article, and also limits re-use of any type from other entities or individuals.
• CC Zero
  • CC Zero is a public dedication tool, which allows creators to give up their copyright and put their works into the worldwide public domain. CC0 allows reusers to distribute, remix, adapt, and build upon the material in any medium or format, with no conditions.

All metadata falls under the Creative Commons CC0 1.0 Universal Public Domain Dedication

Indexing

1. Paper Submission: Research articles are uploaded using the submission procedure at ArXiv.org by their authors. The title, authors, abstract, and primary body of the research are all included in the papers, which are frequently in PDF format.
2. Initial Screening: When papers are submitted, they first go through a screening procedure where they are checked for compliance with the submission rules and regulations of ArXiv. The screening makes sure that the papers adhere to the fundamental standards for repository inclusion.
3. Categorization: The papers are divided into subject areas based on the main area of study after the initial screening. ArXiv’s core categories, subcategories, and sections all have hierarchical structures that group papers into relevant academic disciplines and themes.
4. Metadata Extraction: On the articles that have been submitted, metadata extraction is done to extract pertinent data such the title, author(s), abstract, publication date, and subject category. The metadata assists with repository organization and search by providing descriptive information about the publications.
5. Version Control: Authors can post updated versions of their publications to ArXiv to update or improve their research. Users can obtain the most recent version of a paper or follow its development over time thanks to the indexing process, which makes sure that every version is correctly linked and identified.
6. Search Indexing: The papers that have been indexed are added to ArXiv.org’s search index. In order to accomplish this, an index must be created that enables effective search and retrieval based on a variety of factors, including keywords, authors, titles, categories, and more. New submissions and version upgrades are continuously added to the search index.
7. Access and Retrieval: Once indexed, the papers are accessible to users through the ArXiv website. Users can search for papers, explore categories and subcategories, and navigate the hierarchical structure to discover and retrieve relevant preprints within their areas of interest.

Usage in the context of competitive intelligence

Businesses can utilize competitive intelligence by leveraging ArXiv.org in the following ways:

1. Stay Updated on Emerging Technologies: ArXiv.org provides access to cutting-edge research in various scientific disciplines. Businesses can monitor ArXiv to stay informed about emerging technologies, methodologies, and breakthroughs relevant to their industry. By tracking the latest research trends, businesses can identify potential opportunities for innovation, anticipate market changes, and stay ahead of competitors.
2. Monitor Competitor Activity: ArXiv.org allows businesses to track the research output of their competitors. By monitoring the preprints uploaded by competitor organizations or researchers, businesses can gain insights into their ongoing research projects, areas of focus, and potential technological advancements. This information can help businesses assess the competitiveness of their offerings, identify gaps, and inform their own research and development strategies.
3. Identify Collaborative Opportunities: ArXiv.org serves as a platform for researchers from different institutions and organizations to collaborate and share their work. By exploring the preprints and identifying researchers working on relevant topics, businesses can identify potential collaborative opportunities. Collaborating with researchers can provide access to expertise, facilitate technology transfer, and drive innovation through joint projects.
4. Track Intellectual Property Landscape: ArXiv.org can be used to monitor the intellectual property landscape in a particular field. By analyzing the preprints, businesses can identify potential inventions, novel methodologies, or emerging technologies that could impact their industry. This information can inform intellectual property strategies, including patent filing or licensing opportunities.
5. Inform Strategic Decision-Making: The insights gained from ArXiv.org can help inform strategic decision-making within businesses. By understanding the latest research and technological advancements, businesses can assess the potential impact on their products, services, or business models. This information can influence investment decisions, R&D priorities, market positioning, and overall competitive strategies.

It’s important to note that ArXiv.org primarily focuses on research preprints, which may not always reflect the final published research or commercial applications. Therefore, businesses should exercise caution and verify the status of research before making critical decisions based on preprint content from ArXiv.org.

Summary

Grey literature is a strong tool that can give a company the competitive edge to overtake the market or mislead it into making deadly mistakes. Using it as a basis for your competitive intelligence is a decision worth considering as long as you can stay responsible and put time and effort into understanding.

References

Snook, L. (2023). LibGuides: Grey Literature: What is Grey Literature? Available on: https://libguides.exeter.ac.uk/c.php?g=670055&p=4756572

info.arxiv.org. (2023). License and copyright—ArXiv info. Available on: https://info.arxiv.org/help/license/index.html

info.arxiv.org. (2023). About arXiv—ArXiv info. Available on: https://info.arxiv.org/about/index.html

arXiv.org. (2023). ArXiv.org e-Print archive. Available on: https://arxiv.org/

arXiv.org. (2023). ArXiv API User’s Manual. Available on: https://info.arxiv.org/help/api/user-manual.html