Data gaps are a common topic of discussion when talking about GBIF. Taxonomic, geographic and temporal gaps in the global dataset exist, yet we still have problems quantifying those gaps and measuring how well we are doing in filling those gaps. GBIF already provides analytics and country reports both of which provide useful information on data publishing activities but give little signal to what extent these data publishing activities are filling data gaps. With that in mind, we have started to think about how we can provide more nuanced analytics that quantify gaps and identify where priorities for data mobilisation might start.

Explore our proposal

If you are new here, this post is the third installment in a series about modelling and sharing survey and monitoring (SAM) data on GBIF. My name is Marie, I am a Data Administrator at the GBIF Secretariat, and I am learning how to work with SAM data. In my quest to improve my SAM data literacy skills, I am accompanied by the wise Kate Ingenloff, our SAM data modelling expert as well as Sam the Secretary Bird, our silly mascot.

The newest version of rgbif (3.9.0 - planned release to CRAN the end of September) introduces breaking changes because the default taxonomy has changed from the GBIF Backbone Taxonomy to the COL (Catalogue of Life) Extended Release. Similarly, pygbif v0.7.0 is also making the same transition to COL XR (see details here). To test out the latest version of rgbif before it is released to CRAN, you can download the development version:

Welcome to the second installment of our series on publishing Survey and Monitoring (“SAM”) data to GBIF where I’m simultaneously learning how to do it and teaching you at the same time. I am still supervised by Kate Ingenloff, our survey data mapping expert. If you haven’t already, you can read our first post: https://data-blog.gbif.org/post/sam-introduction/ in which I introduce our mascot, Sam the Secretary bird. If you prefer learning the art of publishing SAM data without fluffy writing and doodles, I recommend reading Kate’s comprehensive guide here: https://doi.

Perhaps, like me, you don’t know much about publishing Survey and Monitoring data(“SAM data” ). Maybe you have heard about event cores and Humboldt extensions, and you aren’t quite sure how to fit data into these. Trust me, this is about to change! I am on a mission to learn everything there is to know about publishing SAM data and spread the word. Don’t worry, I am under the guidance of our survey data mapping expert, Kate Ingenloff (who also happens to be in the office next door), so we are in good hands.

Recently GBIF.org has updated the default taxonomy it uses to organize occurrence records to the Catalogue of Life Extended Release (COL XR). The GBIF backbone was last updated in 2023, and will not be updated again. We encourage users of our APIs to migrate towards using the COL XR, namely by switching the checklistKey parameter to checklistKey=7ddf754f-d193-4cc9-b351-99906754a03b.

NOTE: This is experimental, and the implementation may change. GBIF makes no guarantees about the availability or stability of this tool.

Rule-based annotations is an experimental tool that will allow users to mark certain occurrence data as suspicious. The main goal of the project is to facilitate data cleaning and user feedback to publishers.

As the business and financial sectors increasingly look towards integrating biodiversity within their sustainability frameworks, this overview outlines the different frameworks that exist, the sectors they belong to, and how the GBIF infrastructure and the data it provides can support these sectors.

With GBIF’s new SQL downloads feature, it is now possible to easily create useful custom metrics. In this post, I will create species accumulation curves for various countries/areas using GBIF SQL downloads and R.

GBIF now has a new repository, Community Metrics, for collecting input from the GBIF community on the creation of data products derived from GBIF-mediated data that can complement those already provided through GBIF’s data analytics. The goal of this work is to produce new or further develop existing metrics, indicators and time series (trend) data products to support decision making.

A species accumulation curve is a plot of the total cumulative running total species in a region. These curves are useful for assessing whether more sampling is likely to yield more knowledge of a region. For example, if the curve is increasing rapidly, it is likely that the region is undersampled for the group, and there could be data gaps. Conversely, if the curve is flattening out, it is likely that the region is well sampled for the group.

Google Earth Engine (GEE) is a cloud-based platform designed to process and analyze vast amounts of geospatial data. With access to a rich archive of remote sensing imagery, climate datasets, and geographic information, GEE empowers researchers, conservationists, and policymakers to address critical environmental and biodiversity challenges. This primer introduces GEE, its Python API, and how GBIF users can leverage these tools for biodiversity informatics.

In this tutorial, we explored how to integrate remote sensing data with biodiversity observations to analyze the relationships between environmental factors and species occurrences. Specifically, we focused on analyzing Land Surface Temperature (LST) over time and its relationship to observations of the Northern Cardinal (Cardinalis cardinalis), a well-known songbird species.

This blog post is a tutorial on how to upload collection descriptors in the Global Registry of Scientific Collections (GRSciColl). If you are someone working with a physical collection or work with people who do, you might be interested in making these collections findable. Ideally, the content of these collections would be digitized and made available online on relevant platforms like GBIF.org, iDigBio.org, ALA.org.au, etc. Sharing digital specimen records is a great way to ensure the discoverability of collection content.

GBIF has an experimental feature that allows users to download data from the GBIF database in SQL format. Contact helpdesk@gbif.org to request access. https://techdocs.gbif.org/en/data-use/api-sql-downloads

If your download can be formulated using the traditional predicate downloads, it is usually going to be faster to use regular downloads.

The experimental Occurrence SQL Download API allows users to query GBIF occurrences using SQL. In contrast to the Predicate Download API, the SQL API allows selection of the columns of interest and generation of summary views of GBIF data.

SQL downloads, like predicate downloads, require you to have a GBIF user account.

GBIF users are often interested in getting occurrences of only living species or “no extinct” species. This seemingly straightforward request is often difficult to fulfill in practice.

Currently, there is no occurrence level filter for removing extinct species from an occurrence search. Additionally, there is no authoritative global extinct species list available from GBIF. This article discusses the difficulties and complexities of working with extinct species and some possible solutions.

Since 2022, GBIF has been monitoring changes of occurrenceIDs in datasets to improve the stability of GBIF identifiers. We pause data ingestion when we detect more than half of occurrence records in the latest version have different occurrenceIDs from the previous version (live on GBIF.org). This identifier validation process automatically creates issues on GitHub and GBIF helpdesk will contact the publishers to verify the changes of occurrenceIDs. Our aim in this process is to minimize the changes of GBIF identifiers and to support the needs for citations and data linking.

This article is an adapted version of the rgbif article found here.

It is often useful to create a map from GBIF mediated occurrences. While it is always possible to take a screenshot of map on gbif.org, sometimes a publication quality map is needed.

Evolutionarily Distinct and Globally Endangered or EDGE species have few close relatives on the tree of life and are often extremely unusual in the way they look, live and behave, as well as in their genetic make-up. They represent a unique and irreplaceable part of the world’s natural heritage. After seeing an open call for projects that extend existing protected area networks for the conservation of EDGE species through the EDGE Protected Area and Conserved Area Fund, I wanted to see what data is currently available through GBIF that could help guide decisions on conservation financing.

In 2020 GBIF released a news item “New data-clustering feature aims to improve data quality and reveal cross-dataset connections.” Basically, we run an algorithm across datasets shared with GBIF to search for similarities in occurrences data fields such as location, identifiers and dates. Please read this blog by Marie Grosjean and Tim Robertson for more details on how it works. In general, we can identify linkages between specimens, DNA sequences and literature citations.

When data publishers publish checklists, they will use a a Darwin Core Archive Taxon Core. And although the taxon core terms are already described here, what exactly to put in which field can sometimes be confusing. And there is a lot to read, like here https://github.com/tdwg/tnc/issues/1 Here I am sharing a summary of an email conversation we had with some data publishers on Helpdesk concerning some of the Taxon Core fields.

Preparing a dataset to be shared on GBIF.org can be a daunting task and many publishers realize that not all their data fits in the Darwin Core standard (DwC) and extensions GBIF uses to structure, standardize and display biodiversity data. This blog post will cover what data fits in GBIF, give examples of data that does not fit in the current format of GBIF, and provide guidance to how you can share relevant data in a metadata-only dataset or through a third-party.

When publishers supply GBIF with a scientific name, this name is sometimes not found in the GBIF taxonomic backbone. In these cases, the occurrence record gets a data quality flag called taxon match higher rank. This means that GBIF was only able to match the name to a higher rank (genus, family, order …).

The World Checklist of Vascular Plants (WCVP): Fabaceae is a new GBIF mediated checklist that drastically increases the coverage of the family Fabaceae in the GBIF backbone.

As written about in a previous blog post, GBIF now has database snapshots of occurrence records on AWS. This allows users to access large tables of GBIF-mediated occurrence records from Amazon s3 remote storage. This access is free of charge.

NOTE: The content of this blog post is now maintained in the technical documentation. Please view this source for the most up to date information on the topic. Many data users may suspect they’ve discovered duplicated records in the GBIF index. You download data from GBIF, analyze them and realize that some records have the same date, scientific name, catalogue number and location but come from two different publishers or have slightly different attributes.

You are a data publisher of occurrence data through GBIF.org, care about your data quality, and wonder what to do about the issue flags that show up on your occurrences. You might have noticed a new type flag this year relating to collection and institution codes and identifiers. They are the result of our attempt at linking specimens records to our Registry of Scientific Collections (GRSciColl).

This a GBIF API beginners guide.

The GBIF API technical documentation might be a bit confusing if you have never used an API before. The goal of this guide is to introduce the GBIF API to a semi-technical user who may have never used an API before.

The purpose of the GBIF API is to give users access to GBIF databases in a safe way. The GBIF API also allows GBIF.org and rgbif to function.

During the first-ever virtual GBIF 2021 Global Nodes Meeting, GBIFS hosted a “game show”: a one-hour “battle of Nodes vs. helpdesk”. The not-so-hidden goal was to demonstrate some of the lesser known functionalities of GBIF.org through a fun, interactive session.

You’ve finished an analysis using GBIF-mediated data, you’re writing up your manuscript and checking all the references, but you’re unsure of how to cite GBIF. If you Google it, you’ll probably end up reading our citation guideslines and quickly realize that GBIF is all about DOIs. Datasets have their own DOIs and downloads of aggregated data also have their own DOIs.

But maybe you didn’t download data through the GBIF.org portal. Maybe you relied on an R package like rgbif or dismo that retrived data synchronously from the GBIF API? Maybe a grad student downloaded if for you? Maybe you accessed and analyzed the data using a cloud computing service, like Microsoft Azure or Amazon Web Services? In any case, which DOI do you cite if you don’t have one?

GBIF now has a snapshot of 1.3 billion occurrences_✝ records on Microsoft Azure.

It is hosted by the Microsoft AI for Earth program, which hosts geospatial datasets that are important to environmental sustainability and Earth science. Hosting is convenient because you could now use occurrences in combination with other environmental layers and not need to upload any of it to the Azure. You can read previous discussions about GBIF and cloud computing here. The main reason you would want to use cloud computing is to run big data queries that are slow or impractical on a local machine.

The Global Registry of Scientific Collections, also known as GRSciColl, has been available on GBIF.org since 2019 but it recently got some more attention when we connected it to GBIF occurrences. Now is the perfect time to share a bit of GRSciColl history and what we plan for its future. A brief history of GRSciColl First of all, here are a few facts about GrSciColl today, at the start of 2021:

Post was updated on April 20 2022 to accommodate changes to dwc:establishmentMeans vocabulary handling.

Here I present a checklist for filtering GBIF downloads.

In this guide, I will assume you are familar with R. This guide is also somewhat general, so your solution might differ. This guide is intended to give you a checklist of common things to look out for when post-processing GBIF downloads.

Today, we are talking about the GBIF Species API. Although you might not use it directly, you probably encountered it while using the GBIF web portal:

• Typing a scientific name in the GBIF Occurrence search.
• Seeing a “Taxon Match Fuzzy” flag.
• Using the Species Name matching tool.

This API is what allow us to navigate through the names available on GBIF. I will try to avoid repeating what you can already find in its documentation. Instead, I will attempt to give an overview and answer some questions that we received in the past.

Publishers share datasets, but also manage data quality. GBIF provides access to the use of biodiversity data, but also flags suspicious or missing content. Users use data, but also clean and remove records. Each play an important role in managing and improving data quality..

I was asked to prepare some statistics for the GBIF regional regional meetings being held virtually this year. This blog post is a companion for those meetings.

You can watch a video presentation of the preparation of these meetings here. The presentation of this blog post starts here.

• The North American virtual nodes meeting 2020 was on 5 - 6 May 2020
• The Europe and Central Asia-virtual nodes meeting 2020 was on 11 - 12 May 2020
• The Latin America and Caribbean virtual nodes meeting 2020 will be on 18 - 20 May 2020
• The Africa virtual nodes meeting 2020 will be on 10 - 12 June 2020

GBIF introduced a regional framework across the GBIF Network a little more than a decade ago, with groups based on clusters of national participants. Soon after the publication of Brooks et al. (2016), GBIF adopted their structure to provide a consistent approach to regional reporting and assessment processes map.

This post will cover countries/areas with some political status and not other GBIF Affiliates and Antarctica.

As you probably already know, GBIF.org doesn’t host any data. The system relies on each data provider making their data available online in a GBIF-supported format. It also relies on organization letting GBIF know where to find these data (in other words registering the data). But how to do just that? The good news is that there are several GBIF-compatible systems. They will export or make available the data for you in the correct format and several provide means to register them as datasets on GBIF.

GBIF is now processing occurrence licenses record-by-record.

**iNaturalist research-grade observations**

Previously all occurrence licenses defaulted to their dataset license (provided by the publisher).

Open online APIs are fantastic! You can use someone else’s infrastructure to create workflows, do research and create products without giving anything in return, except acknowledgement. But wait a minute! Why is everyone not using them? Why do we create our own data sources and suck up the costs in time and money? Not to mention the duplication of effort.

Frictionless data is about removing the friction in working with data through the creation of tools, standards, and best practices for publishing data using the Data Package standard, a containerization format for any kind of data. It offers specifications and software around data publication, transport and consumption. Similarly to Darwin Core, data resources are presented in CSV files while the data model is described in a JSON structure.

If you are a (first time) publisher on GBIF and you are trying to decide which type of dataset would best fit your data, this blogpost is for you. All the records shared on GBIF are organized into datasets. Each dataset is associated with some metadata describing its content (the classic “what, where, when, why, how”). The dataset’s content depends strongly on the dataset’s class. GBIF currently support four types of dataset:

Recently a user noticed that there were Asian Red Pandas (Ailuridae) occurring in North America, and wondered if someone had made a mistake. When an occurrence observation comes from a zoo or botanical garden, it is usually considered a living specimen, but when it comes from a museum it is usually called a preserved specimen. This label helps users remove records that they might not want, which come from zoos.

Finding and accessing data There is a lot of GBIF-mediated data available. More than 1.3 B occurrence records covering hundreds of thousands of species in all part of the worlds. All free, open and available at the touch of a button. Users can download data through the GBIF.org portal, via the GBIF API, or one of the third-party tools available for programmatic access, e.g. rgbif. If there is one area in which GBIF has been immensely successful, it’s making the data available to users.

This past week our informatics team has been updating the Backbone taxonomy on GBIF.org. This is a fairly disruptive process which sometimes involves massive taxonomic changes but DON’T PANIC. This update is a good thing. It means that some of the taxonomic issues reported have been addressed (see for example this issue concerning the Xylophagidae family) and that new species are now visible on GBIF. Plus, it gives me an excellent opportunity to talk about the GBIF backbone taxonomy and answer some of the questions you might have.

It is now possible to download up to 100,000 names on GBIF!

Until recently it was not possible to download occurrences for more than a few hundred species at the same time, but it is now possible to request more species names (up to 100,000 taxonkeys).

Citizen Science datasets on GBIF plotted with all other (gray) GBIF datasets (>100K occurrences). There are many citizen science datasets with millions of occurrences (eBird, (Swedish) Artportalen), and the top 3 datasets on GBIF are all citizen science datasets. But in terms of number of unique species, only iNaturalist competes with large museum datasets like Smithsonian NMNH.

It has been suggested that GBIF could make es50 maps similar to what organizations like OBIS are
already doing. I decided to make one for land animals (graph above). link to code

A checklist dataset is a catch-all term describing any dataset that contains primarily a list of taxonomic names. The lines between a checklist dataset and an occurrence dataset can be blurry.

As I mentioned in my previous post, a lot more sequence-based data has been made available on GBIF this past year. MGnify alone, published 295 datasets for a total of 13,285,109 occurrences. Even though most of these occurrences are Bacteria or Chromista, more than a million of them are animals and more than 300,000 are plants. So chances are, that even if you are not interested in bacteria, you might encounter sequence-based data on GBIF.

[Edit 2021-09-16]

Important: To find guidance on how to publish Sequence-based data on GBIF, please consult the following guide:

Andersson AF, Bissett A, Finstad AG, Fossøy F, Grosjean M, Hope M, Jeppesen TS, Kõljalg U, Lundin D, Nilsson RN, Prager M, Svenningsen C & Schigel D (2020) Publishing DNA-derived data through biodiversity data platforms. v1.0 Copenhagen: GBIF Secretariat. https://doi.org/10.35035/doc-vf1a-nr22.

[End edit 2021-09-16]

GBIF is trying to make it easier to share sequence-based data. In fact, this past year alone, we worked with UNITE to integrate species hypothesis for fungi and with EMBL-EBI to publish 295 metagenomics datasets.

Unfortunately, documentation is not as quick to follow. Although we have now an FAQ on the topic, I thought that anyone could use a blog post with some advice and examples.

Note that this blog post is not intended to be documentation. The information written is subject to change, feel free to leave comments if you have any question or suggestion.