Record Cleaner

In May 2011, the National Biodiversity Network (NBN) Trust published a standalone piece of software called NBN Record Cleaner. This tool was designed to help both individual recorders and dataset managers (such as Local Environmental Record Centres and National Recording Schemes) to spot potential problems with biological records. The software flagged issues such as incorrect data formats, invalid species names, dates or grid references in incorrect formats, or species outside of their known geographic distributions or phenology periods.

Record Cleaner online has since been redeveloped by the UK Centre for Ecology and Hydrology (UKCEH) Biological Records Centre (BRC), building on the earlier NBN Record Cleaner software, and with support from Defra, via the Natural Capital and Ecosystem Assessment Programme, and from UKCEH’s National Capability for UK challenges funding. The new web-based version of Record Cleaner was developed to provide a replacement and to open up the option of making Record Cleaner more widely applicable to other recording systems that can access it via an API.

The Biological Recording Company was commissioned as part of the redevelopment project to engage the biological recording community with the Record Cleaner tool through webinars and to produce guidance for using the tool via a series of guidance videos and a publicly available user manual (all of which can be found within this article).

Checking & Validating Your Wildlife Observations

We’ve also packaged up all of the guidance into a free self-study on,ine course that can be accessed via our e-learning platform.

Register Your Free Space

Who is Record Cleaner For?

  • Any individual recorder who wishes to run checks on their own records, e.g. before adding them to iRecord, or before sending to a scheme or centre
  • Recording scheme organisers, records centre staff and anyone who receives biological records from others, and wishes to check if they contain anything unusual or unlikely
  • Being considered by NBN Atlas for checking incoming data that hasn’t already been checked.
  • Online recording platforms, such as iRecord.
  • Has the potential to help screen results from large-scale automated monitoring and DNA-derived data.

How Does Record Cleaner Work?

Record Cleaner is able to assist with both the validation and verification aspects of the biological record data flow pathway (see . This is a two-step process, with validation checks performed first and then verification checks undertaken second.

  • Data validation checks that data is in the correct standardised format (e.g. valid dates and grid references).
  • Data verification checks that data is true (i.e., was the stated species at the stated location on the stated date?).
Data-flow of a biological record, highlighting 7 processes.

The two guidance videos below explain how to prepare and upload biological record datasets for use in Record Cleaner.

Record Cleaner 1: Preparing Your Data
Record Cleaner 2: Uploading Your Data

Validation

During the validation stage, the format of specific fields is checked against a set of built-in rules. This includes checking the following:

  • Taxonomic identifiers such as common names, scientific names or Taxon Version Keys (a unique code applied to each taxon) are checked against the UK Species Inventory.
  • Dates are checked against a range of accepted date formats (see the section titled Recorder Cleaner Guidance: Dates for more info).
  • Spatial references such as grid references or coordinates are checked against accepted formats.
  • Vice county numbers (optional) are checked against the list held for Great Britain and associated spatial data, flagging both incorrect vice county numbers and records where the spatial reference is not within the stated vice county.

Following the validation process, all records that failed one or more checks are highlighted to the user, with the recommendation that these be checked and corrected before proceeding to the verification stage.

The guidance video below explains how to use Record Cleaner to validate data in a biological records dataset.

Record Cleaner 3: Validating Your Data

Verifcation

The verification stage assesses your records against sets of rules provided by National Recording Schemes based on the species occurrence datasets that they hold for their taxa. Importantly, these rules do not objectively determine if a record is correct or incorrect, and they are never used to accept or reject a record. They simply aim to highlight the aspects of the record that may require further consideration.

Types of rules that exist within Record Cleaner include the following: 

  • Identification difficulty: A set of difficulty codes can be applied to taxon groups, usually ranging from 1 to 5. Each difficulty rating includes a description (e.g. ‘Voucher specimen should be retained. Confirmation by an acknowledged expert will be required in most cases’). ID difficulty rules do not cause records to pass or fail, but instead trigger a warning message about the ID difficulty.
  • Distribution: Records fail this rule for a given taxon if they occur in a 10 km grid square where the species has not been recorded previously. Records that fail the rule are noted as ‘close to’ if they fall within a new grid square that is adjacent to a grid square where there is a previous record. They are listed as ‘far from’ if the taxon has not been recorded from any adjacent squares.
  • Phenology: Records fail this rule if they fall outside of the annual activity periods for a given taxon. (e.g., the flight period of a butterfly). Phenology rules can be set to be associated with different life stages (e.g., having different ranges for adult butterflies and caterpillars). Records that fail the rule are noted as ‘close to’ if the record date is within 3 days of the known activity period and as ‘far from’ if they fall outside of this period.
  • Time Period: Rules can be set to flag records that precede the first date for a taxon (e.g., the date that a taxon is thought to have colonised the UK) or occur after the last known record (e.g., the date at which a previously known taxon is thought to have become extinct in the UK).
  • Additional rules: Some rules also trigger other warning messages, including that a taxon is rare, that a taxon always requires checking by a verifier due to rarity or taxonomic uncertainty, or that no rules exist for a specific taxon.

Many of the rules were designed for the original NBN Record Cleaner, and the data that informs the rules may be out-of-date. The UKCEH Biological Records Centre is working with National Recording Schemes to update the rules where possible. The current rule sets can be found on GitHub.

The guidance video below explains how to use Record Cleaner to run the verification checks based on National Recording Scheme rules on a biological records dataset in Record Cleaner.

Record Cleaner 4: Verifying Your Data

Coming soon

Guidance for Recording Schemes

The presentation below is aimed at National Recording Schemes and taxonomic specialists who are responsible for defining and overseeing data validation rules. Martin Harvey (Biological Records Centre) discusses developing and maintaining rules within Record Cleaner.