Data Requirements and Validation

1. Introduction

The R package artfishr implements tools for producing catch and effort estimates from sample-based survey data from small-scales fisheries data following the FAO ARTFISH extrapolation methodology.
To ensure consistency and interoperability, input datasets must comply with standardized formats and validation rules.

This vignette describes:

• the required data formats used in artfishr,
• the sample data and format specifications provided with the package,
• how to generate empty templates for each input data,
• and how to validate input data.

Four core datasets are supported:

1. Active vessels: Number of active vessels per stratum (vessels that can potentially go fishing)
2. Effort (two variants depending on data collection method): fisher interview or boat counting
   
3. Active days: number of days in the month with a fishing activity
4. Landings: landings survey observation data

2. Overview of ARTFISH data formats

Each dataset type follows a standardized structure defined by a JSON specification file.
These schemas describe the column names, units, and validation rules.

Each example file can be accessed directly via system.file():

system.file("extdata/samples/active_vessels.csv", package = "artfishr")

Preview the structure:

readr::read_csv(system.file("extdata/samples/active_vessels.csv", package = "artfishr")) |>  head()

The specification file describes each field and the required data structure.

3. Identify your sampling strategy for calculate the effort

The effort is obtained by multiplying the total number of active vessels by the number of actives days and the activity coefficient. The number of active vessels is given by the frama survey (census of vessel )or by the vessels or fishing licenses registries (when reliable). The number of active days is the number of days in the month with a fishing activity. To calculate the activity coefficient there are two different survey types in the method described by Stamatopoulos that can be used (boat counting or fisher interview). One or the other method must be chosen during the data collection design.

The first one consist in counting the number of boat that have been out on sampled days. The second consist in interviewing the fisher to record the number of days with a fishing activity during the last reference period (7 days for instance). In both method, we calculate the activity coefficient which the probability that a boat have been at sea on a certain day.

The two distinct monitoring approaches are:

1. Fisher interviews (effort_source = "fisher_interview")
   • Data obtained directly from fishers with questionnaires with the question “How many days have you been fishing duting the last x days?” (x is the reference period in days)
   • The template effort_fisher_interview.csv is required (number of days fished within the period of reference) corresponding to the tab B1. Effort (fisher interviews) of the Excel template
   • In this case, the active days survey is not necessary. Total number of days in the month is used by default as the question already included this component. active_day is NULL then and will automatically generated by the system.
2. Boat counting (effort_source = "boat_counting")
   • Data is obtained by direct observation on sampled days. Data collectors count how many boats per fishing unit went at sea on sampled day.
   • Template effort_boat_counting.csv is required (number of boats out in comparison of total potentially active boat) corresponding to the tab B2. Effort (boat counting) of the Excel template
   • The number of active days in the month must be provided. It can come from the active days survey or from estimation based on fisheries knowledge.

Both formats are supported by artfishr and can be validated using the same validation interface.

4. Creating empty data templates

artfishr allows users to programmatically generate empty templates for each dataset, based on their JSON specification.

The package provides a generic method create_artfish_template() to generate templates using the dataset names listed above.

create_artfish_template("artfish_A_active_vessels")

You can also export the template for use with the argument save_as:

create_artfish_template("artfish_A_active_vessels",save_as="active_vessels.csv")

Or use dedicated functions by dataset :

create_active_vessels_template()
create_effort_template(effort_source ="boat_counting") # or "fisher_interview"
create_landings_template(save_as="landings.csv")
create_active_days_template()

5. Validating input datasets

artfishr includes validation utilities to ensure all datasets comply with the expected structure and logical rules.

A global validator is provided:

validate_input_datasets(
  active_vessels,
  effort,
  effort_source = c("fisher_interview", "boat_counting"),
  active_days = NULL,
  landings
)

This function: - checks required fields for each dataset and flags missing or unexpected columns, - checks that field values satisfy the validation rules, - and ensures consistency between them.

Example:

# Load sample datasets
active_vessels <- readr::read_csv(system.file("extdata/samples/active_vessels.csv", package = "artfishr"))
effort <- readr::read_csv(system.file("extdata/samples/effort_fisher_interview.csv", package = "artfishr"))
landings <- readr::read_csv(system.file("extdata/samples/landings.csv", package = "artfishr"))

# Validate all datasets
validate_input_datasets(
  active_vessels = active_vessels,
  effort = effort,
  effort_source = "fisher_interview",
  landings = landings
)

The function returns a structured report with validation results and messages.

artfishr also includes specific validators to allow running checks per dataset type.

# Load sample datasets
active_vessels <- readr::read_csv(system.file("extdata/samples/active_vessels.csv", package = "artfishr"))
effort <- readr::read_csv(system.file("extdata/samples/effort_boat_counting.csv", package = "artfishr"))
landings <- readr::read_csv(system.file("extdata/samples/landings.csv", package = "artfishr"))
active_days <- readr::read_csv(system.file("extdata/samples/active_days.csv", package = "artfishr"))

validate_active_vessels_template(active_vessels)
validate_effort_template(effort,effort_source ="boat_counting") # or "fisher_interview"
validate_landings_template(landings)
validate_active_days_template(active_days)

6. Summary and recommendations

• Use the provided JSON specs and templates to ensure data consistency.
  
• Always validate your datasets before running analyses.
  
• Choose the appropriate effort survey format according to your monitoring method.
  
• Keep datasets synchronized on mandatory fields (e.g., year, fishing unit, landing site).
  
• Extend or customize validation rules where necessary for national contexts.

7. References

de Graaf, G. J., Nunoo, F., Ofori-Danson, P., Wiafe, G., Lamptey, E., & Bannerman, P. (2015). International training course in fisheries statistics and data collection. FAO Fisheries and Aquaculture Circular No. 1091. Rome: Food and Agriculture Organization of the United Nations. 134 pp. Available at: https://www.fao.org/3/i3639e/i3639e.pdf

Stamatopoulos, C. (2002). Sample based fishery surveys: A technical handbook. FAO Fisheries Technical Paper No. 425. Rome: Food and Agriculture Organization of the United Nations. 132 pp. Available at: https://www.fao.org/3/y2790e/y2790e.pdf