FAIDARE merge requestshttps://forgemia.inra.fr/urgi-is/faidare/-/merge_requests2024-01-03T19:21:47+01:00https://forgemia.inra.fr/urgi-is/faidare/-/merge_requests/140Add test data for BrAPI V2, extracted and transformed from scratch2024-01-03T19:21:47+01:00Cyril PommierAdd test data for BrAPI V2, extracted and transformed from scratchCyril PommierCyril Pommierhttps://forgemia.inra.fr/urgi-is/faidare/-/merge_requests/137Prepare metadata files for SoftwareHeritage2023-07-28T11:28:25+02:00Raphaël FloresPrepare metadata files for SoftwareHeritageCodemeta file generated from <https://codemeta.github.io/codemeta-generator/>
Will be useful for adding into HAL & SoftwareHeritage.
See how to submit: <https://inria.hal.science/hal-01872189v2/document>Codemeta file generated from <https://codemeta.github.io/codemeta-generator/>
Will be useful for adding into HAL & SoftwareHeritage.
See how to submit: <https://inria.hal.science/hal-01872189v2/document>Cyril PommierCyril Pommierhttps://forgemia.inra.fr/urgi-is/faidare/-/merge_requests/75chore(deps): apply github dependabot security fixes2021-05-07T16:12:16+02:00Raphaël Floreschore(deps): apply github dependabot security fixes# Bump dependencies from github PR
Sources:
- https://github.com/elixir-europe/plant-faidare/pull/18
- https://github.com/elixir-europe/plant-faidare/pull/17
- https://github.com/elixir-europe/plant-faidare/pull/16
## primsjs
chore(d...# Bump dependencies from github PR
Sources:
- https://github.com/elixir-europe/plant-faidare/pull/18
- https://github.com/elixir-europe/plant-faidare/pull/17
- https://github.com/elixir-europe/plant-faidare/pull/16
## primsjs
chore(deps): bump prismjs from 1.20.0 to 1.23.0 in /frontend
Bumps [prismjs](https://github.com/PrismJS/prism) from 1.20.0 to 1.23.0.
- [Release notes](https://github.com/PrismJS/prism/releases)
- [Changelog](https://github.com/PrismJS/prism/blob/master/CHANGELOG.md)
- [Commits](https://github.com/PrismJS/prism/compare/v1.20.0...v1.23.0)
## y18n
chore(deps): bump y18n from 3.2.1 to 3.2.2 in /frontend
Bumps [y18n](https://github.com/yargs/y18n) from 3.2.1 to 3.2.2.
- [Release notes](https://github.com/yargs/y18n/releases)
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/y18n/commits)
## elliptic
chore(deps): bump elliptic from 6.5.3 to 6.5.4 in /frontend
Bumps [elliptic](https://github.com/indutny/elliptic) from 6.5.3 to 6.5.4.
- [Release notes](https://github.com/indutny/elliptic/releases)
- [Commits](https://github.com/indutny/elliptic/compare/v6.5.3...v6.5.4)
Signed-off-by: dependabot[bot] <support@github.com>Célia MichoteyCélia Michoteyhttps://forgemia.inra.fr/urgi-is/faidare/-/merge_requests/74Draft: Set .gitlab-ci.yml to enable or configure SAST2021-04-21T14:37:16+02:00Raphaël FloresDraft: Set .gitlab-ci.yml to enable or configure SASTSet .gitlab-ci.yml to enable or configure SAST security scanning using the GitLab managed template. You can [add variable overrides](https://docs.gitlab.com/ee/user/application_security/sast/#customizing-the-sast-settings) to customize S...Set .gitlab-ci.yml to enable or configure SAST security scanning using the GitLab managed template. You can [add variable overrides](https://docs.gitlab.com/ee/user/application_security/sast/#customizing-the-sast-settings) to customize SAST settings.Raphaël FloresRaphaël Floreshttps://forgemia.inra.fr/urgi-is/faidare/-/merge_requests/71Add FEM source2021-06-16T15:07:43+02:00Cyril PommierAdd FEM source@celia.michotey , could check you are alright with the dynamic mapping settiing ? We discussed it some months ago.@celia.michotey , could check you are alright with the dynamic mapping settiing ? We discussed it some months ago.Célia MichoteyCélia Michoteyhttps://forgemia.inra.fr/urgi-is/faidare/-/merge_requests/69Draft: Fix/fix email2022-09-22T16:38:15+02:00Maud MartyDraft: Fix/fix emailJérémy DestinJérémy Destinhttps://forgemia.inra.fr/urgi-is/faidare/-/merge_requests/44WIP: genealogy display2019-12-17T18:25:58+01:00Célia MichoteyWIP: genealogy displayJérémy DestinJérémy Destinhttps://forgemia.inra.fr/urgi-is/faidare/-/merge_requests/43Draft: Try to resolve redirection bug.2020-12-04T14:33:09+01:00Célia MichoteyDraft: Try to resolve redirection bug.Cyril PommierCyril Pommierhttps://forgemia.inra.fr/urgi-is/faidare/-/merge_requests/24WIP: Deep result window pagination (getting over the ES max result window)2020-02-21T16:50:33+01:00Guillaume CornutWIP: Deep result window pagination (getting over the ES max result window)For the moment we blocked the pagination of any API to be under the max result window of ElasticSearch (pageSize * page < 1000)
@raphael.flores @celia.michotey @cyril.pommier @jeremy.destin
In this branch, I propose a solution to impl...For the moment we blocked the pagination of any API to be under the max result window of ElasticSearch (pageSize * page < 1000)
@raphael.flores @celia.michotey @cyril.pommier @jeremy.destin
In this branch, I propose a solution to implement deep pagination on BrAPI calls. We've had problems with BrAPI client trying to harvest our big datasets, and we'd like to propose a reasonable solution without having to change all of the BrAPI.
The data discovery calls are for now still restricted on pagination in this branch (see the `@MaxResultWindowPagination` annotation on [DataDiscoveryCriteriaImpl.java](https://forgemia.inra.fr/urgi-is/gpds/merge_requests/new/diffs?merge_request%5Bsource_branch%5D=feat%2Fdeep_result_window_pagination&merge_request%5Bsource_project_id%5D=462&merge_request%5Btarget_branch%5D=master&merge_request%5Btarget_project_id%5D=462#0192afe5a2355a112bb3b3f1928c2db71dc958d8_16_17))
# Implementation
In order to fetch data over the max result window we can either use the scroll API or the search after API. The scroll API cannot be used to fetch pages in random access whereas the search after API can so I chose to implement deep pagination with the search after API.
The search after API requires the result to be sorted (here we use the sort on `_id` by default) and the previous page last document sort values.
The `ESGenericFindRepository` in charge of finding documents by criteria now detects when a request goes over the ES max result window and will:
- Get the previous page last sort values
- Get the current page using ES [search_after](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-search-after.html) filled with the last page, last sort values
When the requested page is under the ES max result window, the repository does a simple search with `size/from` pagination and do not use the `search_after`
To be efficient a cache is used (with a TTL of 1H) to store every page last sort values for every query.
The cache is structured as such: `QueryWithoutPagination(String) => Page{size,from} => LastSortValues(Object[])` (see [ESGenericFindRepository](https://forgemia.inra.fr/urgi-is/gpds/merge_requests/new/diffs?merge_request%5Bsource_branch%5D=feat%2Fdeep_result_window_pagination&merge_request%5Bsource_project_id%5D=462&merge_request%5Btarget_branch%5D=master&merge_request%5Btarget_project_id%5D=462#834da5f062cfc7fcdbbaf5a45c765fb75908862b_33_39)).
If the previous page for the current query is not in cache, the repository will paginate from the last page under the max result window to the current page and store each sort values in cache for future use. In classical scenarios, the BrAPI client will fetch a page for a given query/criteria, the repository will store this page's last document sort values and when the client request the next page, the repository has in cache everything it needs to fetch the current page.
# Remaining TODOs
- [ ] Performance cost estimation
- [ ] Verify memory/CPU consumption for both web server and ES cluster
- [ ] Verify that the performance of document search under the max result window is the same as or close to the code in `master`
- [ ] Verify that the performance of document search over the max result window is reasonable
- [ ] Add extensive repository unit tests with mocks (and maybe even functional tests with real accessing the ES cluster?)
- [ ] Validate or not this approachLater