README.md 9.38 KB
Newer Older
1
2
# Rare project - Data discovery

3
4
5
6
7
8
9
10
11
12
13
14
15
16
- [Rare project - Data discovery](#rare-project---data-discovery)
  - [Contribute](#contribute)
    - [Data](#data)
    - [Code](#code)
  - [Setup](#setup)
    - [Backend](#backend)
    - [Frontend](#frontend)
  - [Build](#build)
  - [CI](#ci)
  - [Documentation](#documentation)
  - [Harvest](#harvest)
  - [Indices and aliases](#indices-and-aliases)
  - [Spring Cloud config](#spring-cloud-config)
  - [Building other apps](#building-other-apps)
17
18
## Contribute

19
If you want to install the program locally, it's cool, just keep reading at [Setup section and beyond](#setup).
20
21
22

### Data

23
You might probably want to know how to contribute to the federation of data. That's great, let's have a look at the [WheatIS/Plant guide](./HOW-TO-JOIN-WHEATIS-AND-PLANT-FEDERATIONS.md) or the [RARe guide](./HOW-TO-JOIN-RARe-FEDERATION.md) to know how to.
24

25
26
27
### Code

If you do want to contribute to code, that's great also, have a look at [CONTRIBUTING.md](./CONTRIBUTING.MD).
28

29
30
31
32
## Setup

### Backend

33
The project uses Spring (5.x) for the backend, with Spring Boot.
34
35
36
37
38

You need to install:

- a recent enough JDK8

39
The application expects to connect on an Elasticsearch instance running on `http://127.0.0.1:9200`.
Exbrayat Cédric's avatar
Exbrayat Cédric committed
40
41
42
43
To have such an instance, simply run:

    docker-compose up

44
And this will start Elasticsearch and a Kibana instance (allowing to explore the data on http://localhost:5601).
Exbrayat Cédric's avatar
Exbrayat Cédric committed
45

46
47
48
Then at the root of the application, run `./gradlew build` to download the dependencies.
Then run `./gradlew bootRun` to start the app.

49
You can stop the Elasticsearch and Kibana instances by running:
Exbrayat Cédric's avatar
Exbrayat Cédric committed
50
51

    docker-compose stop
52
53

or run:
54
55

    docker-compose down 
56

57
to also remove the stopped containers as well as any networks that were created.
Exbrayat Cédric's avatar
Exbrayat Cédric committed
58

59
60
### Frontend

61
The project uses Angular (8.x) for the frontend, with the Angular CLI.
62
63
64

You need to install:

65
- a recent enough NodeJS, v10+ is required for Angular 8.
66
67
68
69
70
- Yarn as a package manager (see [here to install](https://yarnpkg.com/en/docs/install))

Then in the `frontend` directory, run `yarn` to download the dependencies.
Then run `yarn start` to start the app, using the proxy conf to reroute calls to `/api` to the backend.

Raphaël Flores's avatar
Raphaël Flores committed
71
The application will be available on:
72
73
74

- <http://localhost:4000/rare-dev> for RARe (runs with: `yarn start:rare` or simply `yarn start`)
- <http://localhost:4100/wheatis-dev> for WheatIS (runs with: `yarn start:wheatis`)
75
- <http://localhost:4200/data-discovery-dev> for DataDiscovery (runs with: `yarn start:data-discovery`)
76
77

See [./frontend/package.json (scripts section)](./frontend/package.json) for other yarn commands.
78
79
80
81
82
83

## Build

To build the app, just run:

    ./gradlew assemble
84
85
86

or

Raphaël Flores's avatar
Raphaël Flores committed
87
    ./gradlew assemble -Papp=wheatis
88

89
90
91
92
93
94

which is a shortcut for


   ./gradlew assemble -Papp=wheatis && java -jar path/to/wheatis.jar

95
This will build a standalone jar at `backend/build/libs/`, that you can run with either:
96
97

    java -jar backend/build/libs/rare.jar
Raphaël Flores's avatar
Raphaël Flores committed
98
    java -jar backend/build/libs/wheatis.jar
99
    java -jar backend/build/libs/data-discovery.jar
Raphaël Flores's avatar
Raphaël Flores committed
100

101
And the full app running on:
102

103
104
105
- <http://localhost:8080/rare-dev>
- <http://localhost:8180/wheatis-dev>
- <http://localhost:8280/data-discovery-dev>
106
107
108
109
110

## CI

The `.gitlab-ci.yml` file describes how Gitlab is running the CI jobs.

111
112
113
It uses a base docker image named `urgi/docker-browsers`
available on [DockerHub](https://hub.docker.com/r/urgi/docker-browsers/)
and [INRA-MIA Gitlab](https://forgemia.inra.fr/urgi-is/docker-rare).
Raphaël Flores's avatar
Raphaël Flores committed
114
The image is based on `openjdk:8` and adds all stuff needed to run the tests
115
(ie. a Chrome binary with a headless Chrome in `--no-sandbox` mode).
116
117
118
119
120
121

We install `node` and `yarn` in `/tmp` (this is not the case for local builds)
to avoid symbolic links issues on Docker.

You can approximate what runs on CI by executing:

122
    docker run --rm -v "$PWD":/home/rare -w /home/rare urgi/docker-browsers ./gradlew build
Jean-Baptiste Nizet's avatar
Jean-Baptiste Nizet committed
123

124
125
126
127
128
129
130
131
Or also run a gitlab-runner as Gitlab-CI would do (minus the environment variables and caching system):

    gitlab-runner exec docker test

## Documentation

An API documentation describing most of the webservices can be generated using the
build task `asciidoctor`, which executes tests and generates documentation based on snippets generated
132
by these tests. The documentation is generated in the folder `backend/build/asciidoc/html5/index.html`
133
134
135

    ./gradlew asciidoctor

Jean-Baptiste Nizet's avatar
Jean-Baptiste Nizet committed
136
137
## Harvest

138
Harvesting (i.e. importing JSON documents into Elasticsearch) consists in creating the necessary index and aliases and Elasticsearch templates.
Jean-Baptiste Nizet's avatar
Jean-Baptiste Nizet committed
139

Raphaël Flores's avatar
Raphaël Flores committed
140
To create the index and its aliases execute the script below for local dev environment:
141

142
    ./scripts/index.sh -app rare|wheat|data-discovery --local
Jean-Baptiste Nizet's avatar
Jean-Baptiste Nizet committed
143

144
The -app parameter will trigger a harvest of the resources stored in the Git LFS directories `data/rare`, `data/wheatis` and `data/data-discovery` respectively.
Raphaël Flores's avatar
Raphaël Flores committed
145

146
147
## Indices and aliases

148
The application uses several physical indices:
149

150
151
- one to store physical resources, containing the main content
- one to store suggestions, use for the search type-ahead feature only
Raphaël Flores's avatar
Raphaël Flores committed
152
153

Both indices must be created explicitly before using the application. If not, requests to the web services will return errors.
154

155
Each index and alias below refers to `rare` application in `dev` environment, the equivalent shall be created for `wheatis` and `data-discovery` app in `dev` environment as same as in `beta` or `staging` or `prod` environments. For brevity, only `rare-dev` is explained here.
Raphaël Flores's avatar
Raphaël Flores committed
156
157
{: .alert .alert-info}

158
159
160
161
162
163
164
165
The application doesn't use the physical resources index directly. Instead, it uses two aliases, that must be created before using the application:

- `rare-dev-resource-alias` is the alias used by the application to store and search for documents
- `rare-dev-suggestions-alias` is the alias used by the application to store and search for suggestions, only used for completion service.

In normal operations, these two aliases should refer to physical indices having a timestamp such as `rare-dev-tmstp1579877133-suggestions` and `rare-dev-tmstp1579877133-resource-index`. Those timestamps allow for reindexing data without breaking another runnning application having another timestamp. The alias switch being done atomicly, we always see data in the web interface.

Using two aliases is useful when deleting obsolete documents. This is actually done by removing everything and then harvesting the new JSON files again, to re-populate the index from scratch.
166

167
168
## Spring Cloud config

169
On bootstrap, the application will try to connect to a remote Spring Cloud config server to fetch its configuration. The details of this remote server are filled in the `bootstrap.yml` file. By default, it tries to connect to the local server on <http://localhost:8888> but it can of course be changed, or even configured via the `SPRING_CONFIG_URI` environment variable.
170
171
172

It will try to fetch the configuration for the application name `rare`, and the default profile.
If such a configuration is not found, it will then fallback to the local `application.yml` properties.
173

174
175
176
To avoid running the Spring Cloud config server every time when developing the application,
all the properties are still available in `application.yml` even if they are configured on the remote Spring Cloud server as well.

177
If you want to use the Spring Cloud config app locally, see <https://forgemia.inra.fr/urgi-is/data-discovery-config>
178

179
The configuration is currently only read on startup, meaning the application has to be reboot if the configuration is changed on the Spring Cloud server. For a dynamic reload without restarting the application, see <http://cloud.spring.io/spring-cloud-static/Finchley.SR1/single/spring-cloud.html#refresh-scope>
180
to check what has to be changed.
181

182
In case of testing configuration from the config server, one may use a dedicated branch on `data-discovery-config` project and append the `--spring.cloud.config.label=<branch name to test>` parameter when starting the application's executable jar. More info on [how to pass a parameter to a Spring Boot app](https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html#boot-features-external-config).
183

184
185
186
187
188
189
190
191
192
## Building other apps

By default, the built application is RARe. But this project actually allows building other
applications (WheatIS, for the moment, but more could come).

To build a different app, specify an `app` property when building. For example, to assemble
the WheatIS app, run the following command

    ./gradlew assemble -Papp=wheatis
193

194
195
196
You can also run the backend WheatIS application using

    ./gradlew bootRun -Papp=wheatis
197

198
199
Adding this property has the following consequences:

200
201
202
203
- the generated jar file (in `backend/build/libs`) is named `wheatis.jar` instead of `rare.jar`;
- the Spring active profile in `bootstrap.yml` is `wheatis-app` instead of `rare-app`;
- the frontend application built and embedded inside the jar file is the WheatIS frontend application instead of the  RARe frontend application, i.e. the frontend command `yarn build:wheatis` is executed instead of the command `yarn:rare`.

204
205
Since the active Spring profile is different, all the properties specific to this profile
are applies. In particular:
206
207
208

- the context path of the application is `/wheatis-dev` instead of `/rare-dev`;
- the Elasticsearch prefix used for the index aliases is different.
209

Raphaël Flores's avatar
Raphaël Flores committed
210
See the `backend/src/main/resources/application.yml` file for details.
211
212

You can adapt the elasticsearch index used with the following parameter
213
214

    java -jar backend/build/libs/data-discovery.jar --data-discovery.elasticsearch-prefix="data-discovery-staging-"