ScopeMArchiver

An archiver service that allows uploading dataset and registering it with SciCat. It is built on Prefect.io to orchestrate the asynchronous jobs (called flows) to archive and retrieve datasets to/from the ETH LTS.

The full setup is containerized and requires a SciCat instance to run correctly.

Refer to the Github pages for more details.

Quick start

Build all the services:

docker compose --env-file .env.prod --env-file .env.development build

Starting up all services for development:

docker compose --env-file .env.prod --env-file .env.development up -d

Deploy Local Flows

For development and debugging, a local process can serve flows, for example by running python -m archiver.flows. However, some more configuration is required to fully integration with the other services; therefore a VS Code launch command Prefect Flows can be used in launch.json). This allows to debug flows locally and the registered flows have a prefix, DEV_.

Github Pages

The latest documentation can be found in the GitHub Pages

Bulding Github Pages

Github pages are built on mkdocs.

Build docker image first:

docker build . -f docs.Dockerfile -t ghcr.io/swissopenem/scopemarchiver-docs:latest

Start development server

docker run --rm -it -p 8000:8000 -v ${PWD}:/docs ghcr.io/swissopenem/scopemarchiver-docs:latest

Build documentation

docker run --rm -it -p 8000:8000 -v ${PWD}:/docs ghcr.io/swissopenem/scopemarchiver-docs:latest build

Deploy documentation to GitHub Pages

docker run --rm -it -v ~/.ssh:/root/.ssh -v ${PWD}:/docs ghcr.io/swissopenem/scopemarchiver-docs:latest gh-deploy