Developer Guide
We will go through the implementation details for Robustar. For environment setup issues, please follow the frontend setup guide, backend setup guide and launcher setup guide in our repository.
Robustar Architecture
Robustar's architecture is shown below.
Robustar is developed as a Web application wrapped in a docker image. We use Web stack mainly because of the following reasons:
- With Web GUI, we don't need to worry about cross platform issues.
- There is more abundant resources for developing good visual components with Web.
- With a separate frontend and backend, it is easier to refactor Robustar if we want to deploy it on the cloud in a SaaS manner in the future.
For now, users will need to pull the docker image and host the frontend and backend on their own machines. However, we hide the docker layer away with a launcher. They can simply download the launcher, click a few buttons on the GUI, and then use a browser to use Robustar. The launcher is implemented in Python with pyside2
(something similar to PyQT
). It can be compiled for multiple systems. We will talk about the launcher later.
The Frontend
Here is a list of the main stack we used for the frontend.
Library / Tool | Usage |
---|---|
HTML/CSS, JavaScript, Vue.js | Web basics & frontend framework |
Lerna | Monorepo management |
Vuetify | Visual component library |
eslint & prettier | Code Styling |
Axios | API client |
Cypress | Unit tests & end-to-end tests |
Besides, we adopted tui-image-editor
to suit our image annotation needs and placed it in our repository. Thus, we have two separate packages for the frontend: image-editor
package for tui-image-editor
and robustar
for the main frontend. robustar
package relies on image-editor
, and this is managed by lerna
.
The main Robustar interface is in /front-end/packages/robustar/src
. These are the most important folders:
views/
: Each file corresponds to a page that can be accessed with a different URL.components/
: Stores visual components, for example, model visualizers, that can be used in multiple pages.services/
: Stores all API call utilities.
For tui-image-editor
, please visit their GitHub page.
The Backend
Here are the main backend libraries we used:
Library / Tool | Usage |
---|---|
Flask | API server in Python |
python-socketio | Socket connection utility |
SQLite | Database |
Flashtorch | Model saliency map visualization |
pytorch_influence_functions | Influence function calculation |
Pillow | Python's image library |
PyTorch, torchvision, numpy, scipy | Machine learning pipeline utilities |
Tensorboard | Model training visualization |
pytest | Unit test |
The backend is organized as follows:
apis/
: All API endpoint definitionsml/
: All code related to the machine learning pipeline (e.g., dataset definition, influence calculation, trainer initialization, ...)modules/
: Independent modules (usually code adopted from other places)bg_remove/
: For auto-removing the background of images with a image segmentation modelinfluence_module/
: For influence function calculationsvisualize_module
: For saliency map visualization (withflashtorch
)objects/
: All important Python ClassesRAutoAnnotator
: A wrapper for image segmentation model for background removalRDataManager
: Represents the entire dataset (all splits + paired data). A aggregation ofRImageFolders
.RImageFolder
: Represents a data split. Supports data manipulation and interacts with the SQLite database.RModelWrapper
: A wrapper for models that the user is training / interacting with.RServer
: Represents the entire server instance. Stores useful global information.RTask
: Represents a task (e.g., model training, batch auto-annotation, testing...), which usually takes some time to finish. Can be started / updated / stopped.utils/
: Implementation of APIs.tests/
: Test cases.
Backend File System Interaction
The backend will read from / write to the local storage inside the container. Specifically:
- It will initialize a
SQLite
database if it does not exist. All metadata will be stored in the database. These include: - All data records
- All visualization records
- All influence records
- All correctly/incorrectly classified samples
- It will save model's checkpoint.
- It will save the images annotated by the user.
- It will save any pre-calculated results, such as visualization images and background removal proposals.
When starting the container with a launcher, users can mount these files from their file system into the container, so that any changes will be reflected in their file system. Inside the container, the backend will access these files with hard coded paths, which are available in RServer
object.
CI/CD & Tests
Backend tests are available in back-end/tests
. Frontend tests are available in front-end/cypress/tests/components
. Refer to frontend and backend setup guide in our repo to run these tests locally.
We use CircleCI
to perform CI/CD
tasks. The configuration can be found in .circleci/config.yml
. We do the following in the pipeline
- Pull code from GitHub
- Download dependencies and dev dataset
- Run all test cases
- Build docker image
- Push docker image to DockerHub
If the commit is made to the main branch, we will trigger a release build, which has a tag of the format x.y.z
. Otherwise, when the any commit is pushed to other branches, we trigger a dev build with tag x.y.<commit_hash>
.
Docker
We build a docker image in two steps. First, we will build a base image with /docker-base/DOCKERFILE
, which will include all necessary libraries and tools ready, except for everything related to PyTorch. Then, based on the base image, we then build the full image with /DOCKERFILE
. Note that we can supply an argument specifying the CUDA
version to build for, and the script /scripts/install_pytorch.sh
will install the corresponding version.
Launcher
Here are the main launcher libraries and development tools we used:
Library / Tool | Usage |
---|---|
PySide2 | Python’s QT library |
docker | Python’s Docker library |
Qt Designer | GUI QT development tool |
PyUIC | Tool that converts .ui files to .py files |
The launcher is developed adhering to Model–View–Controller(MVC) pattern and is organized as follows:
app.py
: Main file of the launcher.model/
: The launcher model. Each property is defined with the decoration of@property
and its corresponding setter function is defined with@<property name>.setter
.resources/
: All.ui
files developed withQt Designer
.views/
: All view files in pairs. Specifically,xx_view_ui.py
generated directly from a.ui
file withPyUIC
pairs withxx_view.py
, which connects signals in the view to corresponding controller functions.controllers/
: All controller classesmain_ctrl.py
: ContainsMainController
class, including functions controlling the general behavior of the launcher. Two main types of functions shall be distinguished, functions beginning withsetM
andsetV
. The former ones are functions called by views to change the model, and the latter ones are functions called by the model to change the view.docker_ctrl.py
: ContainsDockerController
class, including docker relevant operation functions.
To package the launcher, please follow the launcher setup guide in our repository.