Workflow

NeoPulse® AI Studio Command Line Interface (CLI) is used to manage the training and querying of deep learning models. The general workflow for training and querying of deep learning models can be seen in Figure 1. Either The NeoPulse® AI Studio Command Line Interface (CLI) or the exposed REST APIs can be used to train, query, cleanup, and export the AI models that you create for use with the NeoPulse® Query Runtime (NPQR).

Training can only be done using NeoPulse® AI Studio. The training workflow consists of four parts (see Figure 1). First, the data to be queried must be moved to the machine running either NAIS or NPQR, and a .csv file is created that contains only the input data. Second, an .nml script is written to define the constructs for the AI model. Third, this .nml script is compiled by NAIS, and the resulting model is trained on the data. Finally, once satisfactory results are obtained, the model can the be exported for use on an NPQR instance for production.

Training Workflow
Figure 1: High Level Training Workflow

The query workflow consists of three parts (see Figure 2). First, the data to be queried must be moved to the machine running either NAIS or NPQR, and a .csv file is created that contains only the input data. Second, a query job is submitted. This query job will be queued, and processed in the order received. Once the query has been completed, the results are saved to a .csv file, and can be retrieved either on the instance or via the REST APIs.

NOTE: NAIS can be used for querying, however, queries go into the same queue as training jobs. It's recommended that this is used only for testing purposes, and NPQR is used for queries in production.

Query Workflow
Figure 3: High Level Query Workflow

Best Practices and Requirements

  • NeoPulse® AI Studio CLI needs to be run as root (many commands will not work when not run as root.
  • NML files must be within the /DM-Dash directory when calling the train API. When uploading files via the upload API this automatically done for you (though you must specify absolute path to training CSV within the NML).
  • CSV files can be anywhere on the instance (they must include absolute paths to training data).
  • Training data can be anywhere on the instance.
  • Trim projects after training to remove poorly performing models and iterations.

WARNING: PAY ATTENTION TO AVAILABLE DISK SPACE

NeoPulse® AI Studio will crash (along with the rest of the machine) if there is no space left on the root volume “/”. Make sure you have enough space for the models you are training. NeoPulse® AI Studio saves a copy of the model after every pass through the data (epoch) so you can select which model you want to export. These files can be large depending on the architecture of your model.

TRIM YOUR PROJECTS AFTER TRAINING

AI Studio only saves metrics for your model when you trim or export the model.

PROJECTS THAT HAVE NOT BEEN TRIMMED OR EXPORTED WILL BE REMOVED WHEN RE-STARTING NeoPulse® AI STUDIO