Models is where users can create projects and train their models. As such, the creation of the NML file used for training is also handled here. For now, Models has five components: the "Overview", "Project Data", "Model Editor", "Train", and "A/B Testing" sub-tabs, in which all of the functionality of Models is contained. The main actions available there will be explained in the sections to follow.
Ex: an empty Models Tab (no models have been trained yet)
Creating and Selecting Projects
In NeoPulse® Manager, a project refers to a collection of AI models curated to perform a specific AI task. As such, projects provide a method by which to organize the work done within a single instance of NeoPulse® Manager. In the top left of the Models Tab is the Project Selector, which is a drop-down menu allowing users to switch between all currently-existing projects. Below the Project Selector is an "Add Project" button, which allows users to create a new project and attach to it a name and description. Upon submitting a new project, it will be automatically added to the Project Selector and selected.
NOTE: While in the Models Tab of NeoPulse® Manager, any sub-tabs will only show information specific to the project displayed in the Project Selector, instead of showing information for all projects.
Example: the Project Selector
To edit or delete a project, select it in the Project Selector and head to the "Overview" tab. There, next to the project's title, are two icons. The pencil icon allows users to change the project's title and description, while the trash can icon prompts the user for confirmation of deletion of the project. Note that this also deletes any of the project's previously-trained models, if they exist.
Adding Datasets and Views
Input data is necessary in the training of a project's models. Within NeoPulse® Manager, a dataset and corresponding views must be individually attached to a project prior to creating an NML file and performing training. To add a dataset to a project, first ensure that the correct project is displayed in the Project Selector. Then head to the "Project Data" tab of Models, in which there will be an "Add Dataset To Project" button. Clicking this button creates a drop-down menu listing the titles of all datasets previously uploaded to NeoPulse® Manager (see how to upload datasets here). Simply select the desired dataset and press "Add" to add it to the project.
Once a dataset has been added to a project, it will be displayed in the "Project Data" tab. Specifically, there will be a header showing the dataset's title, the number of columns and rows in the dataset's CSV, and a trash can icon to remove the dataset from the project. Note that clicking this icon only disassociates the dataset from the selected project; it does not delete the dataset from NeoPulse® Manager entirely.
Example: a Project's attached Datasets
Furthermore, underneath this header will be an "Add View" button and a default view which has been automatically created. Within NeoPulse® Manager, views are crucial in easing the process of creating an NML file, as their information directly correlates to the input and output blocks of the source construct. Changes are made to a view through the View Editor, which can be accessed by clicking on an already-existing view. Users will also be brought to the View Editor screen after clicking the "Add View" button and entering a name and description for the new view. Views can also be deleted using their corresponding trash can icon.
While within the View Editor, users have the opportunity to see the dataset's CSV file and change which block of the source construct the column will belong to. Next to each column's header is a toggle-able button that cycles between three options: INP, OUT, and N/A. Columns with the INP option will be included in the input block of the source construct of the NML file, while OUT columns will be included in the output block. Columns with the N/A option will be left out of the NML completely. Once changes have been made to a view, users may press the "Save" button and continue editing, press the "Save & Continue" button to exit the View Editor, or press the "Back" button to revert all changes and exit the view Editor.
Example: the View Editor
Designing Neural Networks
Recall that the information in a dataset's view corresponds to the source construct of an NML file. However, an NML file needs two more constructs in order to be used in a training job. These are the architecture and train constructs, which define the model's neuralflow and training parameters. To finish the creation of the NML file used for training, head to the "Model Editor" tab and click on the "Create Neural Network Design" button. Upon pressing the button, users have the opportunity to add a name and description to the new design. Furthermore, a dataset view must be selected from a drop-down menu. Note that this view can be changed later if the wrong one was selected. Upon entering this information, users may begin designing their neural networks in one of two ways: the "Visual Model Editor" and "NML Editor", which can be selected back-and-forth from a tab underneath the "Back" button at the top of the screen. The features of these editors will be described in further detail below:
Visual Model Editor
The Visual Model Editor provides a Graphical UI with which to design the architecture construct of the NML file to be submitted for training. With this tool, users need not know how to write a single line of NML to train models for a project! This is because the selected view auto-generates the source construct, the visual Model Editor automatically translates the user's design to a valid architecture construct, and the auto-generated train construct (not shown inside the Visual Model Editor) does not need to be changed. This makes the Visual Model Editor a powerful tool for users who are new to the NeoPulse® framework, but also comes in handy for visual people who would prefer to see their design laid out nicely.
The Visual Model Editor tab has a blue gear icon to its right. There, users can select preferences for the design of the Editor itself, such as enabling "Snap to Grid" or toggling the background type between dots and lines. Once the proper preferences have been selected, the Visual Model Editor will reflect those changes. By default, the Visual Model editor shows the inputs/outputs from the selected view connected through an auto block. To delete a connection, use the connection's corresponding trash can icon. Furthermore, inputs, outputs, or layer blocks can be deleted by selecting them and pressing the "delete" key.
To add a new component to the Editor, take note of the "Inputs", "Pretrained", "Core", "Common", "Operations", "Time-Series", and "Outputs" drop-down menus underneath the name/description of the Neural Network Design. Each menu contains layer blocks or the input/output blocks in case they were accidentally deleted. Selecting an option from the menu appends it to the mouse, where it can be dropped anywhere on the screen via a click. Holding the Meta key (i.e. "command" on Mac) while clicking allows for multiple instances of the same selected block to be dropped. The parameters of these blocks can then be edited by double-clicking on them. Once the components have been configured to the user's satisfaction, connect them by dragging the mouse from the right edge of one block to the left edge of another block. This can be used to connect inputs to a layer block, layer blocks to other layer blocks, and layer blocks to the output block.
Example: the Visual Model Editor and an example Neuralflow
NOTE: This particular neuralflow is not necessarily recommended, and is meant to display a sample of the available Keras layers and modifications to their parameters.
Once the Neural Network Design is complete, simply select the "Back" button at the top of the screen to save the design. This returns to user to the base "Model Editor" tab, where the new Neural Network Design will appear alongside any other previously-created designs. However, if the user wishes to modify the source/train constructs to edit some of the parameters used in training, like batch size or the number of epochs, head to the "NML Editor" tab beside the "Visual Model Editor" tab, which will be described in the following section. Note that switching from the Visual Model Editor to the NML Editor will automatically generate the NML code that describes the built neuralflow in the architecture construct of the NML Editor.
For users familiar with how to write NML, the NML Editor may be a nicer tool for you. Moreover, if a user wishes to modify the auto-generated source and train constructs of the NML file, the NML Editor is the only method by which such customization can occur. The NML Editor itself has four sections:
- Oracle Mode Selector: a drop-down menu allowing users to select the overarching AI task for the oracle. The selected Oracle Mode will appear at the top of the Source NML section. More information can be found here.
- Source NML: an NML Source construct which has been automatically generated to match the selected dataset view. The view upon which the section is built can be changed by clicking on the blue text on the right of the Source NML header. This section can also be collapsed by toggling the arrow to the left of the "Source NML" header.
- Architecture NML: an NML Architecture construct which defines the neuralflow of the models to be trained. It uses an
autolayer by default, which is only currently supported for
"classification". Users may need to manually define this section for other Oracle Modes or if a specific model design is desired.
- Train NML: an NML Train construct which defines the parameters of the models' training process. If users wish to set their own parameters, they can do so in this section, but no editing is required for training to occur successfully. This section can be collapsed by toggling the arrow to the left of the "Train NML" header.
Once all desired changes have been added, users must select the "Save NML" button at the bottom right of the NML Editor. Note that this button also generates the neuralflow to be viewed in the Visual Model Editor from the architecture construct, so if the user makes any changes that they wish to see in the Visual Model Editor, be sure to save those changes before heading back to the Visual Model Editor tab. Once the NML has been saved, clicking the "Back" button at the top of the screen will exit and make the new Neural Network Design appear in the "Model Editor" tab alongside any other designs previously created.
Example: the NML Editor with an auto-generated architecture construct matching the example shown in the Visual Model Editor
Now that the Model Design has been completed in the "Model Editor" tab, models may be created and trained for the project displayed in the Project Selector. To train a project's models, head to the "Train" tab. There, under the "Train a Model" header, users will see two drop-down menus. The first is used to select the completed Model Design, while the second is used to select a NeoPulse® Studio machine on which to perform the training (these are managed from the DevOps Tab).
After selecting the desired items from the two drop-downs, upon pressing the "Train" button, users will see their training job appear under the "Jobs" header. This section displays each training job's submission timestamp and status, so users can recognize if an error has occurred, when training is underway, and when training has completed. All submitted training jobs for a project will appear here.
NOTE: The job name is generated as a function of the project name, model design name, and a random hash.
Example: the Train Tab
Instead of training models through the current instance of NeoPulse® Manager, models can be imported to a project. If this method is desired instead, head to the "Overview" tab and select the "Import PIM" button. Then users may select the method by which to import the PIM: from a PIM file on the local machine, from ONNX, from Keras, or from PyTorch. Note that this method of creating models still requires a NeoPulse® Studio machine to be added from DevOps.
Viewing and Testing Trained Models
While a training job is running, a graphical display of the models' training can be seen in the "Overview" tab. The graph has "Iteration" on the x-axis and "Metric Value" on the y-axis so that users may see how a model's metric values changed over the duration of its training. Hovering over a specific data point reveals the PIM to which the data point belongs and all other metric values at the same iteration. Furthermore, all graphs related to that model are highlighted. The scale of the y-axis can be changed using the "scale-Y" button, where a linear or log scale can be selected. By default, all training jobs pertaining to a project are shown, but these may be toggled beneath the x-axis of the graph. Similarly, all metrics are shown by default, but a specific metric can be hidden by clicking on it inside the legend.
Example: a graph in the Overview Tab for a completed training job
Trained models can also be tested using data from within the "A/B Testing" tab. Upon clicking the "New Test" button, users can select an uploaded dataset with which to test trained models. Note that the dataset must be attached to the project from the "Project Data" tab. Then the same graph described above is shown for the purpose of selecting up to 5 PIMs to be tested. Upon submission, a new Test will appear in the "A/B Testing" tab with the status "TEST_SUBMITTED". Once the status becomes "TEST_COMPLETED", the results of the test may be viewed by clicking on the test. And adding graphs with any desired metrics.
Example: the A/B Testing parameter selection screen
Example: the display for a finished test