Direct Hints

Direct hints are used to help the oracle when fine-tuning a model.

All types of direct communication with the oracle are done using the auto keyword. This keyword is used directly within architecture, source, train, and block constructs. There are two types of direct hints: Implicit, and Explicit.

Implicit direct hints:

NOTE: Implicit auto is currently only implemented for describing architectures (in the architecture and block constructs), and does not work anywhere else.

For networks there are three forms of implicit auto:

  • Full Auto: Inside of the neuralflow declaration of the architecture construct, instead of defining an architecture, the auto keyword can be used. This instructs the oracle to figure out automatically what architecture to use.

    Example:

    x -> auto -> y

NOTE: This can only be used in the architecture construct.

NOTE: Currently this is only implemented for oracle("mode") = "classification" tasks. AI oracle generated architectures for other tasks are coming soon.

  • Call Level Auto: Call level auto tells the oracle to automatically decide what parameter values to use for a particular layer of the architecture. The oracle bases its choice of parameters on the oracle("complexity") and oracle("regularization") hints, as well as the number of models specified by the oracle("generated") hint.

    Example:

    x -> Convolution2D: auto -> y

NOTE: This can be used in either architecture or block constructs.

  • Parameter Level Auto: Parameter level auto queries the oracle for a valid parameter range for a single specific parameter.

    Example:

    x -> Convolution2D: [nb_filter = 128, nb_row = auto, nb_col = auto] -> y

NOTE: This can be used in either architecture or block constructs.

Explicit direct hints:

Explicit auto works on any level and is not restricted to any construct. Explicit auto allows the user to specify probability distributions to the oracle for selecting any type of hyper-parameter. There are two different ways of specifying these distributions: choice, and distribution syntax:

Choice syntax

This tells the oracle to choose between different specific values of a given parameter, with a specified probability, when enumerating the space of hyper-parameters.

Syntax:

auto(a ? b ? ... ? z, probabilities=[P(a), P(b) ... P(z)], | name="parameter_name")

For example, a user can specify that the oracle choose hyper-parameters for the nb_filter parameter of a Convolution2D layer in any of the following ways:

  • Tell the oracle to sample the values 512 and 1024 with equal probability.

    Convolution2D: [nb_filter=auto(512 ? 1024)]

  • Tell the oracle to assign equal probabilities to the two values and give the hyper-parameter a name (for identifying the hyper-parameter choices when using list from the CLI).

    Convolution2D: [nb_filter=auto(512 ? 1024 | name="nb_filter1")]

  • Tell the oracle to sample the values 512 and 1024 for the nb_filter parameter with probabilities 0.1 and 0.9, respectively:

    Convolution2D: [nb_filter=auto(512 ? 1024, probabilities=[0.1, 0.9])]

  • Tell the oracle to sample the values 512 and 1024 for the nb_filter parameter with probabilities 0.1 and 0.9, respectively, and provide a name for the hyper-parameter (that will be displayed when using list from the CLI)

    Convolution2D: [nb_filter=auto(512 ? 1024, probabilities=[0.1, 0.9] | name="nb_filter1")]

Distribution syntax:

This tells the oracle to randomly sample hyper-parameter values from a known probability distribution.

Example: Sample the nb_filter parameter from a uniform distribution with a lower bound of ten and an upper bound of 100, as integers.

Convolution2D: [nb_filter=auto(dist="uniform", low=10, high=100, cast="int" | count=10, name="hello")]

NOTE: Only the uniform distribution is currently implemented; more will be added in the future.

The cast parameter specifies the type (float or int) for the sample from the distribution. This parameter is MANDATORY.

If the count parameter isn't specified, the oracle will decide how many samples to choose from the distribution. This value can be changed by setting the following Indirect Hint (the default value is 3):

oracle("default_sample_rate") = 3