ml_logger package

Subpackages

• ml_logger.logger package
  • Submodules
  • ml_logger.logger.base module
  • ml_logger.logger.filesystem module
  • ml_logger.logger.mlflow module
  • ml_logger.logger.mongo module
  • ml_logger.logger.tensorboard module
  • ml_logger.logger.wandb module
  • Module contents
• ml_logger.parser package
  • Subpackages
    • ml_logger.parser.experiment package
      • Submodules
      • ml_logger.parser.experiment.experiment module
      • ml_logger.parser.experiment.parser module
      • Module contents
  • Submodules
  • ml_logger.parser.base module
  • ml_logger.parser.config module
  • ml_logger.parser.log module
  • ml_logger.parser.metric module
  • ml_logger.parser.utils module
  • Module contents

Submodules

ml_logger.logbook module

Implementation of the LogBook class.

LogBook class provides an interface to persist the logs on the filesystem, tensorboard, remote backends, etc.

class ml_logger.logbook.LogBook(config: Dict[str, Any])

Bases: object

This class provides an interface to persist the logs on the filesystem, tensorboard, remote backends, etc.

write(log: Dict[str, Any], log_type: str = 'metric') → None

Write log to loggers.

Parameters

• log (LogType) – Log to write

• log_type (str, optional) – Type of this log. Defaults to “metric”.

write_config(config: Dict[str, Any]) → None

Write config to loggers.

Parameters

[ConfigType] (config) – Config to write.

write_message(message: Any, log_type: str = 'info') → None

Write message string to loggers.

Parameters

• message (Any) – Message string to write

• log_type (str, optional) – Type of this message (log). Defaults to “info”.

write_metadata(metadata: Dict[str, Any]) → None

Write metadata to loggers.

Parameters

metadata (LogType) – Metadata to wite

write_metric(metric: Dict[str, Any]) → None

Write metric to loggers.

Parameters

metric (MetricType) – Metric to write

ml_logger.logbook.make_config(id: str = '0', name: str = 'default_logger', write_to_console: bool = True, logger_dir: Optional[str] = None, filename: Optional[str] = None, filename_prefix: str = '', create_multiple_log_files: bool = True, wandb_config: Optional[Dict[str, Any]] = None, wandb_key_map: Optional[Dict[str, str]] = None, wandb_prefix_key: Optional[str] = None, tensorboard_config: Optional[Dict[str, Any]] = None, tensorboard_key_map: Optional[Dict[str, str]] = None, tensorboard_prefix_key: Optional[str] = None, mlflow_config: Optional[Dict[str, Any]] = None, mlflow_key_map: Optional[Dict[str, str]] = None, mlflow_prefix_key: Optional[str] = None, mongo_config: Optional[Dict[str, Any]] = None) → Dict[str, Any]

Make the config that can be passed to the LogBook constructor.

Parameters

• id (str, optional) – Id of the current LogBook instance. Defaults to “0”.

• name (str, optional) – Name of the logger. Defaults to “default_logger”.

• write_to_console (bool, optional) – Should write the logs to console. Defaults to True

• logger_dir (str, optional) – Path where the logs will be written. If None is passed, logs are not written to the filesystem. LogBook creates the directory, if it does not exist. Defaults to None.

• filename (str, optional) – Name to assign to the log file (eg log.jsonl). If None is passed, this argument is ignored. If the value is set, filename_prefix and create_multiple_log_files arguments are ignored. Defaults to None.

• filename_prefix (str) – String to prefix before the name of the log files. Eg if filename_prefix is “dummy”, name of log files are dummymetric.jsonl, dummylog.jsonl etc. This argument is ignored if filename is set. Defaults to “”.

• create_multiple_log_files (bool, optional) – Should multiple log files be created - for config, metric, metadata and message logs. If True, the files are named as config_log.jsonl, metric_log.jsonl etc. If False, only one file log.jsonl is created. This argument is ignored if filename is set. Defaults to True.

• wandb_config (Optional[ConfigType], optional) – Config for the wandb logger. If None, wandb logger is not created. The config can have any parameters that wandb.init() methods accepts (https://docs.wandb.com/library/init). Note that the wandb_config is passed as keyword arguments to the wandb.init() method. This provides a lot of flexibility to the users to configure wandb. This also means that the config should not have any parameters that wandb.init() would not accept. Defaults to None.

• wandb_key_map (Optional[KeyMapType], optional) – When using wandb logger for logging metrics, certain keys are required. This dictionary provides an easy way to map the keys in the log to be written) with the keys that wandb logger needs. For instance, wandb logger needs a step key in all the metric logs. If your logs have a key called epoch that you want to use as step, set wandb_key_map as {epoch: step}. This argument is ignored if set to None. Defaults to None.

• wandb_prefix_key (Optional[str], optional) – When a metric is logged to wandb, prefix the value (corresponding to the key) to all the remaining keys before values are logged in the wandb logger. This argument is ignored if set to None. Defaults to None.

• tensorboard_config (Optional[ConfigType], optional) – config to initialise the tensorboardX logger. The config can have any parameters that [tensorboardX.SummaryWriter() method](https://tensorboardx.readthedocs.io/en/latest/tensorboard.html#tensorboardX.SummaryWriter) accepts. Note that the config is passed as keyword arguments to the tensorboardX.SummaryWriter() method. This provides a lot of flexibility to the users to configure tensorboard. This also means that config should not have any parameters that tensorboardX.SummaryWriter() would not accept. Defaults to None.

• tensorboard_key_map (Optional[KeyMapType], optional) – When using tensorboard logger for logging metrics, certain keys are required. This dictionary provides an easy way to map the keys in the log (to be written) with the keys that tensorboard logger needs. For instance, tensorboard logger needs a main_tag key and a global_step in all the metric logs. If your logs have a key called epoch that you want to use as step, and a key called mode that you want to use as main_tag, set tensorboard_key_map as {epoch: global_step, mode: main_tag}. This argument is ignored if set to None. Defaults to None.

• tensorboard_prefix_key (Optional[str], optional) – When a metric is logged to tensorboard, prefix the value (corresponding to the key) to all the remaining keys before values are logged in the tensorboard logger. This argument is ignored if set to None. Defaults to None.

• mlflow_config (Optional[ConfigType], optional) – config to initialise an mlflow experiment. The config can have any parameters that [mlflow.create_experiment() method](https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.create_experiment) accepts. Note that the config is passed as keyword arguments to the mlflow.create_experiment() method. This provides a lot of flexibility to the users to configure mlflow. This also means that config should not have any parameters that mlflow.create_experiment would not accept. Defaults to None.

• mlflow_key_map (Optional[KeyMapType], optional) – When using mlflow logger for logging metrics, certain keys are required. This dictionary provides an easy way to map the keys in the log (to be written) with the keys that mlflow logger needs. For instance, mlflow logger needs a step key in all the metric logs. If your logs have a key called epoch that you want to use as step, set mlflow_key_map as {epoch: step}. This argument is ignored if set to None. Defaults to None.

• mlflow_prefix_key (Optional[str], optional) – When a metric is logged to mlflow, prefix the value (corresponding to the key) to all the remaining keys before values are logged in the mlflow logger. This argument is ignored if set to None. Defaults to None.

• mongo_config (Optional[ConfigType], optional) –

  config to initialise connection to a collection in mongodb. The config supports the following keys:

  1. host: host where mongodb is running.

  2. port: port on which mongodb is running.

  3. db: name of the db to use.

  4. collection: name of the collection to use.

  Defaults to None.

Returns

config to construct the LogBook

Return type

ConfigType

ml_logger.metrics module

Implementation of different type of metrics.

class ml_logger.metrics.AverageMetric(name: str)

Bases: ml_logger.metrics.BaseMetric

Metric to track the average value.

This is generally used for logging strings

Parameters

BaseMetric – Base metric class

get_val() → float

Get the current average value.

reset() → None

Reset Metric.

update(val: Union[int, float], n: int = 1) → None

Update the metric.

Update the metric using the current average value and the number of samples used to compute the average value

Parameters

• val (NumType) – current average value

• n (int, optional) – Number of samples used to compute the average. Defaults to 1

class ml_logger.metrics.BaseMetric(name: str)

Bases: object

Base Metric class. This class is not to be used directly.

get_val() → Union[str, int, float]

Get the current value of the metric.

reset() → None

Reset the metric to the default value.

update(val: Any) → None

Update the metric using the current val.

Parameters

val (Any) – Current value. This value is used to update the metric

class ml_logger.metrics.ComparisonMetric(name: str, default_val: Union[str, int, float], comparison_op: Callable[[Union[str, int, float], Union[str, int, float]], bool])

Bases: ml_logger.metrics.BaseMetric

Metric to track the min/max value.

This is generally used for logging best accuracy, least loss, etc.

Parameters

BaseMetric – Base metric class

reset() → None

Reset the metric to the default value.

update(val: Union[str, int, float]) → None

Use the comparison operator to decide which value to keep.

If the output of self.comparison_op(val, self)

Parameters

val (ValueType) – Value to compare the current value with. If comparison_op(current_val, new_val) is true, we update the current value.

class ml_logger.metrics.ConstantMetric(name: str, val: Union[str, int, float])

Bases: ml_logger.metrics.BaseMetric

Metric to track one fixed value.

This is generally used for logging strings

Parameters

BaseMetric – Base metric class

reset() → None

Do nothing for the constant metrics.

update(val: Optional[Union[str, int, float]] = None) → None

Do nothing for the constant metrics.

Parameters

val (Any) – This value is ignored

class ml_logger.metrics.CurrentMetric(name: str)

Bases: ml_logger.metrics.BaseMetric

Metric to track only the most recent value.

Parameters

BaseMetric – Base metric class

update(val: Union[str, int, float]) → None

Update the metric using the current val.

Parameters

val (Any) – Current value. The metric value is set to this value

class ml_logger.metrics.MaxMetric(name: str)

Bases: ml_logger.metrics.ComparisonMetric

Metric to track the max value.

This is generally used for logging best accuracy, etc.

Parameters

ComparisonMetric – Comparison metric class

class ml_logger.metrics.MetricDict(metric_list: Iterable[ml_logger.metrics.BaseMetric])

Bases: object

Class that wraps over a collection of metrics.

reset() → None

Reset all the metrics to default values.

to_dict() → Dict[str, Any]

Convert the metrics into a dictionary for LogBook.

Returns

Metric data in as a dictionary

Return type

LogType

update(metrics_dict: Union[Dict[str, Any], ml_logger.metrics.MetricDict]) → None

Update all the metrics using the current values.

Parameters

metrics_dict (Union[LogType, MetricDict]) – Current value of metrics

class ml_logger.metrics.MinMetric(name: str)

Bases: ml_logger.metrics.ComparisonMetric

Metric to track the min value.

This is generally used for logging least loss, etc.

Parameters

ComparisonMetric – Comparison metric class

class ml_logger.metrics.SumMetric(name: str)

Bases: ml_logger.metrics.AverageMetric

Metric to track the sum value.

Parameters

BaseMetric – Base metric class

get_val() → float

Get the current sum value.

ml_logger.types module

Types used in the package.

ml_logger.utils module

Utility Methods.

ml_logger.utils.compare_keys_in_dict(dict1: Dict[Any, Any], dict2: Dict[Any, Any]) → bool

Check that the two dicts have the same set of keys.

ml_logger.utils.flatten_dict(d: Dict[str, Any], parent_key: str = '', sep: str = '#') → Dict[str, Any]

Flatten a given dict using the given seperator.

Taken from https://stackoverflow.com/a/6027615/1353861

Parameters

• d (Dict[str, Any]) – dictionary to flatten

• parent_key (str, optional) – Keep track of the higher level key Defaults to “”.

• sep (str, optional) – string for concatenating the keys. Defaults to “#”

Returns

[description]

Return type

Dict[str, Any]

ml_logger.utils.make_dir(path: str) → None

Make dir, if not exists.

Parameters

path (str) – dir to make

Module contents