Examples

Installation

Install the latest version in your Python environment from PyPi: confidential-ml-utils.

Exception Handling

First execute pip install confidential-ml-utils to install this library. Then wrap any methods which may throw an exception with the decorator prefix_stack_trace. Here's a simple example. Your code may explicitly raise the Public* exceptions (PublicValueError, PublicRuntimeError, PublicArgumentError, PublicKeyError, PublicTypeError) when you know that the content of the exception does not contain any private content. The messages in these exceptions will be preserved, even if keep_message is set to False.

from confidential_ml_utils.exceptions import prefix_stack_trace

@prefix_stack_trace()
def main():
    print("Hello, world!")

if __name__ == "__main__":
    main()

Logging

Call confidential_ml_utils.enable_confidential_logging to set up data category-aware logging. Then continue to use standard Python logging functionality as before! Add a category=DataCategory.PUBLIC argument to have your log lines prefixed with SystemLog:. Here is a full-fledged example:

# Copyright (c) Microsoft Corporation.
# Licensed under the MIT license.

import argparse
import confidential_ml_utils
from confidential_ml_utils.constants import DataCategory
import logging

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--prefix", default="SystemLog:")
    parser.add_argument("--log_level", default="INFO")
    args = parser.parse_args()

    confidential_ml_utils.enable_confidential_logging(
        args.prefix,
        level=args.log_level,
        format="%(prefix)s%(levelname)s:%(name)s:%(message)s",
    )

    # Output will be:
    # WARNING:root:private info
    # SystemLog:WARNING:root:public info
    logging.warning("private info")
    logging.warning("public info", category=DataCategory.PUBLIC)

    logger = logging.getLogger(__name__)

    # Output will be:
    # SystemLog:INFO:__main__:public info
    # SystemLog:WARNING:__main__:public info
    # WARNING:__main__:private info
    logger.info("public info", category=DataCategory.PUBLIC)
    logger.warning("public info", category=DataCategory.PUBLIC)
    logger.warning("private info")

Examples

The simplest use case (wrap your main method in a decorator) is:

# Copyright (c) Microsoft Corporation.
# Licensed under the MIT license.

"""
Simplest example of how to use the prefix_stack_trace decorator.
"""

from confidential_ml_utils.exceptions import prefix_stack_trace


# Output will be:
# SystemLog: Traceback (most recent call last):
# SystemLog:   File ".\hello-world.py", line 11, in main
# SystemLog:     print(1 / 0)
# SystemLog: ZeroDivisionError: **Exception message scrubbed**
@prefix_stack_trace()
def main():
    print("Hello, world!")
    print(1 / 0)


if __name__ == "__main__":
    main()

Prefixing stack trace

Some configuration options around prefixing the stack trace. You can: - customize the prefix and the exception message - keep the original exception message (don't scrub) - pass an allow_list of strings. Exception messages will be scrubbed unless the message or the exception type regex match one of the allow_list strings.

# Copyright (c) Microsoft Corporation.
# Licensed under the MIT license.

"""
Demonstrate how scrubbing options.
"""

from confidential_ml_utils.exceptions import prefix_stack_trace

# Output will be:
#
# MyCustomPrefix Traceback (most recent call last):
# MyCustomPrefix   File ".\prefix-stack-trace.py", line 11, in main
# MyCustomPrefix     print(1 / 0)
# MyCustomPrefix ZeroDivisionError: **Exception message scrubbed**
@prefix_stack_trace(prefix="MyCustomPrefix")
def custom_prefix():
    print(1 / 0)


# Output will be:
#
# SystemLog: Traceback (most recent call last):
# SystemLog:   File "/mnt/c/code/confidential-ml-utils/docs/samples/prefix-stack-trace.py", line 20, in scrub2
# SystemLog:     print(1 / 0)
# SystemLog: ZeroDivisionError: Private data was divided by zero
@prefix_stack_trace(scrub_message="Private data was divided by zero")
def custom_message():
    print(1 / 0)


# Output will be:
#
# SystemLog: Traceback (most recent call last):
# SystemLog:   File "/mnt/c/code/confidential-ml-utils/docs/samples/prefix-stack-trace.py", line 24, in scrub3
# SystemLog:     print(1 / 0)
# SystemLog: ZeroDivisionError: division by zero
@prefix_stack_trace(keep_message=True)
def keep_exception_message():
    print(1 / 0)


# Output will be:
#
# SystemLog: Traceback (most recent call last):
# SystemLog:   File "/mnt/c/code/confidential-ml-utils/docs/samples/prefix-stack-trace.py", line 28, in scrub4
# SystemLog:     print(1 / 0)
# SystemLog: ZeroDivisionError: division by zero
@prefix_stack_trace(keep_message=False, allow_list=["ZeroDivision"])
def keep_allowed_exceptions():
    print(1 / 0)


# Output will be:
#
# SystemLog: 2020-11-12 16:56:59 Traceback (most recent call last):
# SystemLog: 2020-11-12 16:56:59   File "prefix-stack-trace.py", line 56, in keep_allowed_exceptions
# SystemLog: 2020-11-12 16:56:59     print(1 / 0)
# SystemLog: 2020-11-12 16:56:59 ZeroDivisionError: **Exception message scrubbed**
@prefix_stack_trace(add_timestamp=True)
def add_timestamp():
    print(1 / 0)


if __name__ == "__main__":
    try:
        custom_prefix()
    except:
        pass
    try:
        custom_message()
    except:
        pass
    try:
        keep_exception_message()
    except:
        pass
    try:
        keep_allowed_exceptions()
    except:
        pass
    try:
        add_timestamp()
    except:
        pass

With statements

Use this library with with statements:

# Copyright (c) Microsoft Corporation.
# Licensed under the MIT license.

"""
Simple script with examples of how to use "with statements" to capture
information about failed module imports.
"""

from confidential_ml_utils.exceptions import PrefixStackTrace

# Output will be:
#
# SystemLog: Traceback (most recent call last):
# SystemLog:   File ".\docs\logging\with-statement.py", line 11, in <module>
# SystemLog:     import my_custom_library  # noqa: F401
# SystemLog: ModuleNotFoundError: **Exception message scrubbed**

with PrefixStackTrace():
    # Import statement which could raise an exception containing sensitive
    # data.
    import my_custom_library  # noqa: F401

# Output will be:
#
# SystemLog: Traceback (most recent call last):
# SystemLog:   File ".\docs\logging\with-statement.py", line 22, in <module>
# SystemLog:     import another_custom_library  # noqa: F401
# SystemLog: ModuleNotFoundError: No module named 'another_custom_library'
with PrefixStackTrace(keep_message=True):
    # Import statement which will never raise an exception containing sensitive
    # data.
    import another_custom_library  # noqa: F401

Directly with try / except statements

Using this library directly inside try / except statements:

# Copyright (c) Microsoft Corporation.
# Licensed under the MIT license.

"""
Simple script with examples of how to directly use the function
print_prefixed_stack_trace to capture information about failed module imports.
"""

from confidential_ml_utils.exceptions import print_prefixed_stack_trace_and_raise

try:
    # Import statement which could raise an exception containing sensitive
    # data.
    import my_custom_library  # noqa: F401
except BaseException as e:
    # Output will be:
    #
    # SystemLog: Traceback (most recent call last):
    # SystemLog:   File ".\try-except.py", line 10, in <module>
    # SystemLog:     import my_custom_library
    # SystemLog: ModuleNotFoundError: **Exception message scrubbed**
    print_prefixed_stack_trace_and_raise(err=e)

try:
    # Import statement which will never raise an exception containing sensitive
    # data.
    import another_custom_library  # noqa: F401
except BaseException as e:
    # Output will be:
    #
    # SystemLog: Traceback (most recent call last):
    # SystemLog:   File ".\try-except.py", line 17, in <module>
    # SystemLog:     import another_custom_library
    # SystemLog: ModuleNotFoundError: No module named 'another_custom_library'
    print_prefixed_stack_trace_and_raise(err=e, keep_message=True)

Public exception types

Using the Public* exception types:

# Copyright (c) Microsoft Corporation.
# Licensed under the MIT license.

"""
Sample use of Public* exception types.
"""

from confidential_ml_utils.exceptions import prefix_stack_trace, PublicValueError


def divide(a, b):
    if not b:
        raise PublicValueError("Second argument cannot be null or zero.")
    return a / b


# Output will be:
# SystemLog: Traceback (most recent call last):
# SystemLog:   File ".\docs\logging\public-exceptions.py", line 24, in main
# SystemLog:     divide(1, 0)
# SystemLog:   File ".\docs\logging\public-exceptions.py", line 13, in divide
# SystemLog:     raise PublicValueError("Second argument cannot be null or zero.")
# SystemLog: confidential_ml_utils.exceptions.PublicValueError: SystemLog:Second argument cannot be null or zero.
@prefix_stack_trace()
def main():
    divide(1, 0)


if __name__ == "__main__":
    main()

Exception or Stack trace parsing

The stack_trace_extractor namespace contains simple tools to grab Python or C# stack traces and exceptions from log files. Sometimes the file that has the stack trace you need may also contain sensitive data. Use this tool to parse and print the stack trace, exception type and optionally exception message (careful as exception messages may also potentially hold private data.)

from confidential_ml_utils.stack_trace_extractor import StacktraceExtractor

extractor = StacktraceExtractor()
extractor.extract("log_file")