Skip to content

flash.py

ofrak_components.flash

This component is intended to make it easier to analyze raw flash dumps using OFRAK alone. Most flash dumps have "useful" data mixed in with out-of-band (OOB) data. The OOB data often includes some Error Correcting Codes (ECC) or checksums.

There are several dataclasses that categorize the sections of the dump:

  • FlashResource is the overarching resource. The component expects the user to add this tag in order for this component to be run.
    • FlashOobResource is the region of FlashResource that has OOB data. In the future, there may be multiple of these children resources.
      • FlashLogicalDataResource is the extracted data only with all of the OOB data removed. This will become a FlashOobResource when packed.
      • FlashLogicalEccResource is the extracted ECC only. No other OOB data is included.

FlashAttributes (ResourceAttributes) dataclass

FlashAttributes is for specifying everything about the specific model of flash. The intent is to expand to all common flash configurations. Every block has a format specifier to show where each field is in the block as well as the length. If there is no OOB data, data_block_format may take this form:

data_block_format = [FlashField(field_type=FlashFieldType.DATA,size=block_size),]

Important Notes: Assumes that the provided list for each block format is ordered. Only define a block_format if they are different from other block formats. - A current workaround is adding FlashField(FlashFieldType.ALIGNMENT, 0) Assumes that there are only one of each block format except the data_block_format

FlashEccAttributes (ResourceAttributes) dataclass

Must be configured if the resource includes ECC ecc_magic is assumed to be contained at the start of the file, but may also occur multiple times

FlashField dataclass

FlashField(field_type: ofrak_components.flash.FlashFieldType, size: int)

FlashFieldType (Enum)

DATA_SIZE is the packed size of the DATA only (excluding MAGIC, CHECKSUM, DELIMITER, ECC, etc) TOTAL_SIZE is the size of the entire region (including all DATA, MAGIC, CHECKSUM, DELIMITER, ECC, etc) ALIGNMENT will pad with bytes by default

FlashLogicalDataResource (GenericBinary) dataclass

This is the final product of unpacking a FlashResource. It contains the data without any ECC or OOB data included. This allows for recursive packing and unpacking.

FlashLogicalDataResourcePacker (Packer)

Packs the FlashLogicalDataResource into a FlashOobResource of the format specified by the FlashAttributes

pack(self, resource, config=None) async

Pack the given resource.

Users should not call this method directly; rather, they should run Resource.run or Resource.pack.

Parameters:

Name Type Description Default
resource Resource required
config

Optional config for packing. If an implementation provides a default, this default will always be used when config would otherwise be None. Note that a copy of the default config will be passed, so the default config values cannot be modified persistently by a component run.

None
Source code in ofrak_components/flash.py
async def pack(self, resource: Resource, config=None):
    try:
        flash_attr = resource.get_attributes(FlashAttributes)
    except NotFoundError:
        raise UnpackerError("Tried packing without FlashAttributes")
    data = await resource.get_data()
    bytes_left = len(data)
    original_size = bytes_left
    packed_data = bytearray()
    data_offset = 0

    for c in flash_attr.iterate_through_all_blocks(original_size, False):
        block_data = b""
        block_data_size = flash_attr.get_field_length_in_block(c, FlashFieldType.DATA)
        if block_data_size != 0:
            # Get the data for the current block
            block_data = data[data_offset : data_offset + block_data_size]
            data_offset += block_data_size
            bytes_left -= block_data_size

        packed_data += _build_block(
            cur_block_type=c,
            attributes=flash_attr,
            block_data=block_data,
            original_data=data,
        )

    # Create child under the original FlashOobResource to show that it packed itself
    parent = await resource.get_parent()
    await parent.create_child(tags=(FlashOobResource,), data=packed_data)

FlashLogicalEccResource (GenericBinary) dataclass

The alternate to FlashLogicalDataResource but just includes ECC. Does not include any other OOB data. Generally less useful on its own but provided anyway.

FlashOobResource (GenericBinary) dataclass

Represents the region containing Oob data.

FlashOobResourcePacker (Packer)

Packs the entire region including Oob data back into a binary blob

pack(self, resource, config=None) async

Pack the given resource.

Users should not call this method directly; rather, they should run Resource.run or Resource.pack.

Parameters:

Name Type Description Default
resource Resource required
config

Optional config for packing. If an implementation provides a default, this default will always be used when config would otherwise be None. Note that a copy of the default config will be passed, so the default config values cannot be modified persistently by a component run.

None
Source code in ofrak_components/flash.py
async def pack(self, resource: Resource, config=None):
    # We want to overwrite ourselves with just the repacked version
    packed_child = await resource.get_only_child(
        r_filter=ResourceFilter.with_tags(
            FlashOobResource,
        ),
    )
    if packed_child is not None:
        patch_data = await packed_child.get_data()
        original_size = await resource.get_data_length()
        resource.queue_patch(Range(0, original_size), patch_data)

FlashOobResourceUnpacker (Unpacker)

Unpack a single FlashOobResource dump into logical data using the FlashAttributes.

unpack(self, resource, config=None) async

Unpack the given resource.

Users should not call this method directly; rather, they should run Resource.run or Resource.unpack.

Parameters:

Name Type Description Default
resource Resource

The resource that is being unpacked

required
config

Optional config for unpacking. If an implementation provides a default, this default will always be used when config would otherwise be None. Note that a copy of the default config will be passed, so the default config values cannot be modified persistently by a component run.

None
Source code in ofrak_components/flash.py
async def unpack(self, resource: Resource, config=None):
    try:
        flash_attr = resource.get_attributes(FlashAttributes)
    except NotFoundError:
        raise UnpackerError("Tried unpacking without FlashAttributes")
    ecc_attr: Optional[FlashEccAttributes] = flash_attr.ecc_attributes

    # oob_resource = _create_oob_resource(resource=resource)
    oob_resource = resource
    # Parent FlashEccResource is created, redefine data to limited scope
    data = await oob_resource.get_data()
    data_len = len(data)

    # Now add children blocks until we reach the tail block
    offset = 0
    only_data = b""
    only_ecc = b""
    for c in flash_attr.iterate_through_all_blocks(data_len, True):
        block_size = flash_attr.get_block_size(c)
        block_end_offset = offset + block_size
        if block_end_offset > data_len:
            raise UnpackerError("Expected complete block and received less than expected")
        block_range = Range(offset, block_end_offset)
        block_data = await oob_resource.get_data(range=block_range)

        block_ecc_range = flash_attr.get_field_range_in_block(c, FlashFieldType.ECC)
        if block_ecc_range is not None:
            cur_block_ecc = block_data[block_ecc_range.start : block_ecc_range.end]
            only_ecc += cur_block_ecc
            # Add hash of everything up to the ECC to our dict for faster packing
            block_data_hash = md5(block_data[: block_ecc_range.start]).digest()
            DATA_HASHES[block_data_hash] = cur_block_ecc

        # Check if there is data in the block
        block_data_range = flash_attr.get_field_range_in_block(c, FlashFieldType.DATA)
        if block_data_range is not None:
            if block_ecc_range is not None:
                # Try decoding/correcting with ECC, otherwise just add the data anyway
                try:
                    # Assumes that data comes before ECC
                    if ecc_attr is not None and ecc_attr.ecc_class is not None:
                        only_data += ecc_attr.ecc_class.decode(
                            block_data[: block_ecc_range.end]
                        )[block_data_range.start : block_data_range.end]
                    else:
                        raise UnpackerError(
                            "Tried to correct with ECC without providing an ecc_class in FlashEccAttributes"
                        )
                except EccError:
                    only_data += block_data[block_data_range.start : block_data_range.end]
            else:
                only_data += block_data[block_data_range.start : block_data_range.end]
        offset += block_size

    # Add all block data to logical resource for recursive unpacking
    await oob_resource.create_child(
        tags=(FlashLogicalDataResource,),
        data=only_data,
        attributes=[
            flash_attr,
        ],
    )
    if ecc_attr is not None:
        await oob_resource.create_child(
            tags=(FlashLogicalEccResource,),
            data=only_ecc,
            attributes=[
                ecc_attr,
            ],
        )

FlashResource (GenericBinary) dataclass

The overarching resource that encapsulates flash storage. This will contain a FlashOobResource in most cases. In the future, support for multiple FlashOobResource children should be added.

FlashResourcePacker (Packer)

Packs the FlashResource into binary and cleans up logical data representations

pack(self, resource, config=None) async

Pack the given resource.

Users should not call this method directly; rather, they should run Resource.run or Resource.pack.

Parameters:

Name Type Description Default
resource Resource required
config

Optional config for packing. If an implementation provides a default, this default will always be used when config would otherwise be None. Note that a copy of the default config will be passed, so the default config values cannot be modified persistently by a component run.

None
Source code in ofrak_components/flash.py
async def pack(self, resource: Resource, config=None):
    # We want to overwrite ourselves with just the repacked version
    # TODO: Add supoort for multiple FlashOobResource in a dump.
    packed_child = await resource.get_only_child(
        r_filter=ResourceFilter.with_tags(
            FlashOobResource,
        ),
    )
    if packed_child is not None:
        patch_data = await packed_child.get_data()
        original_size = await resource.get_data_length()
        resource.queue_patch(Range(0, original_size), patch_data)

FlashResourceUnpacker (Unpacker)

Finds the overarching parent for region that includes OOB data. Identifies the bounds based on the FlashAttributes.

unpack(self, resource, config=None) async

Unpack the given resource.

Users should not call this method directly; rather, they should run Resource.run or Resource.unpack.

Parameters:

Name Type Description Default
resource Resource

The resource that is being unpacked

required
config

Optional config for unpacking. If an implementation provides a default, this default will always be used when config would otherwise be None. Note that a copy of the default config will be passed, so the default config values cannot be modified persistently by a component run.

None
Source code in ofrak_components/flash.py
async def unpack(self, resource: Resource, config=None):
    try:
        flash_attr = resource.get_attributes(FlashAttributes)
    except NotFoundError:
        raise UnpackerError("Tried creating FlashOobResource without FlashAttributes")

    start_index = 0
    data = await resource.get_data()
    data_len = len(data)

    if flash_attr.ecc_attributes is not None:
        magic = flash_attr.ecc_attributes.ecc_magic
        if magic is not None:
            ecc_magic_offset = data.find(magic)

            if ecc_magic_offset == -1:
                raise UnpackerError("No header magic found")

            magic_range_in_block = flash_attr.get_field_range_in_block(
                flash_attr.header_block_format, FlashFieldType.MAGIC
            )

            if not magic_range_in_block:
                raise UnpackerError("Did not find offset for magic in header")

            start_index = ecc_magic_offset - magic_range_in_block.start

    # Set fallback, in case the current check for the end of the resource fails
    end_offset = data_len
    header_data_size = flash_attr.get_field_in_block(
        flash_attr.header_block_format, FlashFieldType.DATA_SIZE
    )
    header_total_size = flash_attr.get_field_in_block(
        flash_attr.header_block_format, FlashFieldType.TOTAL_SIZE
    )
    if flash_attr.header_block_format is not None and (
        header_data_size is not None or header_total_size is not None
    ):
        # The header has the size of the entire region including OOB data
        total_ecc_protected_size = 0
        if header_data_size is not None:
            # Found data size in the header, need to calculate expected total size (including OOB)
            data_size_bytes = flash_attr.get_field_data_in_block(
                block_format=flash_attr.header_block_format,
                field_type=FlashFieldType.DATA_SIZE,
                data=data,
                block_start_offset=0,
            )
            oob_size = flash_attr.get_total_oob_size(data_len=data_len, includes_oob=True)
            if data_size_bytes is not None:
                total_ecc_protected_size = oob_size + int.from_bytes(data_size_bytes, "big")
        elif header_total_size is not None:
            # Found total size in header
            total_size_bytes = flash_attr.get_field_data_in_block(
                block_format=flash_attr.header_block_format,
                field_type=FlashFieldType.TOTAL_SIZE,
                data=data,
                block_start_offset=0,
            )
            if total_size_bytes is not None:
                total_ecc_protected_size = int.from_bytes(total_size_bytes, "big")

        if total_ecc_protected_size > data_len:
            raise UnpackerError("Expected larger resource than supplied")

        if total_ecc_protected_size > start_index:
            end_offset = start_index + total_ecc_protected_size

    elif flash_attr.tail_block_format is not None:
        # Tail has either magic, total_size, or data_size to indicate the end
        tail_magic = flash_attr.get_field_in_block(
            flash_attr.tail_block_format, FlashFieldType.MAGIC
        )
        tail_total_size = flash_attr.get_field_in_block(
            flash_attr.tail_block_format, FlashFieldType.TOTAL_SIZE
        )
        tail_data_size = flash_attr.get_field_in_block(
            flash_attr.tail_block_format, FlashFieldType.DATA_SIZE
        )

        if tail_magic is not None:
            # we'll start looking after the header to make sure we don't
            # accidentally find the header magic
            if flash_attr.header_block_format:
                tail_start_index = flash_attr.get_block_size(flash_attr.header_block_format)
            else:
                tail_start_index = start_index
            end_offset = _get_end_from_magic(flash_attr, tail_start_index, data, data_len)
        elif tail_total_size is not None:
            end_offset = _get_end_from_size(
                flash_attr, start_index, data, data_len, FlashFieldType.TOTAL_SIZE
            )
        elif tail_data_size is not None:
            end_offset = _get_end_from_size(
                flash_attr, start_index, data, data_len, FlashFieldType.DATA_SIZE
            )

    # Create the overarching resource
    return await resource.create_child(
        tags=(FlashOobResource,),
        data_range=Range(start_index, end_offset),
        attributes=[
            flash_attr,
        ],
    )