D2D2T – Disk to Disk to Tape. Backedup data from a disk source system is first written to a target disk system and then sent on to a tape library.

D2D – Disk to disk. A term used to describe backup storage systems with the backup of a disk-based source system stored in another disk-based target system and not a tape library.

CSI – Container Storage Interface. This is an industry standard interface that can be used by any storage system provider and works across various container orchestration systems, such as Kubernetes, Mesos and Cloud Foundry. Envisage a storage supplier writing a single CSI plugin so their storage could be used by containers orchestrated by any CSI-supporting orchestrator.

The specification is formally described here. With it, a container orchestrator interacts with a plugin though remote procedure calls (RPCs.) These, also known as Application Programming Interface (API) calls, are used for;

• Dynamic provisioning and deprovisioning of a volume.
• Attaching or detaching a volume from a node.
• Mounting/unmounting a volume from a node.
• Consumption of both block and mountable volumes.
• Local storage providers (e.g., device mapper, lvm).
• Creating and deleting a snapshot (source of the snapshot is a volume).
• Provisioning a new volume from a snapshot (reverting snapshot, where data in the original volume is erased and replaced with data in the snapshot, is out of scope).

CPU – Central Processing Unit – the main processor in a server, desktop or laptop computer. It executes machine-level instructions and is a general purpose processor. There can be multiple processor cores inside a CPU, and these cores can execute in parallel. The cores can have multiple threads also executing instructions in parallel. Intel makes x86 architecture CPUs, as does AMD. Other CPU architectures include Arm and Risc-V.

Container – A software entity that contains application code to run microservices with a standard interface to other containers. Unlike a virtual machine a container does not contain a guest operating system and is, therefore, smaller than the equivalent application code running in a virtual machine. Containers are loaded and run – instantiated -and orchestrated by software such as Kubernetes. They were originally stateless with no state information preserved between their instantiations. Stateful containers do have data about their state stored between instantiations.

ROW – Redirect-on-Write. See snapshot entry.

Snapshot – A virtual copy of a file or RAID volume or object at a point in time created in a database, hypervisor, file system or storage system that relies on the original. We’ll use volume to refer to all three here. A snapshot is a virtual copy. It has two main purposes: easy recovery of deleted or corrupted files, and a source for replication or backup. In order for the snapshot to protect against media failure, it must be transformed into a physical copy via replication or backup to another devices. If you copy this snapshot to another system it becomes a real copy. 

There are two ways of writing snapshots; Copy-on-Write (COW) and Redirect-on-Write (ROW). A COW system takes place when blocks in the volume are going to be overwritten. A copy is made of the to-be-overwritten blocks and written in a separate place or snapshot area. Then the fresh data overwrites the old data in the volume. There are three I/O operations needed; a read followed by two writes.

The ROW method treats snapshots as a list of pointers. When blocks in a volume are going to be overwritten the snapshot system is updated with the pointers to the about-to-be-overwritten blocks and then the new data is written to a fresh area of the drive. Thus the existing old data blocks are added to the snapshot system without having to be read and copied. There is only one I/O operation; a write, plus the snapshot pointer updating. Consequently ROW snapshots are faster than COW snapshots.

Compression – the shrinking of a file or object’s size by replacing repeated strings of bits with fewer bits and a pointer. This is done so that files take up less storage space or needs less bandwidth when transmitted across a network link. When the file is received at the far end of the link it may be decompressed, as a stored compressed file would be when it is read. The compression-decompression processes could be carried out by a codec hardware device or performed using software.

Compression techniques can be applied to text files, audio files, images and video files, and can be done in a lossless manner or a lossy way, meaning that some information is permanently lost. Audio files may contain frequencies that the human ear cannot detect  and removing them won’t affect the perceived quality of the heard audio file but will reduce its size. Applying lossy compression to images and videos can reduce the perceived quality of both.

Applying lossy compression to text files is self-defeating as losing information obscures the meaning of the text.

Compression techniques

Algorithms are used to scan files and find repeated bit sequences. They are replaced by a pointer which identifies the original bit sequence and the location in the file where it should be written when the file is decompressed. The original bit sequences can be stored in a dictionary appended to the file.

JPEG – (Joint Photographic Experts Group) is a standard format for compressed image data which is lossy in that some data is lost. This can be most effective in reducing image file sizes while still delivering acceptable image quality.

MPEG – (Moving Picture Experts Group) – A way of compressing video files that stores changes between frames in a video and not the entire frame. There are different types of MPEG compression, such as MPEG2, MPEG2 and MPEG4.

Lempel-Ziv – Abraham Lempel and Jacob Ziv published the  LZ77 and LZ78 lossless compression papers in 1977 an 1978 respectively. Repeated bit sequences in a file are detected by reading a section of the file, a window. When identified the repetitive sequence is stored in the file as a length-distance pair (pointer). This says replace the next N characters with the sequence found at earlier location X in the file. Then the window is moved up the file (sliding window in LZ77) and the repetition search repeated.

The file is thus reduced in size and can be exactly rebuilt (rehydrated) when it is read.

The LZ78 algorithm stores repeated bit sequences in a dictionary and uses a pointer added to the file to identify which dictionary sequence to insert at a particular location.

The LZ78 algorithm was improved by a later LZW algorithm that pre-initializes a dictionary used with all possible characters or with an emulation of a pre-initialized dictionary.

The Lempel-Ziv algorithms are widely used.

ZIP – ZIP or archive folders are produced by losslessly compressing files and putting them inside a ZIP folder to save bandwidth and time with a network transmission or storage on a portable device. When read they are “un-zipped.”

Note. See also deduplication.