Remember when we used to work with punching cards? Yeah, me neither. They used to be such an intricate form of technology. The ICL system for example used at most two vertically punched holes on each column to represent data using 24-bit words with four 6-bit characters unlike our current standard of data representation using 32-bit words with with four 8-bit characters. This compelled programmers to be very organized, extremely careful with their usage of memory and really hampered execution speeds. From this, the modern computer has greatly evolved to be what it is today, making things more efficient thanks to caching mechanisms.
One may be made to wonder, why does caching exist today? Well the answer is very simple, memory access is slow in comparison to CPU speed. As we know based off of Moore’s law, the number of transistors on a microchip roughly doubles every two years, leading to an exponential increase in computing power over time. Hence memory access speeds have to keep up if we’re to ensure seamless computation. In comes caching. Without caching, CPU cycles would be wasted waiting for data. There exist different levels of caches based on how fast they are, also known as a memory hierarchy. The memory hierarchy arranges different kinds of storage devices in a computer based on their size, cost and access speed, and the roles they play in application processing.
The levels of memory hierarchy range from L0, L1, L2, L3, L4 to cloud. The lower the level, the closer it is to the hardware. Registers (L0), are small amounts of high speed memory that are basically part and parcel of the CPU system, they are used to temporarily store data and instructions needed for quick access during processing. L1, L2 & L3 caches are hardware level caches and are very close to the CPU, they are sometimes referred to as SRAMs (Static Random Access Memory), Operating System-level caches such as disk buffers and swap memory are considered to be L3. There also exist Application-level caches (e.g. database query caching and in-memory stores like redis) which are considered to be the “highest level” of the caching hierarchy since they are implemented within application code itself. On an internet level, there exist Network & Cloud caching, this includes CDNs (Content Delivery Networks), edge computing and browser caches just to name a few.
You might then ask yourself, since there exist all these different types and levels of caches, how do they really improve computing and programming? For one, caching improves performance, it enables faster execution times in programs e.g. caching in compilers and web applications. In addition, caching improves scalability, this is done through cloud computing mechanisms which greatly enhances content delivery and API responses. Thirdly, caching improves efficiency, it reduces redundancy and workload immensely hence improving power consumption in data centres. Ultimately, programming today is more about efficient resource utilization, it’s all about how much we can get out of the scarce resources that are presently available, as opposed to early computing where memory was scarce.
Taking everything into account, caching in all its levels has really allowed computing to move from mechanical storage to new levels of AI-driven predictive models of caching. Currently, modern optimizations such as speculative execution, branch prediction, and AI-assisted caching in CPUs and cloud platforms are highly considered as mechanisms to optimize compute and performance while reducing the costs. Imagine debugging a punch-card based system with no cache, good luck sorting through the stacks of paper!