Concurrency is the ability of different parts or units of a program, algorithm, or problem to be executed out-of-order or in partial order, without affecting the outcome. This allows for parallel execution of the concurrent units, which can significantly improve overall speed of the execution in multiprocessor and multicore systems.
Parallel computing is a type of computation in which many calculations or processes are carried out simultaneously. Large problems can often be divided into smaller ones, which can then be solved at the same time.
20 years ago, CPUs could only do one thing at a time. However, computers still ran multiple applications in parallel. But how?
A thread is an independently schedulable entity. It allows running code concurrently (but not parallelly).
This Fake Parallelism consists of doing a bit of a task, then switch to a different one. Rinse and repeat.
From the perspective of a program, this looks like we are running in parallel. From the perspective of a user, it also looks like parallel, as computers can do basic operations really fast. However, it is actually not doing things in parallel.
Computers nowadays are truly capable of parallel computations… but they still fake it a lot. This is because there are a lot more applications (hundreds) to be run parallelly than cores in the computers.
And, when do we switch? How do we switch? The CPU has a hardware mechanism based on timers to do this action of interrupting a task and switching to another one.
What is involved in switching tasks? Switching from one to another tasks takes time (same as humans do). We need t:
This is an expensive operation, hence there is some difficulty to decide how much time it should take for switching. The more we switch, the more expensive it is, and the less we switch, the less “parallel” it looks like.
Not everything can be done in parallel: too expensive! Divide-and-conquer algorithms are usually easy algorithms to parallelize.