Python Concurrency: Understanding asyncio

in

Alright! Let’s talk about how to limit concurrency with Python asyncio. Concurrency refers to the ability of our program to handle multiple tasks at once without slowing down performance. However, sometimes we want to limit the number of parallel tasks that can run simultaneously for various reasons like resource constraints or avoiding overloading a system.

To achieve this in Python using asyncio, we use semaphores. A semaphore is a synchronization primitive that allows us to control access to shared resources by limiting the number of threads or processes that can acquire them at any given time. In our case, we’re going to limit the number of parallel tasks that can run simultaneously using asyncio’s built-in `Semaphore` class.

Here’s an example:

# Import the necessary libraries
import asyncio # import the asyncio library for asynchronous programming
from random import randint # import the randint function from the random library
from time import sleep # import the sleep function from the time library

MAX_CONCURRENT = 3 # set the maximum number of concurrent tasks to 3
semaphore = asyncio.Semaphore(MAX_CONCURRENT) # create a semaphore object with the maximum number of permits set to MAX_CONCURRENT

async def my_task():
    await semaphore.acquire() # acquire a permit from the semaphore before proceeding
    print("Starting task...")
    sleep(randint(1, 5)) # simulate some work being done for this task
    print("Task finished!")
    semaphore.release() # release the permit when we're done with it

tasks = [my_task() for _ in range(10)] # create a list of tasks to run
loop = asyncio.get_event_loop() # create an event loop
for task in tasks:
    loop.run_until_complete(task) # start running the tasks one by one, with at most MAX_CONCURRENT running simultaneously

In this example, we set `MAX_CONCURRENT` to 3 and create a semaphore object using `asyncio.Semaphore()`. We then define our task function called `my_task`, which acquires a permit from the semaphore before proceeding with its work (in this case, simulating some random amount of time between 1 and 5 seconds). After finishing its work, it releases the permit using `semaphore.release()`.

We create a list of tasks to run called `tasks`, which is populated by calling our task function for each item in a range (in this case, we’re creating 10 tasks). We then start running these tasks one by one using the event loop and `loop.run_until_complete()`. The semaphore ensures that at most `MAX_CONCURRENT` tasks are running simultaneously, preventing any resource contention or overloading issues.

Note: If you’re new to asyncio, it may be helpful to read our previous guide on Python concurrency using asyncio before diving into this one.

SICORPS