How to parallelise external API calls with Elixir to make them super fast
I'm working on a Phoenix application which uses the Instagram API via the Elixtagram library. There is a particular page where I need to display a bunch of users who need to be retrieved from the API... Here's the "before" (sequential) code:
def users_list(%Account{favourite_users: users}) do
Elixtagram.configure
users
|> Enum.map(fn(u) -> Elixtagram.user(u.ig_id).username end)
|> Enum.join(", ")
end
What I'm doing here is simply passing in a list of Instagram IDs and returning a string of comma separated usernames. The obvious problem here is that Elixtagram is making an API call which can take some relatively large amount of milliseconds, and it's silly to wait for each one to finish before starting the next one, especially when we're using a language with such easily accessible concurrency as Elixir.
Here's the "fixed" code:
def users_list(%Account{favourite_users: users}) do
Elixtagram.configure
users
|> Enum.map(fn(u) -> Task.async(fn -> Elixtagram.user(u.ig_id).username end) end)
|> Enum.map(&Task.await/1)
|> Enum.join(", ")
end
The difference here is that instead of blocking for each request in the Enum.map, I'm filling the list up with async tasks, which I then wait for on the next line. What this means is that the amount of requests we make here has little impact on the performance of the code (until we saturate the network...).
Good luck with your concurrency adventures, friends!