Book a Call
Get a Quote

Unlocking the Power of Dynamic Resources in Retool - Resource Pooling

Lander Audian
October 7, 2024
15 min read
Unlocking the Power of Dynamic Resources in Retool - Resource Pooling

In Retool, dynamic resources open up powerful possibilities by allowing developers to seamlessly swap between different resources—such as REST APIs—based on logic or credentials. Imagine having multiple API credentials for different environments or users. With dynamic resources, you can easily switch between them without duplicating configurations.

In this post, we'll explore how dynamic resources work, use cases, and tips to maximize efficiency in your Retool apps.

What Are Dynamic Resources?

Dynamic resources enable flexibility by allowing your Retool app to dynamically choose which resource to call, based on logic you define. These resources can be anything supported by Retool, such as databases, REST APIs, or even Google Sheets.

For instance, if you have multiple API credentials (for different clients or environments), you can define multiple REST API resources in Retool and switch between them as needed.

IMPORTANT:  One thing to remember is that once a query is set to run dynamic resources, it will determine the current type of the resource - if it’s either a REST API, Retool DB, Google Sheets, or something else. Once the query is saved with that type of resource, you should ONLY switch with the resource IDs that has the same type!

What Are Dynamic Resources?

How Dynamic Resources Work

To utilize dynamic resources in Retool, you first define multiple resources, such as REST APIs, databases, or other integrations. You then implement control logic or use variables within your app to determine which resource to call for a particular action. At runtime, Retool dynamically selects and uses the appropriate resource based on this logic, allowing you to switch between resources seamlessly without the need to manually adjust settings.

Some Use Cases of Dynamic Resources

Environment-Based API Calls

When developing in multiple environments (development, staging, production), switching between different API credentials is crucial. With dynamic resources, you can set up separate REST APIs for each environment and switch between them depending on your app’s state or user selection.

Load-Balancing Requests Across Multiple APIs

If you have multiple APIs providing the same data in different regions, dynamic resources in Retool allow you to route requests to the closest API, reducing latency. This helps distribute traffic evenly, improving performance and preventing any one server from being overloaded.

Pooling resource credentials to overcome API rate limits

Pooling credentials involves rotating between multiple sets of API credentials to distribute the load of requests and avoid hitting the rate limits imposed by an API provider. This approach can be useful when working with services that limit the number of requests per account within a certain time frame. By dynamically switching between different credentials, you can ensure that your app maintains consistent performance even when faced with these constraints.

Resource Pooling

If you're working with APIs that impose strict rate limits, pooling credentials can help distribute your requests across multiple accounts or API keys.

Think of it like a busy supermarket with multiple cashiers, where each cashier can only handle a limited number of customers per hour. If you have a long line of customers waiting to check out, relying on just one cashier would quickly hit a bottleneck. However, by opening more cash registers (each representing a different cashier), you can spread the workload, allowing more customers to check out at the same time.

Resource Pooling

In the same way, when you're working with APIs that have strict rate limits, each API key is like a cashier with a limited capacity. By pooling credentials (like opening more cash registers), you can distribute your API requests across multiple accounts or API keys, ensuring that no single key hits its limit too quickly, and your "customers" (requests) keep flowing smoothly.

I guess you might say, isn’t that also like load balancing already? Well yeah, the GIF above also works for load balancing, but here’s some key differences of load balancing vs resource pooling:

Key Differences:

  • Load balancing
    • Focuses on distributing requests efficiently across multiple active resources or servers.
    • It’s about handling concurrent requests efficiently across multiple identical resources to avoid overload.
    • For managing capacity—ensuring no server or instance is overwhelmed.
  • Pooling
    • Focuses on reusing or rotating between limited resources (such as a fixed set of API keys) to maximize utilization and avoid hitting API rate limits.
    • Managing a limited set of resources to avoid exhausting or overusing them.
    • For managing scarcity—making sure a finite resource (like an API key) doesn’t hit the limits quickly as it would when there’s only one or a few resource .

Now, let’s see how we can pool multiple resources in Retool.

Set the query to use dynamic resource.

We can set a query to be running a resource dynamically by clicking the button with the ƒ symbol beside resource selection dropdown.

Set the query to use dynamic resource

Creating multiple API resources in Retool.

Before you start, you'll need to create the pool of resources you’ll be using, which is a separate API resource for each set of credentials you plan to use.

  1. Go to Resources in Retool.
  2. Click Create New and select REST API.
  3. Configure the API resource using the first set of credentials or keys. (API key, authentication method, etc.).
  4. Repeat steps 2-3 for additional API credentials, while giving each resource a unique and descriptive name (e.g: Resource1, Resource 2, …, Resource 10)

IMPORTANT: To overcome API rate limits, we basically need to have a set of resources which are REST APIs with same base URL endpoint. These REST API resources will be making requests to the endpoint, only having different credentials or keys configured for each, thus, a pool of resources.

Getting the resource ID of the resources you’ll be using.

While Retool still doesn’t have a simple way to view resource ID or the list of resources’ IDs, a way we could get them currently is by going inside any app, adding the query to use the resource we want to see the resource ID of and simply clicking the the button with the ƒ symbol (Basically, how we set them to use dynamic resource).

Getting the resource ID of the resources you’ll be using

IMPORTANT:  We must save the resource IDs somewhere, either in a variable, transformer, config variable or whichever we prefer to store it in to pull up the ids later on.

Utilizing config variables is a good way to add or remove resources without having to go in the app.

Utilizing config variables is a good way to add or remove resources without having to go in the app

ℹ Setting a resource ID automatically detects the resource type upon saving the query. So make sure that we are saving the correct resource IDs, which are the REST API resources we’ve just created.

Setting up resource ID rotation logic.

To dynamically switch between resources, we will use a variables to track the important states within the app we are building, such as current resource ID . Each time you make a request, the states will be updated.

  1. Create a variable to store the current resource ID.  (For this guide - var_resourceId)
Setting up resource ID rotation logic

We must use a state variable and not additionalScope as it’ll not work for resource id switching.

  1. Create another variable for the current index (For this guide - var_currentIdx, default = 0). This will be the index to reference the resource IDs from the list.  (From the list of resource IDs we stored, from an earlier step.)
  2. Write the logic to rotate between the resource IDs. Each time a request is made, the state variables will be updated to point to the next resource in the list

Two ways in which we could set the rotation of the resource IDs are:

  • Moving on to the next resource ID in the list every time
const nextIndex = (currentIndex + 1) % resources.length
  • Getting a random resource ID from the list every time.
const nextIndex = Math.floor(Math.random() * resources.length)

Code Preview of the Resource Rotation Flow

Here’s a simple pseudocode to preview of how the entire logic flow works:

  • Defining the list of resource IDs saved
// LIST OF RESOURCE IDS 
// you can pull them from anywhere you stored it in from earlier step.
/* 
when using config vars, you have to convert them to array as config vars 
doesn't support arrays. (e.g) assuming they are strings separated by comma: 
configVars.RESOURCE_LIST.split(',') 
*/

const resources = ['resourceId1', 'resourceId2', 'resourceId3', ..., 'resourceId10']

	// OR when using environment config variables, assuming the list is named RESOURCE_LIST
	
const resources = retoolContext.configVars.RESOURCE_LIST.split(',')

Written in a JS query in  Retool

  • Logic for rotating resource IDs to be used, using var_resourceId and var_currentIdx
// CURRENT RESOURCE ID
const currentIndex = var_currentIdx.value //0
await var_resourceId.setValue(resources[currentIndex]) //resourceId1

// PREPARING NEXT RESOURCE ID TO USE

	// 1st method - Moving on to the next resource ID in the list every time.
	const nextIndex = (currentIndex + 1) % resources.length //1
	
				//OR
				
	// 2nd method - Getting a random resource ID from the list every time.
	const nextIndex = Math.floor(Math.random() * resources.length) //random 
	
// ASSIGN NEXT INDEX TO BE USED FOR NEXT RUN
await var_currentIdx.setValue(nextIndex) //1

//... Finally, run the dynamic query in an event handler. 

Written in a JS query in  Retool

Finally, with the value  of var_resourceId pointing to the correct resource ID we can run the dynamic query in the event handler OR, if we want to run the query within the code block above instead of in an event handler, we can use .trigger() at the end but we’ll need to have the following setting checked for the setting of variable values to take effect first:

Keep variable references inside the query in sync with your app

Toggle to keep the variables we used .setValue() with, to use the new value after that line in the JS query. (Can be found in “Advanced” tab)

Here’s a preview of the code above adjusted for that:

// CURRENT RESOURCE ID
const currentIndex = var_currentIdx.value //0
await var_resourceId.setValue(resources[currentIndex]) //resourceId1

// RUN THE QUERY HERE
// ...

// PREPARING NEXT RESOURCE ID TO USE

	// 1st method - Moving on to the next resource ID in the list every time.
	const nextIndex = (currentIndex + 1) % resources.length //1
	
				//OR
				
	// 2nd method - Getting a random resource ID from the list every time.
	const nextIndex = Math.floor(Math.random() * resources.length) //random 
	
// ASSIGN NEXT INDEX TO BE USED FOR NEXT RUN
await var_currentIdx.setValue(nextIndex) //1

Written in a JS query in  Retool

Here’s a shorter one that works using random method (with variable sync setting enabled):

In this version we don’t even need a variable for the current index as we’ll be using a random index each time, hence no need to save current or next index.

// CURRENT RESOURCE ID
const randomIndex = Math.floor(Math.random() * resources.length)
await var_resourceId.setValue(resources[randomIndex]) // random resourceId

// RUN THE QUERY HERE
// ...

Written in a JS query in  Retool

That’s some of the few ways for the flow or cycle of pooling REST API resources to be used on each dynamic query run.

There are certainly more ways to pool resources in Retool other than the one above, it’s just a matter of one’s creativity in regards to implementation as Retool allows limitless methods or ways of implementing something.

More Tips on Using Dynamic Resource in Retool

  • Organize Resources with Clear Naming: Adopt consistent and descriptive naming conventions for your resources to prevent confusion when switching between them. You can use other resources than just REST API which you can run dynamically so better to have them properly organized and named.
  • Leverage State Management in Retool: Utilize Retool's state variables to store the current resource selection. This strategy enables dynamic changes to the resource in use without modifying the application’s core logic.
  • Error Handling for Unavailable Resources: Implement robust error handling to manage scenarios where a resource may be unavailable. This includes retry logic to use another available resource, providing fallback logic or user-friendly error messages to maintain application stability.
  • Scaling for Multiple Users or User Groups: When building applications for multiple users, use dynamic resources to distribute requests across different resources. By assigning different users or groups of users to different resources, you can ensure that the app scales efficiently and avoids performance slowdowns as the number of users increases. This also allows for better resource allocation and handling of higher traffic volumes without overloading any single resource.

Conclusion

This adaptability not only streamlines workflows but also enhances the application's ability to handle various scenarios efficiently. With the ability to dynamically switch between different resources based on user needs or system conditions, applications become more resilient.  As developers increasingly embrace these dynamic capabilities, they can build solutions that ultimately leads to a better user experience and improved application reliability.

You’ve reached the end of this blog. I hope this was helpful and has given you some useful insights!

Thanks for sticking around! Till next time!

Get in Touch

Ready to bring your dashboard vision to life? Contact Retoolers today, and let us help you create a powerful, intuitive dashboard that meets your exact requirements.

Lander Audian
Retool Developer