I’m trying to figure out how to configure pagination when connecting to external data source. I’m working with Prey API and need to pull in all records or iterate through the pages to get all record to show up on one table. Prey max limit is 100 record per page. How would I configure the model so that it iterate the pages and pull all records?
I don’t have a Prey account to test with, but based on their documentation I believe you will want to set the ‘Offset parameter’ field in the model options to ‘page’. You can then explicitly call out the page size in the Data source URL, like
devices?page_size=100
Querying with no limit should then repeatedly send requests with the ‘page’ URL parameter iterating until all records are retrieved. Let me know if you get error messages with this, I can’t test it directly myself but I’m happy to help troubleshoot as necessary!
Thank you for the reply! @benjamin.tevelow
I set it up as you suggested and no success with iterating through the pages. It is able to pull 100 record through the Data source URL. And only seem to pull just records from the 1st page.
I tried inputting the following in the ‘Offset parameter’ field:
page
page=
page=1
page=2
‘page’
@benjamin.tevelow Are there any more suggestions you can add?
Hi @Bobbyvil -
Sorry for the delayed response. Looking at your set up, I think my initial suggestion isn’t able to function the way I thought it would, and it may need to be more explicitly set to query with a limit, and then increment the paging. Could you try the following:
Delete the `page-size` query parameter from your Data Source URL.
Set the `Limit Method` drop-down to ‘Single URL Parameter’
Set the `Limit Parameter` field to ‘page-size’ and ‘Max number of records’ to 100
This will still only pull in the first 100 records on page load, but you should then then be able to iteratively call ‘Load more records’ in order to pull in the next set of 100. If you need this to happen automatically, it will probably need an event driven action sequence.
Unfortunately, all APIs function a little differently, so it can be hard to predict behavior. I’m looking around for one that will work similarly to Prey that I can use to experiment with. If you don’t mind, try out the above and see how it goes, and I can keep poking around from there.
Hi @benjamin.tevelow - Thank you for you reply!
I tried what you suggested and there is no ‘Load more’ than the 100 records.
I tried setting the Pagination method to None and to Single URL parameter with the following :
page
page=
page=2
Left it blank
From Postman: This appears at the end of the JSON when calling device list. I don't know if I should use the href somewhere in Skuid.
Also want to point out that with the Limit parameter, when I set the Max Record to 30, I do see the ‘Load More’. However, when I click load more, nothing loads and remains at 30 records. I’m wondering if this related to as well I can’t pull in more than 100 records.