How to set this up in best way - batch php file

Server Config
1

I would like to call one of the API from Pokemon using PHP every minute and store the result in either database OR JSON file. when visitors come to my site and visit one of my pages i would like them to read this JSON file or DB and display results in the website. I would like that page to read this file every couple of minutes. The way I am thinking of doing this

batch file that has infinite loop which calls PHP (which calls pokemon api) every minute
JS file on one of my pages that reads the file or DB every 2 minutes

I think DB option would be better because DB will figure out issues about access to pokemon stored data. What if JS tries read at the same while record is being inserted or file is being read?

Can anyone share options please

1 Like
2

A cron job can be used to run a PHP script at regular intervals on your server. Though I’m not sure that’s the best approach for your use case. It really depends upon how you need this to work.
Does it need to run every minute, or could you update on demand?
You could just make the API call on page load when a user makes the server request. or do you need it to update while the user in still on the page? It may be something done better with javascript, to update in realtime. But I’m still not sure exactly what it is you want to do.

2 Likes
3

I cant update on demand because JS (visitors on my site) they can be 100s of that that they can visit that page and I am limited how many times I can call API. My idea is for my PHP to call API every minute which means 60 calls per minute which is below my limit. Does that answer your question?

5

It would still be better to do it on demand, like this:

if user requests data then
	if my copy of data is older than 1 minute then
		call api and update my copy in db
	return local copy of data

That way, you api calls are only as busy as your users, and you’re not calling the api every minute during the middle of the night, etc.

1 Like
6

I need to rewrite the data into the file or DB after every API call. so for example

12:01 call api and store item A in the file or DB
12:02 call api and store item B over item A
12:03 call api and store item C

so at 12:03 there is only item C in the file

7

What you are looking for is cache. You can make the request every time a user visits, but then you store the result in a cache. As long as the cache is fresh it will be served, and when it goes stale you fetch a fresh copy from the API. If you set the cache to cache for at least minute you will remain within api limits.

The one thing you may need to take care of is a cache stampede, where your cache is stale and you get a large number of visitors all of a sudden so you do an API request for each of them (as they’ve all concluded at the same time the cache is stale and none of them has refilled it yet). There are solutions for this though. For example the Symfony Cache component has cache stampede protection. See https://symfony.com/doc/current/components/cache.html

This is easier to implement and doesn’t waste loads of cpu cycles like doing a update each minute would.

2 Likes
9

@rpkamp How is that different from just calling API from client every minute?

I call from client and display results and you are saying display, store in cache and then after 1 minute is up call again? I don’t see the difference. Can you maybe try to explain it in different way please. How is storing in cache saving number of API calls?

10

Well, if you have visitors every minute of every day then I suppose the result is the same. But if there are periods with less traffic, such as the middle of the night, you’ll save a lot of work for your server, as nobody is looking at the data so there is no need to have a fresh copy.

Besides for cron jobs you have to worry about things like

  1. What if the server is reboot during a cron run?
  2. What if for some reason a cron run takes longer than a minute and the second already starts and you have two running concurrently?
  3. You can prevent the previous with a lock, but what if a cron run gets the lock and never finishes? The whole system is locked.

Cron jobs sound simple, but you get a boat load of accidental complexity.

11

@rpkamp have some logic in that does x many calls during the opening time but after 6pm I call it less often. I have about 150-200 visitors per day spread out throught the day so i thought if i write it to a file on server and then have client read from the file.
Your point #2 - when I said cron job I guess I used that term loosely. In my testing environment (XAMPP) I run a batch script that calls PHP code every minute. this PHP codes calls API and stores returned data in a file. File is in the same location where my web pages are. This is not confidential data.