Hello,
Upstox provides two APIs: one for intraday and one for historical data requests.
This design although has merits at other places, it has a serious disadvantage when it comes to downloading Equity Options.
Consider this:
Case #1:
These are all the Equity Derivative symbols currently issued by NSE (as of Mar 2, 2024)
Equity Option contracts: 26680
Case #2:
If we only consider current expiry, still the number of instruments is not small:
Equity Option contracts: 12093
Now, we need to query 2 times for the same instrument. One on intraday API and other on historical API.
Below is how the numbers look:
Case #1: 26680 * 2 => 53360 queries
Case #2: 12093 * 2 => 24186 queries
If we consider each request is served in a second, considering rate limit and network latency, we are talking about 53360 seconds that is 14h 49m 20s!!! In other case it is 6h 43m 6s!!
This is insane and querying for that long is not even possible in first case because the data clears up at midnight.
A very simple solution can save us all from this insanity.
Upstox must merge all intraday data into historical as soon as market closes and the close price is published by exchange. NSE publishes it to be max by 15:40 IST, so starting around 15:45 IST on safer side. Considering data is merged in next 15mins, users should be able to query starting 16:00 IST in one go!
This will help bring down the number of queries by half - not just on expiry days, but everyday!
Upstox can choose NOT to clear data from their intraday DB, so this change is backward compatible and someone who is only interested in intraday data, can still query it alone.
Benefit: Upstox and users both save considerable computing resources and time. Greener for planet as computational resources would be used efficiently.
Another logic: the trading day has finished, so the data is actually historical 
An alternative can be if Upstox provides the data using FTP or in csv/json files. This will save a lot of hassle for people who are interested in back tests.
Regards,
SRK