Double tick data in a single timestamp - attached jpg

Developer API
1

1000083140
1000083140317Ɨ191 27.6 KB

As you can see in the image. I use websocket to make my own Time and sales database. Alot of times, i get double tick with same quantity and price twice or sometimes three times in a row with same timestamp. Sometime it repeats big quantites like 5000, two times in same timestamp which cause large fluctuations in analyzed data. Please fix this issue or provide valid soultion to this solution. Thank you and have a nice day.

2

Hi @Daljit_singh
It sounds like youā€™re getting duplicate tick dataā€”sometimes two or three timesā€”with the same price, quantity, and timestamp from our websocket feed. This can definitely cause your analyzed data to fluctuate, especially with large quantities. This issue often comes down to how your application processes the data on your end, and client network conditions can also play a significant role.

The best way for you to fix this is to add deduplication logic to your system. Before you store any tick, just check if an identical entry (same price, quantity, and timestamp) is already there. If it is, simply discard the new one. Weā€™re also checking our backend systems and network setups to make sure weā€™re not sending duplicates from our side.

Thanks!

3

Thank you ketan sir for your speedy response. I will implement your logic in my code. One thing more that i we can use, as your websocket provide tick data in milliseconds, we can include the full ltt (in ms) to ensure no valid traded are missed. Is that way better then deduplication?

4

@Daljit_singh I would like to add to my previous comment: I have confirmed with our internal team that when no trades occur within a given timestamp tick difference, the same tick is returned to the user. Please plan your data usage accordingly.

Thanks!

5

Removed comment

6

@Daljit_singh
Just to clarify, the above message was posted by meā€”I accidentally used my other account while replying in a hurry.

Thanks!

7

Sorry for misunderstanding ketan sir. I will remove my comment. Thank you for your service. :slight_smile:

1 Like
8

@Daljit_singh I Also faced this issue, the websocket feed has two important timestamp information that helped me filter out thingsā€¦

currentTs : The timestamp of the tick itself, i use it to filter the market hours live ticks

tick_ts_ms = int(data['currentTs'])
# Convert tick ts (e.g. 1746157500000) to datetime (rounded to minute)
tick_ts = tick_ts_ms / 1000  # e.g., 1746157561414
tick_dt = datetime.fromtimestamp(tick_ts)

#Skip Before 9:15:00
if tick_dt.hour < 9 or (tick_dt.hour == 9 and tick_dt.minute < 15):
     return
            
# Skip if after 15:30:01
if tick_dt.hour > 15 or (tick_dt.hour >= 15 and tick_dt.minute >= 30):
     return

ltt : The last traded time of the instrument.

I keep storing the the last processed ltt for instruments and only if the ltt changes i process the tick. This helps me avoid duplication in my dataā€¦also helps in less storage usage.

ltt = int(ltpc.get("ltt", 0))
second_ts = ltt // 1000
last_sec = last_processed_second[instrument_key]
if last_sec != -1 and second_ts > last_sec:
        #Process here
        store_data()

#Update the last processed second
last_processed_second[instrument_key] = second_ts

There is one more check you can make, some instruments may remain illiquid for the day and may not have a ltt of today, you may want to avoid them also. So from one of my initial tick, I set a variable current session date. If the tick date for today doesnt match with the date from ltt i do not process that data for that intrument.

ltt_dt = datetime.fromtimestamp(ltt / 1000).date()
if ltt_dt != current_session_date:
        #Stale Tick, skip...
        continue
2 Likes
9

Thank you soo much Raghav for the information you provided. I will implement your logic in my own code as well. Love to Upstox community and developers

2 Likes