Maximum permissible number of instrument keys for fetching live market data via websockets connection

Developer API
1

Hello, I tried fetching the live market data of 8k+ instruments in one go and got the bellow error:
websockets.exceptions.PayloadTooBig: Uncompressed payload length exceeds size limit (? > 1048576 bytes)
I tried to traceback the error and went through protocol.py, framing.py and permessage_deflate.py files but didn’t understand anything. Request you to help me in finding out the maximum permissible number of instrument keys for fetching live market data via websockets connection.

2

Please keep the max count at 100 instruments at a time. We are internally discussing increasing the max count with the concerned team. We will inform you once we make progress on it.

3

Noted. Thanks for support!

4

Hello Pradeep,

As you suggested to subscribe to 100 instruments at a go so I am planning to run a loop that subscribes to instruments in batches of 100. I want to know is the server providing the WebSocket service have its own limitations or restrictions. Do you recommend adding delay in between the batches?

5

Hello, Pradeep I am using below code for subscribing to multiple instruments at a time.

# Function to create batches of instrument keys
def chunks(lst, n):
       for i in range(0, len(lst), n):
yield lst[i:i + n]

# Function to subscribe to instruments
async def subscribe_to_instruments(websocket, instruments):
data = {
    "guid": "someguid",
    "method": "sub",
    "data": {
        "mode": "option_chain",
        "instrumentKeys": instruments
    }
}
binary_data = json.dumps(data).encode('utf-8')
await websocket.send(binary_data)

# Function to subscribe to all instruments in parallel
async def subscribe_to_all_instruments(websocket, instruments):
tasks = [subscribe_to_instruments(websocket, batch) for batch in chunks(instruments, BATCH_SIZE)]
await asyncio.gather(*tasks)

async def fetch_market_data():
try:
    ssl_context = ssl.create_default_context()
    ssl_context.check_hostname = False
    ssl_context.verify_mode = ssl.CERT_NONE
    configuration = upstox_client.Configuration()
    api_version = '2.0'
    configuration.access_token = access_token

    response = get_market_data_feed_authorize(api_version, configuration)

    async with websockets.connect(response.data.authorized_redirect_uri, ssl=ssl_context) as websocket:
        print('Connection established')
        await asyncio.sleep(1)

    # Subscribe to all instruments in batches
    await subscribe_to_all_instruments(websocket, instrument_keys)    

    while True:
        message = await websocket.recv()
        decoded_data = decode_protobuf(message)
        data_dict = MessageToDict(decoded_data)
        print(json.dumps(data_dict))
        # Call the provided callback function with the data
        # await callback(data_dict)

except websockets.exceptions.ConnectionClosedError as e:
    logging.error(f"Connection closed unexpectedly: {e}")
except websockets.WebSocketException as e:
    logging.error(f"WebSocket exception: {e}")
except upstox_client.ApiException as e:
    logging.error(f"Upstox API exception: {e}")
except Exception as e:
    logging.error(f"An unexpected error occurred: {e}")

I am getting below error:
ERROR:root:WebSocket exception: code = 1000 (OK), no reason

Request you to help me with this. Thank you!

6

BATCH_SIZE=100.
Since over the websocket we have the capacity to subscribe up to 100 instrument_keys.