Instruments: Complete.json ... required "newline-delimited JSON" (NDJSON) or "JSON Lines (JSONL)" format

Developer API
1

Namaskara,

Complete Json file has entry as below (only few json objects selected… as the file is too big 48Mb unzipped).

[{“weekly”:false,“segment”:“BSE_FO”,“name”:“BANKEX”,“exchange”:“BSE”,“expiry”:1742927399000,“instrument_type”:“PE”,“asset_symbol”:“BANKEX”,“underlying_symbol”:“BANKEX”,“instrument_key”:“BSE_FO|845329”,“lot_size”:30,“freeze_quantity”:900.0,“exchange_token”:“845329”,“minimum_lot”:30,“asset_key”:“BSE_INDEX|BANKEX”,“underlying_key”:“BSE_INDEX|BANKEX”,“tick_size”:5.0,“asset_type”:“INDEX”,“underlying_type”:“INDEX”,“trading_symbol”:“BANKEX 52700 PE 25 MAR 25”,“strike_price”:52700.0},{“segment”:“BSE_EQ”,“name”:“KAKATIYA TEXTILES LTD.”,“exchange”:“BSE”,“isin”:“INE092E01011”,“instrument_type”:“X”,“instrument_key”:“BSE_EQ|INE092E01011”,“lot_size”:1,“freeze_quantity”:100000.0,“exchange_token”:“521054”,“tick_size”:1.0,“trading_symbol”:“KAKTEX”,“qty_multiplier”:1.0},{“segment”:“BSE_EQ”,“name”:“RAJKAMAL SYNTHETICS LTD.”,“exchange”:“BSE”,“isin”:“INE376L01013”,“instrument_type”:“X”,“instrument_key”:“BSE_EQ|INE376L01013”,“lot_size”:1,“freeze_quantity”:100000.0,“exchange_token”:“514028”,“tick_size”:1.0,“trading_symbol”:“RAJKSYN”,“qty_multiplier”:1.0},{“segment”:“BSE_EQ”,“name”:“NILKAMAL LTD.”,“exchange”:“BSE”,“isin”:“INE310A01015”,“instrument_type”:“B”,“instrument_key”:“BSE_EQ|INE310A01015”,“lot_size”:1,“freeze_quantity”:100000.0,“exchange_token”:“523385”,“tick_size”:5.0,“trading_symbol”:“NILKAMAL”,“short_name”:“NILKAMAL”,“qty_multiplier”:1.0},{“weekly”:false,“segment”:“BSE_FO”,“name”:“SENSEX50”,“exchange”:“BSE”,“expiry”:1738088999000,“instrument_type”:“CE”,“asset_symbol”:“SENSEX50”,“underlying_symbol”:“SENSEX50”,“instrument_key”:“BSE_FO|835992”,“lot_size”:25,“freeze_quantity”:1800.0,“exchange_token”:“835992”,“minimum_lot”:25,“asset_key”:“BSE_INDEX|SENSEX50”,“underlying_key”:“BSE_INDEX|SENSEX50”,“tick_size”:5.0,“asset_type”:“INDEX”,“underlying_type”:“INDEX”,“trading_symbol”:“SENSEX50 27000 CE 28 JAN 25”,“strike_price”:27000.0},{“segment”:“BSE_EQ”,“name”:“IRFC-7.75%-15-4-33-PVT”,“exchange”:“BSE”,“isin”:“INE053F08270”,“instrument_type”:“F”,“instrument_key”:“BSE_EQ|INE053F08270”,“lot_size”:1,“freeze_quantity”:100000.0,“exchange_token”:“974629”,“tick_size”:1.0,“trading_symbol”:“775IRFC33”,“qty_multiplier”:1.0},{“weekly”:false,“segment”:“NCD_FO”,“name”:“USDINR”,“exchange”:“NSE”,“expiry”:1740508199000,“instrument_type”:“CE”,“asset_symbol”:“USDINR”,“underlying_symbol”:“USDINR”,“instrument_key”:“NCD_FO|14280”,“lot_size”:1,“freeze_quantity”:10000.0,“exchange_token”:“14280”,“minimum_lot”:1,“tick_size”:0.25,“asset_type”:“CUR”,“underlying_type”:“CUR”,“trading_symbol”:“USDINR 89.75 CE 25 FEB 25”,“strike_price”:89.75,“qty_multiplier”:1000.0},{“weekly”:false,“segment”:“NCD_FO”,“name”:“USDINR”,“exchange”:“NSE”,“expiry”:1740508199000,“instrument_type”:“PE”,“asset_symbol”:“USDINR”,“underlying_symbol”:“USDINR”,“instrument_key”:“NCD_FO|14281”,“lot_size”:1,“freeze_quantity”:10000.0,“exchange_token”:“14281”,“minimum_lot”:1,“tick_size”:0.25,“asset_type”:“CUR”,“underlying_type”:“CUR”,“trading_symbol”:“USDINR 89.75 PE 25 FEB 25”,“strike_price”:89.75,“qty_multiplier”:1000.0},{“weekly”:false,“segment”:“NCD_FO”,“name”:“USDINR”,“exchange”:“NSE”,“expiry”:1740508199000,“instrument_type”:“CE”,“asset_symbol”:“USDINR”,“underlying_symbol”:“USDINR”,“instrument_key”:“NCD_FO|14282”,“lot_size”:1,“freeze_quantity”:10000.0,“exchange_token”:“14282”,“minimum_lot”:1,“tick_size”:0.25,“asset_type”:“CUR”,“underlying_type”:“CUR”,“trading_symbol”:“USDINR 90 CE 25 FEB 25”,“strike_price”:90.0,“qty_multiplier”:1000.0},{“weekly”:true,“segment”:“BSE_FO”,“name”:“SENSEX”,“exchange”:“BSE”,“expiry”:1739298599000,“instrument_type”:“CE”,“asset_symbol”:“SENSEX”,“underlying_symbol”:“SENSEX”,“instrument_key”:“BSE_FO|845331”,“lot_size”:20,“freeze_quantity”:1000.0,“exchange_token”:“845331”,“minimum_lot”:20,“asset_key”:“BSE_INDEX|SENSEX”,“underlying_key”:“BSE_INDEX|SENSEX”,“tick_size”:5.0,“asset_type”:“INDEX”,“underlying_type”:“INDEX”,“trading_symbol”:“SENSEX 74000 CE 11 FEB 25”,“strike_price”:74000.0},{“weekly”:false,“segment”:“NCD_FO”,“name”:“USDINR”,“exchange”:“NSE”,“expiry”:1740508199000,“instrument_type”:“PE”,“asset_symbol”:“USDINR”,“underlying_symbol”:“USDINR”,“instrument_key”:“NCD_FO|14283”,“lot_size”:1,“freeze_quantity”:10000.0,“exchange_token”:“14283”,“minimum_lot”:1,“tick_size”:0.25,“asset_type”:“CUR”,“underlying_type”:“CUR”,“trading_symbol”:“USDINR 90 PE 25 FEB 25”,“strike_price”:90.0,“qty_multiplier”:1000.0},{“segment”:“NSE_EQ”,“name”:“GTPL HATHWAY LIMITED”,“exchange”:“NSE”,“isin”:“INE869I01013”,“instrument_type”:“EQ”,“instrument_key”:“NSE_EQ|INE869I01013”,“lot_size”:1,“freeze_quantity”:100000.0,“exchange_token”:“21195”,“tick_size”:1.0,“trading_symbol”:“GTPL”,“short_name”:“GTPL Hathway”,“qty_multiplier”:1.0,“security_type”:“NORMAL”},{“weekly”:false,“segment”:“BSE_FO”,“name”:“BANKEX”,“exchange”:“BSE”,“expiry”:1738088999000,“instrument_type”:“CE”,“asset_symbol”:“BANKEX”,“underlying_symbol”:“BANKEX”,“instrument_key”:“BSE_FO|845336”,“lot_size”:15,“freeze_quantity”:900.0,“exchange_token”:“845336”,“minimum_lot”:15,“asset_key”:“BSE_INDEX|BANKEX”,“underlying_key”:“BSE_INDEX|BANKEX”,“tick_size”:5.0,“asset_type”:“INDEX”,“underlying_type”:“INDEX”,“trading_symbol”:“BANKEX 52700 CE 28 JAN 25”,“strike_price”:52700.0},{“segment”:“BSE_EQ”,“name”:“HDFCB-8.35%-13-5-26-PVT”,“exchange”:“BSE”,“isin”:“INE040A08500”,“instrument_type”:“F”,“instrument_key”:“BSE_EQ|INE040A08500”,“lot_size”:1,“freeze_quantity”:100000.0,“exchange_token”:“953957”,“tick_size”:1.0,“trading_symbol”:“835HDFCB26”,“qty_multiplier”:1.0},{“weekly”:false,“segment”:“NCD_FO”,“name”:“USDINR”,“exchange”:“NSE”,“expiry”:1740508199000,“instrument_type”:“CE”,“asset_symbol”:“USDINR”,“underlying_symbol”:“USDINR”,“instrument_key”:“NCD_FO|14284”,“lot_size”:1,“freeze_quantity”:10000.0,“exchange_token”:“14284”,“minimum_lot”:1,“tick_size”:0.25,“asset_type”:“CUR”,“underlying_type”:“CUR”,“trading_symbol”:“USDINR 90.25 CE 25 FEB 25”,“strike_price”:90.25,“qty_multiplier”:1000.0},{“weekly”:false,“segment”:“NCD_FO”,“name”:“USDINR”,“exchange”:“NSE”,“expiry”:1740508199000,“instrument_type”:“PE”,“asset_symbol”:“USDINR”,“underlying_symbol”:“USDINR”,“instrument_key”:“NCD_FO|14285”,“lot_size”:1,“freeze_quantity”:10000.0,“exchange_token”:“14285”,“minimum_lot”:1,“tick_size”:0.25,“asset_type”:“CUR”,“underlying_type”:“CUR”,“trading_symbol”:“USDINR 90.25 PE 25 FEB 25”,“strike_price”:90.25,“qty_multiplier”:1000.0},{“segment”:“BSE_EQ”,“name”:“PFSPL-13.50%-30-10-27-PVT”,“exchange”:“BSE”,“isin”:“INE514Q07296”,“instrument_type”:“F”,“instrument_key”:“BSE_EQ|INE514Q07296”,“lot_size”:1,“freeze_quantity”:100000.0,“exchange_token”:“975170”,“tick_size”:1.0,“trading_symbol”:“1350PFSP27”,“qty_multiplier”:1.0},{“segment”:“BSE_EQ”,“name”:“ZEE MEDIA CORPORATION LIMITED”,“exchange”:“BSE”,“isin”:“INE966H01019”,“instrument_type”:“B”,“instrument_key”:“BSE_EQ|INE966H01019”,“lot_size”:1,“freeze_quantity”:100000.0,“exchange_token”:“532794”,“tick_size”:1.0,“trading_symbol”:“ZEEMEDIA”,“short_name”:“ZEE MEDIA”,“qty_multiplier”:1.0},{“weekly”:false,“segment”:“NCD_FO”,“name”:“USDINR”,“exchange”:“NSE”,“expiry”:1740508199000,“instrument_type”:“CE”,“asset_symbol”:“USDINR”,“underlying_symbol”:“USDINR”,“instrument_key”:“NCD_FO|14270”,“lot_size”:1,“freeze_quantity”:10000.0,“exchange_token”:“14270”,“minimum_lot”:1,“tick_size”:0.25,“asset_type”:“CUR”,“underlying_type”:“CUR”,“trading_symbol”:“USDINR 88.5 CE 25 FEB 25”,“strike_price”:88.5,“qty_multiplier”:1000.0},{“weekly”:false,“segment”:“NCD_FO”,“name”:“USDINR”,“exchange”:“NSE”,“expiry”:1740508199000,“instrument_type”:“PE”,“asset_symbol”:“USDINR”,“underlying_symbol”:“USDINR”,“instrument_key”:“NCD_FO|14271”,“lot_size”:1,“freeze_quantity”:10000.0,“exchange_token”:“14271”,“minimum_lot”:1,“tick_size”:0.25,“asset_type”:“CUR”,“underlying_type”:“CUR”,“trading_symbol”:“USDINR 88.5 PE 25 FEB 25”,“strike_price”:88.5,“qty_multiplier”:1000.0},{“weekly”:false,“segment”:“NCD_FO”,“name”:“USDINR”,“exchange”:“NSE”,“expiry”:1740508199000,“instrument_type”:“CE”,“asset_symbol”:“USDINR”,“underlying_symbol”:“USDINR”,“instrument_key”:“NCD_FO|14272”,“lot_size”:1,“freeze_quantity”:10000.0,“exchange_token”:“14272”,“minimum_lot”:1,“tick_size”:0.25,“asset_type”:“CUR”,“underlying_type”:“CUR”,“trading_symbol”:“USDINR 88.75 CE 25 FEB 25”,“strike_price”:88.75,“qty_multiplier”:1000.0},{"we

The entire file is just one line !! That means If i read into a buffer, I have to read the entire file in one go… std::getline(file, this->scratch_pad) … the entire file is read…
that is humongous…

Now worst part is I am now forced to work on char by char … to build a json object and work on it and then proceed to the next json object (char by char)..
This is the most inefficient way to process json information… till date I never worked this way.

The simple fix is to add a new line char (‘\n’) at end of each complete json object. So that I read a json object at time and process the file accordingly.

Please for heavens sake introduce ‘\n’ (newline, unix style) character at end of each valid json object …

eg: {“segment”:“BSE_EQ”,“name”:“KAKATIYA TEXTILES LTD.”,“exchange”:“BSE”,“isin”:“INE092E01011”,“instrument_type”:“X”,“instrument_key”:“BSE_EQ|INE092E01011”,“lot_size”:1,“freeze_quantity”:100000.0,“exchange_token”:“521054”,“tick_size”:1.0,“trading_symbol”:“KAKTEX”,“qty_multiplier”:1.0}, \n

This is often referred to as “newline-delimited JSON” (NDJSON) or “JSON Lines (JSONL)” format.

On, linux if you execute the following command: cat complete.json | jq -c ‘.’ > output.ndjson

You will get NDJSON format. Just have a look and find out how easy its to process this file after conversion to NDJSON.

{“weekly”:false,“segment”:“BSE_FO”,“name”:“BANKEX”,“exchange”:“BSE”,“expiry”:1742927399000,“instrument_type”:“PE”,“asset_symbol”:“BANKEX”,“underlying_symbol”:“BANKEX”,“instrument_key”:“BSE_FO|845329”,“lot_size”:30,“freeze_quantity”:900.0,“exchange_token”:“845329”,“minimum_lot”:30,“asset_key”:“BSE_INDEX|BANKEX”,“underlying_key”:“BSE_INDEX|BANKEX”,“tick_size”:5.0,“asset_type”:“INDEX”,“underlying_type”:“INDEX”,“trading_symbol”:“BANKEX 52700 PE 25 MAR 25”,“strike_price”:52700.0}
{“segment”:“BSE_EQ”,“name”:“KAKATIYA TEXTILES LTD.”,“exchange”:“BSE”,“isin”:“INE092E01011”,“instrument_type”:“X”,“instrument_key”:“BSE_EQ|INE092E01011”,“lot_size”:1,“freeze_quantity”:100000.0,“exchange_token”:“521054”,“tick_size”:1.0,“trading_symbol”:“KAKTEX”,“qty_multiplier”:1.0}

The coding to make change from JSON output to NDJSON output: Less than 1 minute.

Regards
Rathnadhar K V

2

That one missing ‘\n’ char at end of every JSON object, resulted in excessive burning of CPU power just convert json to ndjson at client end.

Please, in future, think of end scenarios and how every design decision has an impact on downstream software.

Debate in detail before implementation.

Perhaps put it up for community review so that you can best brains to work on your design decisions for free.