03 September 2009

Converting log files into csv

Here is how I converted a log file contains lines like

type:Trade, clientOrderId:20634240922, marketOrderId:bsqj0, marketEventId:ccpfv, clientEventId:ccpfv, status:PARTIAL_FILL, orderParameters:[market:RPR, party:MYCO, instrument:USDCAD, side:BUY, quantity:1000000.0, minimumQuantity:10000.0, maximumShowQuantity:1000000.0, price:1.0975700000000002, type:LIMIT, quoteType:INDICATIVE, timeInForce:GTC, properties:null], transactDateTime:2009-08-27T00:00:15.712Z, receivedDateTime:2009-08-27T00:00:15.712Z, openQuantity:980000.0, executedQuantity:20000.0, averageExecutedPrice:1.0975700000000002, marketTradeId:c4v, clientTradeId:null, tradeQuantity:20000.0, tradePrice:1.0975700000000002, counterparty:SOMECO, settlementDateTime:2009-08-28T00:00:00.000Z, taker:false, comments:null

into a csv file that looked more like:

c4v, 20634240922, 2009-08-27T00:00:15.712Z, MYCO, USDCAD, BUY, PARTIAL_FILL, 20000.0, 1.0975700000000002,

tail -F mylogfile.log | sed 's/.*, clientOrderId:\(\S*\),.*, status:\(\S*\),.*, party:\(\S*\), instrument:\(\S*\), side:\(\S*\),.*, transactDateTime:\(\S*\),.*, marketTradeId:\(\S*\),.*, tradeQuantity:\(\S*\), tradePrice:\(\S*\).*/\7, \1, \6, \3, \4, \5, \2, \8, \9/'