Python Multiprocessing

I created a buffered sender and receiver application to demonstrate multiprocessing in Python. The sender computes some fictitious data and sends it to the receiver which writes it to a file. The data is sent when the buffer is filled up. The receiver is slow (due to a delay) and it demonstrates how multiprocessing can allow you to acquire data from a fast source without missing any data and write it to file! You can use similar ideas to write data acquired from a hardware device to the hard-disk without keeping the device tied-up when the data is being written to the file.

from multiprocessing import Process, Queue import time def receiver(queue): ## Read from the queue index = 0 with open("test.txt",'w') as file: while True: msg = queue.get() # Read from the queue and do nothing if (msg == 'DONE'): break elif not len(msg): continue for m in msg: file.write(str(m)+"\n") time.sleep(0.1) # Some expensive operation (plotting or other processing) print index, len(msg) index+=1 def sender(count, queue): ## Write to the queue. Sending does not stop if receiver is busy. bsize = 1000 #buffer size buff = [] for ii in xrange(0, count): # d = usbdata buff.append(d) if len(buff)==bsize: #Buffer is filled up! #time.sleep(0.1) queue.put(buff) # Write 'count' numbers into the queue buff = [] queue.put(buff) queue.put('DONE') print "Sender is done" if __name__=='__main__': count = 10**4 queue = Queue() # receiver() reads from queue # sender() writes to queue receiver_p = Process(target=receiver, args=((queue),)) receiver_p.daemon = True receiver_p.start() # Launch receiver() as a separate python process _start = time.time() sender(count, queue) # Send a lot of stuff to receiver() receiver_p.join() # Wait for the receiver to finish print " %s numbers took %s seconds" % (count, (time.time() - _start))