Compare 2 csv files and output different rows to a 3rd CSV file using Python 2.7

Question:

I am trying to compare two csv files and find the rows that are different using python 2.7. The rows are considered different when all columns are not the same. The files will be the same format with all the same columns and will be in this format.

oldfile.csv
ID      name     Date          Amount
1       John     6/16/2015     $3000
2       Adam     6/16/2015     $4000

newfile.csv
ID      name     Date          Amount
1       John     6/16/2015     $3000
2       Adam     6/16/2015     $4000
3       Sam      6/17/2015     $5000
4       Dan      6/17/2015     $6000

When I run my script i want the output to be just the bottom two lines and written in a csv file unfortunately I simply cant get my code to work properly. What I have written below prints out the contents of the oldfile.csv and it does not print the different rows. what i want the code to do is print out the last to lines in a output.csv file. i.e.

output.csv
3       Sam      6/17/2015     $5000
4       Dan      6/17/2015     $6000

Here is my code python 2.7 code using the csv module.

import csv

f1 = open ("olddata/olddata.csv")
oldFile1 = csv.reader(f1)
oldList1 = []
for row in oldFile1:
    oldList1.append(row)

f2 = open ("newdata/newdata.csv")
newFile2 = csv.reader(f2)
newList2 = []
for row in newFile2:
    newList2.append(row)

f1.close()
f2.close()

output =  [row for row in oldList1 if row not in newList2]

print output

unfortunately the code only prints out the content of oldfile.csv. I have been working on it all day and trying different variations but I simply can not get it to work correctly. Again, your help would be greatly appreciated.

You're currently checking for rows that exist in the old file but aren't in the new file. That's not what you want to do.

Instead, you should check for rows that exist in the the new file, but aren't in the new one:

output =  [row for row in newList2 if row not in oldList1]

Also, your CSV files are TSVs, so they won't be loaded properly. You should instruct the csv module to use TSV to open your files. Your code can also be simplified.

Here's what you could use:

import csv

f1 = open ("olddata/olddata.csv")
oldFile1 = csv.reader(f1, delimiter='\t')
oldList1 = list(oldFile1)

f2 = open ("newdata/newdata.csv")
newFile2 = csv.reader(f2, delimiter='\t')
newList2 = list(newFile2)

f1.close()
f2.close()

output1 =  [row for row in newList2 if row not in oldList1]
output2 =  [row for row in oldList1 if row not in newList2]

print output1 + output2

Related:

Calling function and passing arguments multiple times

How can I resolve my variable's unexpected output?

Displaying a 32-bit image with NaN values (ImageJ)

Identify that a string could be a datetime object

How to change the IP address of Amazon EC2 instance using boto library

Using counter on array for one value while keeping index of other values

What is the difference between <> and == in python?

In sklearn, does a fitted pipeline reapply every transform?

How to use template within Django template?

How to check for multiple attributes in a list

Python - Opening and changing large text files

Peewee: reducing where conditionals break after a certain length

Spring-integration scripting with Python

Sum of two variables in RobotFramework

Pandas Dataframe Complex Calculation

Slicing a Python OrderedDict

SQLAlchemy. 2 different relationships for 1 column

how to fetch a column in browse_record_list in orm browse method in openERP

SyntaxError: invalid syntax?

Pandas - Dropping multiple empty columns

How do I read this list and parse it?

Matplotlib: Plot the result of an SQL query

group indices of list in list of lists

How to remove structure with python from this case?
python,python-2.7
How to remove "table" from HTML using python? I had case like this: paragraph = ''' <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit. Quidem molestiae consequuntur officiis corporis sint.<br /><br /> <table> <tr> <td> text title </td> <td> text title 2 </td> </tr> </table> <p> lorem ipsum</p> ''' how...

sys.argv in a windows environment

python,windows,python-3.x
I'm attempting to learn python using the book 'a byte of python'. The code: import sys print('the command line arguments are:') for i in sys.argv: print(i) print('\n\nThe PYTHONPATH is', sys.path, '\n') outputs: the command line arguments are: C:/Users/user/PycharmProjects/helloWorld/module_using_sys.py The PYTHONPATH is ['C:\\Users\\user\\PycharmProjects\\helloWorld', 'C:\\Users\\user\\PycharmProjects\\helloWorld', 'C:\\Python34\\python34.zip', 'C:\\Python34\\DLLs', 'C:\\Python34\\lib', 'C:\\Python34', 'C:\\Python34\\lib\\site-packages']...

odoo v8 - Field(s) `arch` failed against a constraint: Invalid view definition

python,xml,view,odoo,add-on
I want to create a new view with a DB-view. When I try to install my app, DB-view was created then I get error: 2015-06-22 12:59:10,574 11988 ERROR odoo openerp.addons.base.ir.ir_ui_view: Das Feld `datum` existiert nicht Fehler Kontext: Ansicht `overview.tree.view` [view_id: 1532, xml_id: k. A., model: net.time.overview, parent_id: k. A.] 2015-06-22...

How does the class_weight parameter in scikit-learn work?

python,scikit-learn
I am having a lot of trouble understanding how the class_weight parameter in scikit-learn's Logistic Regression operates. The Situation I want to use logistic regression to do binary classification on a very unbalanced data set. The classes are labelled 0 (negative) and 1 (positive) and the observed data is in...

represent an index inside a list as x,y in python

python,list,numpy,multidimensional-array
I have a list which contains 1000 integers. The 1000 integers represent 20X50 elements of dimensional array which I read from a file into the list. I need to walk through the list with an indicator in order to find close elements to each other. I want that my indicator...

Count function counting only last line of my list

python,python-2.7
Count function counting only last line of my list N = int(raw_input()) cnt = [] for i in range(N): string = raw_input() for j in range(1,len(string)): if string[j] =='K': cnt.append('R') elif string[j] =='R': cnt.append('R') if string[0] == 'k': cnt.append('k') elif string[0] == 'R': cnt.append('R') print cnt.count('R') if I am giving...

how to enable a entry by clicking a button in Tkinter?

python,tkinter
I need to activate many entries when button is clicked please do not write class based code, modify this code only because i need to change the whole code for the project as i did my whole project without classes from Tkinter import * import ttk x='disabled' def rakhi(): global...

Find the tf-idf score of specific words in documents using sklearn

python,scikit-learn,tf-idf
I have code that runs basic TF-IDF vectorizer on a collection of documents, returning a sparse matrix of D X F where D is the number of documents and F is the number of terms. No problem. But how do I find the TF-IDF score of a specific term in...

Twilio Client Python not Working in IOS Browser

javascript,python,ios,flask,twilio
I have created a simple twilio client application to make phone calls from Web Browser to phones. I used a sample Flask app to generate a secure Capability Token and used twilio.min.js library to handle calls from my HTML. The functionality works fine in Computer Browsers ans Android Phone Browsers,...

Python: can't access newly defined environment variables

python,bash,environment-variables
I can't access my env var: import subprocess, os print os.environ.get('PATH') # Works well print os.environ.get('BONSAI') # doesn't work But the env var is well added in my /home/me/.bashrc: BONSAI=/home/me/Utils/bonsai_v3.2 export BONSAI And I can access this env var from a new terminal....

trying to understand LSH through the sample python code

python,similarity,locality-sensitive-hash
the concise python code i study for is here Question A @ line 8 i do not really understand the syntax meaning for "res = res << 1" for the purpose of "get_signature" Question B @ line 49 (SOLVED BY myself through another Q&A) "xor = r1^r2" does not really...

Python recursive function not recursing

python,recursion
I'm trying to solve a puzzle, which is to reverse engineer this code, to get a list of possible passwords, and from those there should be one that 'stands out', and should work function checkPass(password) { var total = 0; var charlist = "abcdefghijklmnopqrstuvwxyz"; for (var i = 0; i...

Python Popen - wait vs communicate vs CalledProcessError

python,python-2.7,error-handling,popen
Continuing from my previous question I see that to get the error code of a process I spawned via Popen in python I have to call either wait() or communicate() (which can be used to access the Popen stdout and stderr attributes): app7z = '/path/to/7z.exe' command = [app7z, 'a', dstFile.temp,...

The event loop is already running

python,python-3.x,pyqt,pyqt4
I have the following 5 files: gui.py # -*- coding: utf-8 -*- from PyQt4 import QtCore, QtGui try: _fromUtf8 = QtCore.QString.fromUtf8 except AttributeError: def _fromUtf8(s): return s try: _encoding = QtGui.QApplication.UnicodeUTF8 def _translate(context, text, disambig): return QtGui.QApplication.translate(context, text, disambig, _encoding) except AttributeError: def _translate(context, text, disambig): return QtGui.QApplication.translate(context, text, disambig)...

Inserting a variable in MongoDB specifying _id field

python,mongodb,pymongo
I want to insert a variable, say, a = {1:2,3:4} into my database with a particular id "56". It is very clear from the docs that I can do the following: db.testcol.insert({"_id": "56", 1:2, 3:4}) However, I cannot figure out any way to insert "a" itself, specifying an id. In...

Sort when values are None or empty strings python

python,list,sorting,null
I have a list with dictionaries in which I sort them on different values. I'm doing it with these lines of code: def orderBy(self, col, dir, objlist): if dir == 'asc': sorted_objects = sorted(objlist, key=lambda k: k[col]) else: sorted_objects = sorted(objlist, key=lambda k: k[col], reverse=True) return sorted_objects Now the problem...

How do variables inside python modules work?

python,module,python-module
I am coming from a Java background with Static variables, and I am trying to create a list of commonly used strings in my python application. I understand there are no static variables in python so I have written a module as follows: import os APP_NAME = 'Window Logger' APP_DATA_FOLDER_PATH...

Python: histogram/ binning data from 2 arrays.

python,histogram,large-files
I have two arrays of data: one is a radius values and the other is a corresponding intensity reading at that intensity: e.g. a small section of the data. First column is radius and the second is the intensities. 29.77036614 0.04464427 29.70281027 0.07771409 29.63523525 0.09424901 29.3639355 1.322793 29.29596385 2.321502 29.22783249...

Strange Behavior: Floating Point Error after Appending to List

python,python-2.7,behavior
I am writing a simple function to step through a range with floating step size. To keep the output neat, I wrote a function, correct, that corrects the floating point error that is common after an arithmetic operation. That is to say: correct(0.3999999999) outputs 0.4, correct(0.1000000001) outputs 0.1, etc. Here's...

Replace nodejs for python?

python,node.js,webserver
i'm working in a HTML5 multiplayer game, and i need a server to sync player's movement, chat, battles, etc. So I'm looking for ways to use python instead nodejs, because i have I have more familiarity with python. The server is simple: var express = require('express'); var app = express();...