Compare 2 csv files and output different rows to a 3rd CSV file using Python 2.7

Question:

I am trying to compare two csv files and find the rows that are different using python 2.7. The rows are considered different when all columns are not the same. The files will be the same format with all the same columns and will be in this format.

oldfile.csv
ID      name     Date          Amount
1       John     6/16/2015     $3000
2       Adam     6/16/2015     $4000

newfile.csv
ID      name     Date          Amount
1       John     6/16/2015     $3000
2       Adam     6/16/2015     $4000
3       Sam      6/17/2015     $5000
4       Dan      6/17/2015     $6000

When I run my script i want the output to be just the bottom two lines and written in a csv file unfortunately I simply cant get my code to work properly. What I have written below prints out the contents of the oldfile.csv and it does not print the different rows. what i want the code to do is print out the last to lines in a output.csv file. i.e.

output.csv
3       Sam      6/17/2015     $5000
4       Dan      6/17/2015     $6000

Here is my code python 2.7 code using the csv module.

import csv

f1 = open ("olddata/olddata.csv")
oldFile1 = csv.reader(f1)
oldList1 = []
for row in oldFile1:
    oldList1.append(row)

f2 = open ("newdata/newdata.csv")
newFile2 = csv.reader(f2)
newList2 = []
for row in newFile2:
    newList2.append(row)

f1.close()
f2.close()

output =  [row for row in oldList1 if row not in newList2]

print output

unfortunately the code only prints out the content of oldfile.csv. I have been working on it all day and trying different variations but I simply can not get it to work correctly. Again, your help would be greatly appreciated.

You're currently checking for rows that exist in the old file but aren't in the new file. That's not what you want to do.

Instead, you should check for rows that exist in the the new file, but aren't in the new one:

output =  [row for row in newList2 if row not in oldList1]

Also, your CSV files are TSVs, so they won't be loaded properly. You should instruct the csv module to use TSV to open your files. Your code can also be simplified.

Here's what you could use:

import csv

f1 = open ("olddata/olddata.csv")
oldFile1 = csv.reader(f1, delimiter='\t')
oldList1 = list(oldFile1)

f2 = open ("newdata/newdata.csv")
newFile2 = csv.reader(f2, delimiter='\t')
newList2 = list(newFile2)

f1.close()
f2.close()

output1 =  [row for row in newList2 if row not in oldList1]
output2 =  [row for row in oldList1 if row not in newList2]

print output1 + output2

Related:

The event loop is already running

Find the tf-idf score of specific words in documents using sklearn

How can I resolve my variable's unexpected output?

Identify that a string could be a datetime object

Pandas - Dropping multiple empty columns

how to enable a entry by clicking a button in Tkinter?

Pandas Dataframe Complex Calculation

What is the difference between <> and == in python?

group indices of list in list of lists

How do variables inside python modules work?

Python recursive function not recursing

Displaying a 32-bit image with NaN values (ImageJ)

Matplotlib: Plot the result of an SQL query

represent an index inside a list as x,y in python

SyntaxError: invalid syntax?

odoo v8 - Field(s) `arch` failed against a constraint: Invalid view definition

Slicing a Python OrderedDict

Sum of two variables in RobotFramework

Python: histogram/ binning data from 2 arrays.

How to remove structure with python from this case?
python,python-2.7
How to remove "table" from HTML using python? I had case like this: paragraph = ''' <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit. Quidem molestiae consequuntur officiis corporis sint.<br /><br /> <table> <tr> <td> text title </td> <td> text title 2 </td> </tr> </table> <p> lorem ipsum</p> ''' how...

How does the class_weight parameter in scikit-learn work?

python,scikit-learn
I am having a lot of trouble understanding how the class_weight parameter in scikit-learn's Logistic Regression operates. The Situation I want to use logistic regression to do binary classification on a very unbalanced data set. The classes are labelled 0 (negative) and 1 (positive) and the observed data is in...

Python - Opening and changing large text files

python,replace,out-of-memory,large-files
I have a ~600MB Roblox type .mesh file, which reads like a text file in any text editor. I have the following code below: mesh = open("file.mesh", "r").read() mesh = mesh.replace("[", "{").replace("]", "}").replace("}{", "},{") mesh = "{"+mesh+"}" f = open("p2t.txt", "w") f.write(mesh) It returns: Traceback (most recent call last): File...

Spring-integration scripting with Python

python,spring-integration,jython
I'm trying to use Python with spring-integration and jython-standalone-2.7.0: Here is my application context: <int:inbound-channel-adapter id="in" channel="exampleChannel" > <int:poller fixed-rate="1000" /> <int-script:script lang="python" location="script/message.py" /> </int:inbound-channel-adapter> <int:channel id="exampleChannel" /> <int-ip:udp-outbound-channel-adapter id="udpOut" channel="exampleChannel" host="192.168.0.1" port="11111" /> Here is my script in Python: print "Python"...

Sort when values are None or empty strings python

python,list,sorting,null
I have a list with dictionaries in which I sort them on different values. I'm doing it with these lines of code: def orderBy(self, col, dir, objlist): if dir == 'asc': sorted_objects = sorted(objlist, key=lambda k: k[col]) else: sorted_objects = sorted(objlist, key=lambda k: k[col], reverse=True) return sorted_objects Now the problem...

How to check for multiple attributes in a list

python,python-2.7
I am making a TBRPG game using Python 2.7, and i'm currently making a quest system. I wanted to make a function that checks all of the quests in a list, in this case (quests), and tells you if any of of the quests in the list have the same...

How do I read this list and parse it?

python,list
I'm using requests and the output I get from the sites API is a list, I've been stuck trying to parse it to get the data from it. I use r = requests.get(urlas, params=params) r.json() to get the data I want. Here is a snippet of the list [{'relation_type': None,...

Count function counting only last line of my list

python,python-2.7
Count function counting only last line of my list N = int(raw_input()) cnt = [] for i in range(N): string = raw_input() for j in range(1,len(string)): if string[j] =='K': cnt.append('R') elif string[j] =='R': cnt.append('R') if string[0] == 'k': cnt.append('k') elif string[0] == 'R': cnt.append('R') print cnt.count('R') if I am giving...

sys.argv in a windows environment

python,windows,python-3.x
I'm attempting to learn python using the book 'a byte of python'. The code: import sys print('the command line arguments are:') for i in sys.argv: print(i) print('\n\nThe PYTHONPATH is', sys.path, '\n') outputs: the command line arguments are: C:/Users/user/PycharmProjects/helloWorld/module_using_sys.py The PYTHONPATH is ['C:\\Users\\user\\PycharmProjects\\helloWorld', 'C:\\Users\\user\\PycharmProjects\\helloWorld', 'C:\\Python34\\python34.zip', 'C:\\Python34\\DLLs', 'C:\\Python34\\lib', 'C:\\Python34', 'C:\\Python34\\lib\\site-packages']...

Using counter on array for one value while keeping index of other values

python,collections
After reading the answers on this question How to count the frequency of the elements in a list? I was wondering how to count the frequency of something, and at the same time retreive some extra information, through something like an index. For example a = ['fruit','Item#001'] b = ['fruit','Item#002']...

Calling function and passing arguments multiple times

python,function,loops
I want to call the function multiple time and use it's returned argument everytime when it's called. For example: def myfunction(first, second, third): return (first+1,second+1,third+1) 1st call: myfunction(1,2,3) 2nd call is going to be pass returned variables: myfunction(2,3,4) and loop it until defined times. How can I do such loop?...

Python Popen - wait vs communicate vs CalledProcessError

python,python-2.7,error-handling,popen
Continuing from my previous question I see that to get the error code of a process I spawned via Popen in python I have to call either wait() or communicate() (which can be used to access the Popen stdout and stderr attributes): app7z = '/path/to/7z.exe' command = [app7z, 'a', dstFile.temp,...

How to change the IP address of Amazon EC2 instance using boto library

python,amazon-web-services,boto
How can I assign a new IP address (or Elastic IP) to an already existing AWS EC2 instance using boto library.

Twilio Client Python not Working in IOS Browser

javascript,python,ios,flask,twilio
I have created a simple twilio client application to make phone calls from Web Browser to phones. I used a sample Flask app to generate a secure Capability Token and used twilio.min.js library to handle calls from my HTML. The functionality works fine in Computer Browsers ans Android Phone Browsers,...

Python: can't access newly defined environment variables

python,bash,environment-variables
I can't access my env var: import subprocess, os print os.environ.get('PATH') # Works well print os.environ.get('BONSAI') # doesn't work But the env var is well added in my /home/me/.bashrc: BONSAI=/home/me/Utils/bonsai_v3.2 export BONSAI And I can access this env var from a new terminal....

Strange Behavior: Floating Point Error after Appending to List

python,python-2.7,behavior
I am writing a simple function to step through a range with floating step size. To keep the output neat, I wrote a function, correct, that corrects the floating point error that is common after an arithmetic operation. That is to say: correct(0.3999999999) outputs 0.4, correct(0.1000000001) outputs 0.1, etc. Here's...

SQLAlchemy. 2 different relationships for 1 column

python,sqlalchemy
I have a simple many-to-many relationship with associated table: with following data: matches: users: users_mathces: ONE user can play MANY matches and ONE match can involve up to TWO users I want to realize proper relationships in both "Match" and "User" classes users_matches_table = Table('users_matches', Base.metadata, Column('match_id', Integer, ForeignKey('matches.id', onupdate="CASCADE",...

How to use template within Django template?

python,html,django,templates,django-1.4
I have the django template like below: <a href="https://example.com/url{{ mylist.0.id }}" target="_blank"><h1 class="title">{{ mylist.0.title }}</h1></a> <p> {{ mylist.0.text|truncatewords:50 }}<br> ... (the actual template is quite big) It should be used 10 times on the same page, but 'external' html elements are different: <div class="row"> <div class="col-md-12 col-lg-12 block block-color-1"> *django...

Replace nodejs for python?

python,node.js,webserver
i'm working in a HTML5 multiplayer game, and i need a server to sync player's movement, chat, battles, etc. So I'm looking for ways to use python instead nodejs, because i have I have more familiarity with python. The server is simple: var express = require('express'); var app = express();...

Peewee: reducing where conditionals break after a certain length

python,peewee
This is what I have: SomeTable.select.where(reduce(operator.or_, (SomeTable.stuff == entry for entry in big_list))) The problem arises when I have a relatively large list of elements in big_list and I get this: RuntimeError: maximum recursion depth exceeded Is there another way to approach this that doesn't involve splitting up the list...

how to fetch a column in browse_record_list in orm browse method in openERP

python-2.7,orm,openerp-7
I'm beginner in openERP. I'm trying to get a column in a table. While using ORM browse method and iterating that object i got the result in browse_record_list as browse_record(table.name,21). I want to fetch that particular id 21 alone through that browse method but instead im getting same browse_record as...

Inserting a variable in MongoDB specifying _id field

python,mongodb,pymongo
I want to insert a variable, say, a = {1:2,3:4} into my database with a particular id "56". It is very clear from the docs that I can do the following: db.testcol.insert({"_id": "56", 1:2, 3:4}) However, I cannot figure out any way to insert "a" itself, specifying an id. In...

In sklearn, does a fitted pipeline reapply every transform?

python,scikit-learn,pipeline,feature-selection
Apologies if this is obvious but I couldn't find a clear answer to this: Say I've used a pretty typical pipeline: feat_sel = RandomizedLogisticRegression() clf = RandomForestClassifier() pl = Pipeline([ ('preprocessing', preprocessing.StandardScaler()), ('feature_selection', feat_sel), ('classification', clf)]) pl.fit(X,y) Now when I apply pl on a new set, pl.predict(X_classify); is RandomizedLogisticRegression going...

trying to understand LSH through the sample python code

python,similarity,locality-sensitive-hash
the concise python code i study for is here Question A @ line 8 i do not really understand the syntax meaning for "res = res << 1" for the purpose of "get_signature" Question B @ line 49 (SOLVED BY myself through another Q&A) "xor = r1^r2" does not really...